Abstract:

Reward models (RMs) have become an indispensable fixture of the language model (LM) post-training playbook, enabling policy alignment and test-time scaling. Research on the application of RMs in code generation, however, has been comparatively sparse, with existing work largely focusing on execution feedback. This choice constrains post-training to optimizing functional correctness over self-contained executable code. In this work, we examine the training and evaluation of multilingual, multi-criteria code RMs. To this end, we first compile Themis-CodeRewardBench, a benchmark to evaluate code RMs across five preference dimensions (i.e., criteria) and eight programming languages, on which we profile 50+ code, math, and general-purpose RMs. Observing the limited proficiency of current RMs beyond scoring for functional correctness, we develop Themis-CodePreference, the largest open-source collection of code preferences to date (more than 350k preference pairs), and use it to train Themis-RM, a suite of multilingual code reward models for flexible multi-criteria scoring, ranging in size from 600M to 32B parameters. Our experiments and ablations demonstrate positive scaling trends, strong cross-lingual transfer when training on diverse preferences, and the importance of multi-criteria training for reliable code reward modeling.

Themis reward models are trained using the Bradley-Terry preference framework with a multi-stage data pipeline that mines, filters, scores, and assembles high-quality code preference pairs from open-source repositories. The models are evaluated on Code RewardBench (CRB), a benchmark of 8,866 preference pairs spanning 5 quality aspects and 8 programming languages.

Pipeline Overview

The end-to-end pipeline has three phases: dataset construction, model training, and evaluation.

                          DATASET CONSTRUCTION
                          ────────────────────
  BigQuery (github_repos)
      │
      ▼
  ┌─────────────────────┐   ┌───────────────────┐   ┌──────────────────┐
  │ 1. Commit Mining    │──▶│ 2. Repo Filtering │──▶│ 3. Ext Filtering │
  │    (SQL)            │   │    (allowlists)   │   │    (lang → ext)  │
  └─────────────────────┘   └───────────────────┘   └──────────────────┘
                                                            │
      ┌─────────────────────────────────────────────────────┘
      ▼
  ┌──────────────────────┐   ┌──────────────────┐   ┌──────────────────┐
  │ 4. Content Retrieval │──▶│ 5. Deduplication │──▶│ 6. Aspect Filter │
  │    (git fetch)       │   │    (MinHash LSH) │   │    (ModernBERT)  │
  └──────────────────────┘   └──────────────────┘   └──────────────────┘
                                                            │
      ┌─────────────────────────────────────────────────────┘
      ▼
  ┌──────────────────────┐   ┌──────────────────┐   ┌──────────────────┐
  │ 7. LLM Scoring &     │-─▶│ 8. LLM-as-a-Judge│──▶│ 9. Training Data │
  │    Instruction Synth │   │    (A/B voting)  │   │    Assembly      │
  └──────────────────────┘   └──────────────────┘   └──────────────────┘
                                                            │
                          MODEL TRAINING                    │
                          ──────────────                    │
      ┌─────────────────────────────────────────────────────┘
      ▼
  ┌───────────────────────────────────────────────────────────────────┐
  │ Bradley-Terry preference training with FSDP2 on multi-node GPUs   │
  │ (BT loss + LM regularisation + magnitude penalty, Liger kernels)  │
  └───────────────────────────────────┬───────────────────────────────┘
                                      │
                          EVALUATION  │
                          ──────────  │
      ┌───────────────────────────────┘
      ▼
  ┌───────────────────────────────────────────────────────────────────┐
  │ Code RewardBench: 8,866 pairs × 5 aspects × 8 languages           │
  │ Evaluated across scalar, MoE, and generative RM architectures     │
  └───────────────────────────────────────────────────────────────────┘

Results

Themis-RM models achieve best-in-class accuracy on Themis-CodeRewardBench, a code-specific reward model benchmark, while also matching or exceeding much larger models on established general-domain benchmarks (RewardBench V1, RewardBench V2, JudgeBench). Models are grouped by parameter class; bold marks the best in each group.

Datasets

All datasets are available on HuggingFace:

Related Work

Distributed Training Tutorial — A companion tutorial by us that walks through multi-node distributed training of scalar reward models on cloud GPU clusters. Covers cluster provisioning, high-speed networking, container management, and FSDP-based training. Useful as a standalone guide for anyone looking to reproduce the Themis training setup or adapt it to their own reward modelling workloads. Follows a simplified recipe that leverages the Axolotl framework for training reward models with the Bradley-Terry loss.

Citation

@article{themis2025,
  title={Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoring},
  author={Paul, Indraneil and Gurevych, Iryna and Glava\v{s}, Goran},
  journal={arXiv preprint arXiv:xxxx.xxxxx},
  year={2025}
}