Containerised tasks · hidden tests · real-run scoring
Comparing coding ability across models and agents by reading code does not reproduce. Tasks must run automatically and results must be judged automatically.
One directory per task: repository snapshot, visible tests, hidden tests, run script, reference patch. Scoring looks only at test results and timeouts, never at prose.
Hidden and visible tests are separated to defeat hard-coding; runs happen in isolated containers with CPU and time limits; results are JSON for easy aggregation.
Download source ↓ (free with an account)