This is an engineering task benchmark for AI coding ability

: give a model a real repository and a clearly written requirement, then see whether it changes only what should change, turns the hidden tests green, and stays within scope. Requirement spec + Reproducible environment + Hidden tests + Reference patch + Automatic scoring — one command decides pass or fail. Every number below comes from a real run on this machine (logs downloadable).

3self-built runnable target repositories
6auto-scored tasks
6/6reference implementations pass
0/6unfixed versions fail

Method

Every task shares the same five-part structure; miss one part and it is not a valid task.

Requirement spec

In each task directory, the task.md: background, what to change, per-item acceptance criteria, and what is explicitly out of scope.

Reproducible environment

The target repo is a small runnable codebase (PHP / Python / vanilla JS) with its own tests; scoring looks only at hidden tests and timeouts.

Hidden tests

Acceptance tests are not in the repo: they are injected at evaluation time. Visible tests only guard against regressions.

Reference patch

Each task ships a solution.patchreference implementation patch -p1 as a patch that applies cleanly, proving the task is solvable and that difficulty is distinguishable.

Automatic scoring

run_task.sh One command returns PASS / FAIL with timings; runningrun_all.sh the whole packscore.py produces an aggregate report.

Anti-cheat design

Edits to visible tests are caught by hidden tests; both directions run (unfixed must fail, patched must pass) so nothing passes by appearance.

Tasks

3 repositories × 2 tasks each = 6 tasks; weights total 100.

ID	Target repo	Type	Difficulty	What it probes	Weight	Hidden cases
T01	repo-php PHP utility library	Bug fix	★★☆☆☆	Multibyte string semantics, boundary handling, output must be valid UTF-8, existing behaviour preserved	15	12
T02	repo-php PHP utility library	Feature	★★★☆☆	Business rule implementation, money rounding, free-shipping threshold edge, empty cart handling	20	24
T03	repo-py Python utility	Bug fix	★★★☆☆	Unsorted input, merging adjacent ranges, invalid input validation, pure functions without side effects	20	16
T04	repo-py Python utility	Feature	★★★☆☆	New feature, input validation with clear negative cases, round-trip consistency with existing formatters	15	14
T05	repo-web Vanilla JS library	Bug fix	★★☆☆☆	Classic algorithm correctness (Luhn check digit), separator tolerance, no regressions	12	20
T06	repo-web Vanilla JS library	Feature	★★★★☆	URL encode/decode, multi-value key collapsing, null/undefined and array semantics, round-trip consistency	18	24
Total					100	110

Reference pass rate 6/6, unfixed failure rate 6/6; regression suite retained 100%。

Timings: unfixed total 0.88s, reference total 0.79s (this machine, process startup included).

Report files:report/report.json ·report/run_all.html Full log:report/summary.html

·

Summary excerpt: bash run_all.sh Real run output (stdout from this machine)

The two blocks below are real excerpts from

T01 repo-php mode=base

: the same task T01 fails unfixed and passes once the reference patch is applied.

Unfixed · baseline

Reference · fixed

T01 repo-php mode=fixed

Visible tests all green, seven hidden tests failing — the task genuinely bites, it is not a giveaway.

With the reference patch applied, regressions and acceptance both pass, proving the task is solvable and the bar is machine-checkable.

Task	Repo	Unfixed	Reference fix	Unfixed time	Reference time
T01	repo-php	FAIL	PASS	0.15s	0.10s
T02	repo-php	FAIL	PASS	0.10s	0.10s
T03	repo-py	FAIL	PASS	0.19s	0.20s
T04	repo-py	FAIL	PASS	0.20s	0.17s
T05	repo-web	FAIL	PASS	0.12s	0.11s
T06	repo-web	FAIL	PASS	0.12s	0.11s

Scoring

Scoring on “did it go green” alone misses half the picture, so there are three dimensions.

① Correctness

All hidden tests must pass. Coverage includes happy paths, boundary values, invalid input and exception branches.

Hidden cases are injected only at evaluation time — they are not in the repo
Assertions pin exact return values and exception types

Weight 60%

② No regressions

The repo's own visible tests must stay green; deletions or contract changes are rejected.

Visible and hidden suites run together
PASS requires visible green and hidden green

Weight 25%

③ Code quality and edge handling

Were only the right places touched, were dependencies added, are empty states and invalid input handled, is naming maintainable.

The reference patch stays minimal and serves as the baseline
“Out of scope” items in task.md are scored too

Weight 15%

Extending it to any repository

The pack is not tied to these three repos — swap in your own project and the structure holds.

Add the target repository

Put a real project (or a part of it that runs standalone) into repos/ , and make sure one test command runs it.
Write the requirement

Create tasks/T0X/task.md with background, what to change, per-item acceptance criteria, and explicit out-of-scope items.
Split visible and hidden tests

Keep the regression baseline visible; put acceptance cases into tasks/T0X/hidden_tests/ and inject them at evaluation time.
Write the reference implementation and a patch

Once done, generate a patch that applies cleanly with diff -ruN base fixed > solution.patch . patch -p1 Add meta.env and you are connected
Fill in

and the two test commands; REPO needs no changes, and new tasks are picked up automatically.run_task.sh Run both directionsrun_all.sh : unfixed must fail everywhere, patched must pass everywhere. Only then is a task valid.
The template and conventions live in

bash run_all.sh; the scoring summary is produced by

and the excerpt is shown on this page. bench/readme.htmlDuanYuan · Notes & Experiments python3 score.py 冀ICP备2026047080号-1 report.json 冀公网安备13020702000398号