From Runnable to Deliverable: Four Hard Hurdles in Building High-Difficulty Evaluation Tasks

Over the past six months, building training environments and evaluation tasks (Evals) for large models has become the core track of the data industry. The early logic was simple: hire a few programmers, package an open-source project or an off-the-shelf business problem into a virtual environment, then write an automatic scoring script. If the task ran and the test produced a score, the dataset was delivered.

But in the last month or two, both model labs buying data and evaluation institutes have run into the same trouble: today's models are far too good at finding shortcuts inside evaluation environments.

If a task is merely 'runnable', after a few rounds the model quickly learns its tricks. It does not need to actually solve the business problem; it only needs to find the clues left by the task author, or figure out the scoring script's temperament, to earn full marks. This is not the model 'cheating' -- its pattern matching has grown strong enough to bypass the capability the task meant to measure and hit the gaps in the scoring rules directly.

Recently, while refining a batch of hard terminal and complex engineering tasks (Terminal-Bench style) internally, we deliberately picked the 10 hardest tasks out of a candidate pool of more than 70, had one of the strongest current coding models run them 80 times (8 independent attempts per task), and then turned the tasks, environments and grading scripts inside out for inspection.

After that round, our strongest feeling was: getting a task to 'run' may take half a day; but making it truly capable of revealing a model's depth, and solid enough to deliver to clients for training and evaluation, means clearing four hard hurdles.

Hurdle 1: Don't test 'following a clean manual'; test 'who to believe when several sources disagree'.

Many coding or office evaluation tasks on the market today resemble a university final exam: hand over a clean spreadsheet and a crystal-clear manual, then ask the model to write some code and compute a number. Top models now score high on these with little effort. Yet put them to work in a real company and they often still fail. Why?

Because in real work, almost nobody hands you a clean manual.

When designing this batch of hard tasks, we deliberately changed the approach: instead of giving the model a single clean dataset, we threw several materials from different sources -- even mutually contradictory ones -- at it all at once.

For example, one task asked the model to work out the cheapest way to buy cloud servers. A traditional version would supply one month of server utilisation records and ask for the optimal plan. But in a real enterprise, the finance team's cloud bill, the ops team's monitoring dashboard and the system's own auto-scaling log are three sets of books that frequently disagree.

Take an hourly rented server that was switched off after only 20 minutes. The monitoring dashboard records 90% load for those 20 minutes; but the finance billing system invoices on the hour, not only showing the charge an hour late, but also folding the remaining 40 minutes of powered-off time into the average as '0% load'.

If the model simply merges the tables and averages them, the procurement plan will lose money. Like a seasoned financial auditor, it must first understand each system's own accounting rules, remove the mismatched timestamps and anomalous bills, and only then compute the real figure.

Another task concerns a data-centre tape library. The machine room keeps two sets of logs: a ledger maintained daily by administrators, and scan records generated automatically by the tape drive. Someone has accidentally stuck the same barcode label on two different tapes, and a power cycle has reset the machine's internal clock to the factory default.

If the model takes shortcuts and sorts by filename or timestamp, it falls straight into the trap. It has to follow the boot sequence number from each machine restart, reconstructing the timeline like a detective, before it can tell which tape holds the newest data.

Putting these everyday workplace messes into tasks means a seasoned domain expert often needs 3 to 8 hours to finish one. But only tasks like these reveal whether a model merely memorises code templates or can genuinely untangle complex business situations.

Hurdle 2: Public tasks have long been memorised; you must write programs to 'generate the scene' yourself.

The second headache in building evaluation tasks is memorisation. Today's foundation models have effectively read the public codebases, technical forums and question banks on the internet. If you simply copy an existing open-source project or a historical incident as a task, the model often does not need to reason on the spot -- it can recite the answer from pretraining memory.

To close off memorisation completely, our approach is: never reuse existing online data; instead, for each business scenario, write a dedicated 'task generator' that produces an entirely new business scene from scratch.

To test, for instance, whether a model can help a film composer align studio audio tracks, we did not look for existing music material online. We generated a brand-new orchestral rehearsal recording live, using mathematical formulas and audio synthesis algorithms.

The generator precisely embedded metronome clicks and the engineer's clapperboard pulses into the audio, and also fabricated a complete set of accompanying rehearsal schedules, score bar tables and studio switch logs.

Even oceanographic survey ships' water-temperature calibration data, legacy banking systems' binary transaction ledgers, and municipal land survey coordinates were all computed on the spot by underlying physical formulas or business-rule scripts.

This brings two concrete benefits. First, none of this data has ever appeared on the internet, so the model could not possibly have memorised the answer in advance.

Second, the tasks become 'living water'. Change the random seed and parameters in the generator, and even if a client's model grinds through the environment a thousand times, every entry shows fresh billing figures, fault locations and audio waveforms -- solving at the root the problem of a data pack being memorised and rendered useless after two months.

But while programmatic generation solves memorisation, it introduces a new risk: the generator script is itself a complete answer key. If the environment is packaged carelessly, that answer leaks to the examinee. Which leads to the third hurdle.

Hurdle 3: The examinee and the answer key must never be in the same room.

This is not paranoia. Not long ago, overseas institutes auditing well-known leaderboards found that many data vendors, to save effort, would run the generator inside the container to create the task, then execute a delete command to remove the generator and answer key, and finally hand the packaged container to the model as an exam.

Sounds fine, doesn't it? But in a computer system this is like tearing up a sheet of answers, tossing the scraps into the wastebasket in the corner of the room -- and forgetting to carry the wastebasket out.

Today's agents are operating-system veterans. When stuck on a task, they really do traverse the filesystem, comb through shell history, read residual data under /tmp and /proc, and even try to recover deleted generator scripts from filesystem journals, copying the answer key verbatim.

A subtler case: some tasks require the model to write a processing program; after submission, the grading script runs the model's program on several fresh test cases to check correctness. If the answer keys for those new test cases sit in the same filesystem, the model's program can quietly open the answer file at runtime and echo its contents.

To leave no loophole, we set a hard rule in environment engineering: the container where the model takes the exam and the containers that generate tasks and grade answers must be strictly isolated.

First, the generator runs only inside a dedicated 'task-authoring container'. Once the task data is produced, we extract only the task files and place them in a clean 'exam container'. The generator, its logs and all intermediate artefacts stay behind in the authoring container and never enter the exam room.

Even if the model turns the filesystem upside down and reads every line of shell history, it cannot find a single line of generator code -- because the authoring process never happened in that environment at all.

Second, once the model submits, we destroy the exam container immediately and carry only its answer sheet into a separate 'grading container'. Even for tasks that must execute the model's code live, we revoke all read permission on files holding the answer key before running it, so the model's code cannot touch the answers at the filesystem level.

Only with exam discipline at this level is every point the model earns genuinely computed on merit.

Hurdle 4: Beyond blocking model shortcuts, audit the grading script itself.

Many assume that once the tasks are hard and the exam room sealed, evaluation is done. But after running a top model 80 times on these 10 tasks, we found that what truly tests a team's professionalism is the final step: going back to audit the grading script we wrote ourselves, to see whether it 'let a bad answer through' or 'wronged a good one'.

The simplest example: why can a model that cannot solve the task score full marks by submitting a blank sheet?

In some financial-risk or physical-data tasks, the grader checks how far the model's number deviates from the answer key -- say, with a rule like 'if the error exceeds 0.001, mark it wrong'. Sounds airtight, right? But what if the model fails to compute and fills the cell with a null value or NaN?

Under the IEEE 754 floating-point standard, any comparison with NaN -- greater than, less than, equal to -- evaluates to false. So the 'error > 0.001' check simply fails to fire, the grader catches no exception, and cheerfully awards the blank sheet full marks.

Our post-mortem found the root cause: on receiving the model's output, the grader jumped straight into numeric comparison, missing the first gate -- 'is this a valid number at all'. Once we added that step, blank sheets could no longer slip through.

Another case is a comic instance of 'anti-cheating collateral damage'. For one task, to stop the model copying system reference files, the engineer writing the grader added a rule: 'if the submitted result matches the answer key byte for byte, it must be copied, so score zero.'

That was meant to catch cheating, but it backfired when we actually ran models on the task: the task was a deterministic computation, so a fully correct answer yields exactly one unique table layout. Which means a model that honestly computes everything 100% right would be flagged as copying by the anti-cheating rule and given zero.

We finally replaced that rule with an audit of the model's runtime filesystem access logs -- checking whether it touched paths it should not -- rather than a result-level heuristic of 'identical means copied'.

There is also a subtler class of problem: the task statement and the grading criteria are not aligned.

In the cloud-server cost task mentioned earlier, the model was in fact very sharp: in 7 of 8 attempts it saw through the mismatch between the billing and monitoring logs and got the hardest business logic right, arriving at a saving ratio of 19.7738%. The grader gave it zero 7 times. Why? Because the task statement only said 'output the precise number' without specifying how many decimal places to keep.

The model followed common engineering practice and kept 4 decimal places, while the grader rigidly demanded 6 (19.773822%).

In another land-boundary drawing task, the model computed the areas and boundaries of all 66 parcels correctly, yet failed 6 times because 'the outer frame dimensions were wrong'.

We zoomed into the drawing for a long time before finding that the answer-key generator had left one extra residual vertex at the edge of the map -- a floating-point precision artefact, a coordinate deviation of less than one billionth, invisible to the eye -- which inflated the answer key's bounding box. The model's clean drawing was instead penalised for having a smaller bounding box than the answer key.

After patching all such grading flaws, we established two iron rules for task design: any task involving numeric output must state precision, units and significant digits explicitly in the statement; and every answer-key generator must go through at least one round of 'residual data' auditing to confirm no floating-point noise contamination.

Once all grading loopholes were fixed, the final scorecard for these 80 exams came in: the strongest coding model today has an overall pass rate of about 27.5% (22 passes in 80 independent attempts). Per-task pass rates range from one eighth to three eighths.

In full candour: 8 attempts per task is a compromise under engineering cost constraints; the sample is not large and the per-task pass rate has a fairly wide fluctuation band. What we value more is not the decimal-point precision of that percentage, but the qualitative analysis of failure modes.

Yet even the 27.5% figure itself says something important: against the strongest model available, this task set still has ample discriminative power -- nowhere near a ceiling effect.

We also cross-validated on two models from different technical families, confirming that these tasks are not 'coincidentally hard' for one particular model, but genuinely test cross-model general capabilities such as multi-source reasoning, tool use and complex engineering decisions.

More valuable, I think, was spreading out all 58 failed answer sheets and reviewing them one by one. Across those 58 failures, the reasons the model fell short fell cleanly into two categories.

One category is genuinely insufficient capability. When fixing a data-centre switch configuration, for example, the model -- just to make the few failing interface tests pass -- simply deleted dozens of other perfectly healthy interfaces from the config. That is like cutting the wiring to the bedroom and kitchen to stop the living-room light flickering. Zero marks for that paper is entirely fair.

But the other category is a warning to us as task authors. In the cloud-billing and land-boundary tasks above, the model had already chewed through 99% of the hard bones and tripped only on formatting details the author never specified. Without that granular human review, a cold score alone would make you believe 'the model cannot compute a cloud bill or draw a map at all'.

Take it apart and the truth appears: the problem is that our ruler was miscalibrated, not that the model's hand was unsteady.

That distinction is crucial. Especially when the same data is used as a reward signal in post-training: a flawed grader sends out not only a wrong score but a wrong training signal -- it teaches the model dangerous policy preferences like 'just delete the extra config to score high' or 'precision does not matter, close enough is fine'.

Why spend so much effort wrestling with these 10 tasks and 80 test trajectories? Because the craft of evaluation and training data has changed.

When large models could only do simple homework, the competition was about who could hire more part-timers and stockpile tasks faster. But as models move into deep water -- taking on financial accounting, data-centre troubleshooting, engineering retrofits, work that actually saves enterprises money -- a crude question bank not only fails to measure true capability, it feeds wrong gradient signals into post-training and pushes the model further down the wrong path.

In the end, a hard evaluation set you dare deliver to a client is never just code stuffed into a container. The author must truly understand the messiness of the business; the environment engineer must block the smartest models' loopholes; and the person writing grading rules must be willing to pore over dozens of failure records, separating what the model genuinely cannot do from where our own ruler was off.

And the next phase of evaluation competition, we believe, will no longer be about who has the biggest question bank or stocks tasks fastest. It will be about whose grading withstands a model auditing it back, whose environment engineering blocks the most extreme shortcut paths, and who can still produce a discriminative new task set after every leap in model capability.

Only when this invisible, painstaking work is done solidly can every score and every data point we deliver stand up to scrutiny.

About DataMining We provide high-quality original task data and executable evaluation environments for large-model and agent R&D. What we deliver is not second-hand CoT, but Harbor Tasks with deterministic Verifiers, positive-negative hedged atomic Rubrics and environment state snapshots. Our evaluation and data coverage includes: Deep SWE / Frontier SWE long-horizon code engineering, ALE / Terminal-Bench real-workstation operation, HLE strict multimodal production lines, Harvey LAB and Automation Bench professional agents, 35+ atomic-Rubric industry workflow packs, and standardised Win11 / Ubuntu desktop Agent images. Benchmarks, figures and official releases cited here are public third-party results; capability judgements in this article are our own internal conclusions under the same measurement protocol and do not represent official third-party positions.