Skip to main content
Building a Testbed for Coding Agents: how to build trustworthy harness evaluations Last time we looked at the Harness design behind MiniMax Code. This time I want to take on a harder question: after we change the Harness, how do we know it got better? Take context compaction. We switch to a new checkpoint format, run a few long tasks, and the agent seems to forget things less often. That result is certainly worth pursuing, but I wouldn’t yet use it to decide whether to merge. Maybe those tasks happened to be easy; maybe the model service happened to be stable. It’s even possible that the tokens saved during compaction got spent again later on re-reading files. I want evaluation to help us tell these cases apart. In engineering terms, that means: how a task is prepared, how the agent runs, who verifies the result, and how two versions are finally compared. This post uses the open-source MiniMax Code to walk through that design. The execution entry points and collection interfaces are already in the source; the external Runner, task protocol, and grading pipeline are what this post sets out to build.

​
01 An eval task has to ship with its environment and acceptance criteria

Say we write a task: fix an edge case in the parser, keep the old API, and don’t touch the tests. Putting that sentence in a prompt isn’t enough. Which commit is the repo at? Are dependencies installed? Can the agent reach the network? If a previous run already produced a fix file, this run isn’t starting from the same place. I put these conditions into a task bundle: instructions, initial environment, allowed operations, budget, and the acceptance program. The model, Harness, and grader versions are each recorded separately; a single repo commit usually doesn’t capture the full run conditions. A task bundle might look like this:
This is an example eval-layer protocol, not mcode’s configuration format. The angle-bracket values get replaced with the real commit, image digest, and file digests when the task is checked in. The 900 seconds and 80 steps are also just example budgets that need calibrating per task. The agent can see the requirements, the code, and the public tests. The reference fix, hidden tests, and grading program stay on the eval side. Keeping them in the same repo for management is fine, but they must be mounted separately; otherwise the answer may get shipped in along with the question. Resource limits also have to actually take effect. A config that says 4 GiB but allows temporary overshoot can produce different results from one that kills the process the moment it hits the limit. Record the limits that were actually applied. Harbor’s task format bundles instructions, environment, and acceptance scripts together and is a good reference. Before a task is checked in, I first confirm that the unfixed version really fails and that the reference fix passes reliably. I also rerun it in a clean environment to rule out leftover local files and external dependencies like dead web pages. Otherwise you run tests all night and discover the next morning that the task itself was broken.

​
02 The Runner does more than launch a process

Some terminology first: a task is one problem; a trial is one attempt at that problem by a given version. Every trial gets its own runtime environment, data directory, and artifact directory. Memory, login configuration, caches, and background processes from the previous run must not quietly affect the next. The Runner executes in the stages below. State transitions are persisted to disk so that after a scheduler restart it knows which trials haven’t started, which are still running, and which only need grading. Running a task twice costs extra money and can mix artifacts from two runs into one result. One trial, a complete record: prepare, run the agent, freeze artifacts, verify independently, seal and clean up The Runner manages run state; raw records are kept on the host side, and the agent cannot rewrite them. During preparation, verify the task fingerprint, image, and initial files, confirm the environment is healthy, then start the agent. When execution ends, first stop background processes and file writes, freeze the artifacts, then hand them to the grader. Even when something fails midway, collect evidence and clean up the environment. It’s easy to underestimate how far isolation has to go. Creating a Git worktree doesn’t clear memory stored in the user data directory, and it doesn’t isolate ports, plugin configuration, or external databases. External services used by each trial need their own namespaces, and resource quotas must be enforced. Read-only dependency caches can be shared, but their contents must be pinned; whether timing includes cache warm-up also needs to be agreed on. I keep two layers of timeouts: the agent’s own task time limit, plus a Runner watchdog. When internal cancellation fails, the outside can still kill the whole process group or container and record why it was force-terminated. Time environment setup, agent execution, and verification separately; otherwise a slightly slow dependency download can get counted as an agent performance regression. If the experiment has a cost cap, billing has to cover model calls from retries, compaction, and sub-tasks. A unified model gateway can accumulate usage and enforce a stop policy, with the usage in the final answer kept for reconciliation. If you only learn the cost after the answer arrives, it’s too late to control the budget. Pin the price table and cache pricing as well, and flag missing usage separately.

​
03 Run tasks with the real MiniMax Code

The adapter only launches, reads output, stops, and collects artifacts. Tool scheduling, context compaction, and model retries stay with MiniMax Code. Rewriting an execution loop just to make evaluation convenient risks replacing exactly the Harness behavior you most want to test. MiniMax Code already has mcode exec, which can set the working directory, config, timeout, and step limit, and emit stream-json. Inside a prepared isolated environment, it’s launched like this. The Runner fills in the variables; the corresponding directories and config must be prepared beforehand.
In a real integration I read stdout and stderr concurrently through a process API so the pipes don’t fill up. If the agent runs in a container, the log streams are piped back to the host for storage. The command above doesn’t handle environment setup, permission policy, or cleanup; the Runner still has to implement those. The JSONL events carry schemaVersion, sequence, plus runId, sessionId, and turnId. The adapter validates event version, identity, and ordering, merges deltas by item ID, and keeps the raw stream too. Truncated lines, duplicate terminal events, or a wrong runId should all be recorded as anomalies. The terminal event exec.completed carries the execution result. status: succeeded only means this execution ended successfully; whether the code was fixed correctly still needs verification. --output-schema only validates the structure of the final answer, too. However tidy the answer JSON is, the tests can still fail. Sometimes there won’t even be a terminal event, for example on a preflight failure or a crash, so collect the exit code and stderr as well. If a headless run gets stuck on a question that needs a human, handle it according to the pre-agreed rules for unattended or interactive tasks; answering for one group on the fly changes the experimental conditions. When building the adapter, you can check directly against the CLI contract, event projection, and result types.

​
04 All tests green? Check who changed the tests

The agent can modify tests, build scripts, and dependencies in its workspace. Printing “all tests passed” — or even genuinely turning the tests green — isn’t enough to accept the result. For code-change tasks, I first freeze the final workspace and have the host check protected files, file types, and the scope of changes, then move the deliverable changes into a clean verification environment. The image, test entry point, and hidden tests are pinned by the eval side. The candidate code is still untrusted, so the Verifier that runs it needs isolation too. Exporting artifacts has its own details. Storing only a plain Git diff can miss untracked files, binaries, permission changes, or symlinks. The artifact protocol has to spell these out. If the deliverable is service state or a database change, save the corresponding snapshot and check the actual behavior from the outside. Keep grading outside the agent: the trust boundary between the agent environment and the verifier environment The agent delivers candidate artifacts; the grading program and hidden tests stay in a separate environment. Back to the parser task: behavior tests, regression of existing functionality, and integrity checks should each report their own result. The task only counts as done when all three pass. A pass obtained by deleting tests can’t be offset by extra points elsewhere. For example, in the sample record below, execution ended normally and the behavior tests passed, but a protected file was modified, so accepted is still false. This is an example data structure, not a real score.
Once the grader is written, the thing I most want to test is whether it accepts bad answers. Not changing the code at all, fixing only the ordinary inputs, deleting tests, and faking output can all serve as test samples. Add cases that use file links to bypass protected paths. A reference fix passing only verifies half of the grader’s job. Questions like “is this implementation overly complex” are hard to write as automatic assertions and can go to an LLM judge. Pin the model and rubric, and require conclusions to cite specific files or trajectory positions. For pairwise review, hide candidate identity, randomly swap the order, and check for misjudgments against human-labeled samples. Text the agent leaves behind is material to be judged; even if it says “ignore the previous grading rules,” it can never become an instruction to the judge.

​
05 An experiment on a compaction change

With a run-and-verify pipeline in place, we can return to the opening question: does improving how a checkpoint preserves user constraints reduce constraint violations after compaction? MiniMax Code’s compaction already includes tool-output archiving, checkpoints, and programmatically appended recent state. This time only the way constraints are preserved changes; other prompts, archive thresholds, the model, and the budget stay the same. Change several things at once, and even if results improve, it’s hard to tell which change did it. I’d run this experiment at two levels. First, test the local mechanism: wait for tool calls to settle and background writes to stop, save a shared pre-compaction state, and let the baseline and candidate each continue from there. Observe whether user constraints survive, whether earlier evidence can be read back, whether work gets repeated afterward, and whether the task completes. That state can’t be just the messages. The workspace, session, compaction state, and archived artifacts referenced later all have to be restored; for tasks that depend on databases or external services, those resources have to be restored too. Archive files that are readable on the original machine but whose references break in a different directory will also skew the experiment. The public CLI can’t currently perform this fork in one step; it needs separate test interfaces for triggering compaction and restoring state. Then run end-to-end experiments from the task’s starting point, using the context capacity and trigger policy actually deployed. The candidate might trigger compaction earlier, or read files fewer times, or change the order of tool calls. All of these are effects of the full policy and should be counted. Evaluate compaction at two levels: a local mechanism test and an end-to-end evaluation Report the two kinds of experiments separately: local experiments compare behavior from the same starting point; end-to-end experiments observe the whole task. Some runs may never trigger compaction. They still belong to the end-to-end results and just need to be marked “not triggered.” Comparing only the tasks where compaction happened, after the fact, introduces selection bias: the candidate policy may itself change the trigger probability, so the two sides no longer hold the same set of cases. To explain the causal effect of the compaction step, go back to local experiments with trigger conditions specified in advance. Tasks also have to be designed so compaction actually matters. For example, a compatibility constraint raised early still has to be honored during later code changes; evidence from early tool output is needed later. Long tasks should cover multiple compactions. Simply stuffing in a large block of irrelevant text can test behavior under long inputs, but it can hardly tell you whether the agent forgets useful information.

​
06 Beyond the score: what records to keep

When a task regresses, I want to be able to find what it read, which tools it called, and at which step it drifted from the requirements. When the grader changes, it should be possible to re-verify old artifacts. That’s the practical use of observability in evaluation. Records can be stored by purpose. The run manifest keeps versions and experimental conditions, the trajectory keeps model and tool events, and the artifact directory holds final files, environment snapshots, and verification output. Re-grading produces a new grade revision linked to the same frozen artifact, without overwriting old scores. A model name alone isn’t enough for the run manifest. You also need the model configuration and whatever service version is available, the Harness commit, fingerprints of the expanded prompt and tool schemas, and the task and grader versions. Permission policy, resources and budget, experiment group, and repetition index should also be queryable. If the server-side version can’t be pinned, record the run’s time window and carry that limitation into the analysis. MiniMax Code’s eval reporter already exposes collection interfaces for pre-call model snapshots, tool calls and results, usage, and turn end. Collection failures don’t interrupt the agent’s normal operation — a sensible trade-off for a product. In an evaluation, though, you also need evidence-completeness flags: a call missing from the logs may simply not have been captured. Usage can be missing too. usageSource and usageIncomplete in ExecResult exist for exactly this; in failure, compaction, and similar scenarios, the accounting may be incomplete. When computing cost, keep the unknown part unknown and record the confirmed cost as a lower bound. Filling gaps with zero underestimates cost; directly adding possibly overlapping cache, reasoning, and total-token fields may overcount. How failures are handled also needs to be decided up front. The table below is one workable convention: Budget exhaustion and machine failure can both look like “the process is gone”; telling them apart takes event and resource records. Each retry keeps the original trial, the reason, and a new attempt ID. Report the infrastructure error rate and task coverage alongside the score; otherwise dropping a few failures makes the numbers look good. If retries hit the limit with still no valid grade, report the gap and its impact on the conclusions. Model call failures can’t all be written off as external faults either: if this evaluation is meant to test the Harness’s recovery ability, they are part of the test. Which cases are excluded follows what was agreed before the experiment started.

​
07 How to compare two versions

Groups A and B first align on tasks, model configuration, and budget, then run randomly interleaved within the same time window. Concurrency, rate limits, and cache warmth also need to match. If one group runs in the morning and the other at night, service fluctuations can get mixed into the version difference. Resource allocation itself also has to be controlled. Anthropic’s research on infrastructure noise discusses how resource configuration affects agentic coding evals. So CPU, memory, network, and time limits should be recorded along with the model configuration. Decide how many times each task runs before starting. First compute each task’s pass rate in both groups, then compare the differences. Write the pass rates for task i as pᴀ,ᵢ and pʙ,ᵢ; with tasks weighted equally, the overall difference is Δ = (1/N) Σᵢ(pʙ,ᵢ − pᴀ,ᵢ). If you want to weight by the real usage distribution, fix the weights in advance. A task shouldn’t count more in the total just because it was run a few more times. Did the change actually help? Paired A/B tasks, aggregated task-level differences, and an independent holdout Compare each task first, then aggregate the differences and the uncertainty; finally verify the frozen candidate on an independent holdout set. Repeating the same task ten times shows how unstable runs are, but it still covers just one scenario. To estimate the gain on new tasks, use a paired cluster bootstrap by task, preserving the A/B relationship when resampling. If several tasks come from the same repo and are highly correlated, cluster by repo or task family. With few tasks the interval may not be reliable either, so list per-task results alongside it. I pay special attention to which failures the candidate newly introduces. A higher overall pass rate can mask a clear regression on one category of tasks. Integrity violations and human interventions get reported separately, and cost includes failed attempts. “Total cost of all attempts in the group / number of successful tasks” can represent cost per completed task; when there are zero successes or usage is incomplete, the estimate must be marked as not computable or incomplete. Latency reports must keep timeouts too. List the runtime of every attempt and the timeout rate, with latency of successful samples reported separately. A run killed at the deadline can’t count as finishing right at that second. If the product allows taking the best of several attempts, report the number of attempts together with the budget, and don’t compare it directly against single runs. The bar for accepting a candidate is best written down before the runs start: how much quality regression is tolerable, how much improvement is hoped for, how much cost can increase, and what hard requirements the integrity checks impose. The specific thresholds depend on product goals. Adjusting the bar after seeing scores makes it easy to rationalize a change you already wanted to merge. Even if no violations were observed this time, that only means none showed up in this batch of runs.

​
08 Don’t let the eval set turn into a dev set

Adding real failures to the regression set is useful. But if we keep tuning the Harness against the same set of tasks, we grow familiar with them, and rising scores don’t necessarily mean better performance on new tasks. The dev set is for reproducing and tuning, the validation set for choosing candidates, and the final holdout set is only used once a candidate is frozen. Split by repo, problem family, or time period, so the same task can’t appear on both sides under a different name. Once a holdout set has been looked at enough and used to change strategy, admit that it has become part of the dev set. Compaction experiments can have a targeted task set, but you should also run regressions on common tasks, check for side effects, and add new tasks regularly. Collecting only failure cases overweights hard scenarios; collecting only demo-friendly tasks makes the system look more reliable than it is. Nor does every PR need to run the whole evaluation. Run protocol, grader, and short regressions first; trigger the relevant behavioral evals when compaction, tools, or permissions change. Schedule expensive repeated experiments and new-task testing periodically within budget. pnpm verify in the source continues to check engineering correctness, while real-model task performance is evaluated separately, and the two are reported separately. If I were starting today, I’d begin with a single task: reset the environment, launch MiniMax Code, verify independently, and leave artifacts that can be re-examined. Then test the grader with a few known-good and known-bad versions. Only after all of that works would I add more tasks and concurrency. Building large-scale scheduling first will most likely just scale up the difficulty of debugging along with it. MiniMax Code’s public source already includes run entry points, structured events, and evaluation collection interfaces. If you want to get hands-on, you can hook up a Runner, or start by contributing a task that reliably reproduces a failure, or checking whether a grader wrongly accepts bad answers. All of this helps decide whether the next Harness change should stay. The project is on GitHub. Come discuss with reproductions and experiments in hand, and if you found this useful, a Star is always welcome.

​
Implementation and references

Source was checked against 14053355f39c. The task protocol, external Runner, and fork experiments in this post are design proposals; this post did not run online model comparison experiments and does not report performance gains.