Shipping Agents to Production: Architecture Restraint and the Two Evaluation Loops
Three recurring failure modes in agent projects, and the engineering answer: workflows first, context discipline, an offline gate plus online monitoring, and attributable traces.
Over the past three years, instruction following and native tool use have improved steadily, while frameworks have turned planning, memory and state management into standard components. Building an agent has never been easier. Yet production projects keep failing in three familiar ways: they stay at demo stage because real traffic exposes dirty data and undefined boundaries; they stall during scale-up because every prompt fix regresses an existing capability; or they lose funding because nobody can quantify the business gain.
The common cause is not model quality. It is the absence of a reliable judgement mechanism and of engineering discipline around it.
01Workflows first, autonomous agents only when needed
Anthropic's guidance draws a hard line between a workflow, where predefined code paths control the order of model and tool calls, and an agent, where the model decides its own path, tools and planning strategy. Workflows suit well-specified, low-tolerance business steps; agents suit dynamic exploration and long-horizon reasoning.
Most business cases need a well-designed workflow plus tool use, not a multi-agent system. Premature multi-agent orchestration adds state, latency and failure modes before the single-agent baseline is even measured. Tool design matters more than architecture: descriptions determine call accuracy, and a "think tool" that lets the model emit intermediate reasoning before acting measurably improves multi-step success.
02Context engineering and the long-horizon harness
Long tasks fail through context, not through reasoning alone. A harness has to decide what stays in the window, what is compacted, what is written to external memory, and how state is reconstructed after a retry. Anthropic's five predefined workflow patterns cover most deterministic business scenarios; the moment a run becomes long, snapshotting the environment and pinning the initial state become part of the task definition.
Practical rules: keep the instruction small and stable, externalise bulky observations, make every tool call idempotent where possible, and record a resumable state after each meaningful step. Without this, a retry is a different experiment.
03The two loops that drive improvement
A production agent needs two evaluation loops, not one.
The offline loop is a gated regression suite: a private task set, a fixed attempt budget, deterministic verifiers, and pass/fail plus rubric-based quality checks for open-ended outputs. Every prompt, tool or model change runs against it before release. The online loop monitors live traffic: tool call error rates, trajectory length, fallback frequency, user corrections and cost per completed task. Offline tells you whether you broke something; online tells you whether the thing you shipped still matters.
The two loops must share a trace schema. If online incidents cannot be replayed as offline cases, the system never learns.
04Making the loops executable
Offline gating needs versioned tasks, pinned environments and a grader whose own error rate is measured. Online monitoring needs sampling rather than full capture: retain traces that support attribution, such as the failing tool call, the state hash and the deviation step. Alert on distribution shifts in trajectory length and fallback rate; both move before headline success rate does.
05Observability is the foundation
Traces are not logs. A useful trace binds inputs, tool calls, intermediate states, verifier verdicts and the final outcome into one reproducible record. That record is what makes a regression attributable, a rubric auditable and a vendor delivery checkable.
06Our delivery standard
DataMining was founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University. We build original task data and executable evaluation environments — bootable containers, deterministic verifiers, private test suites and provenance manifests — so teams can run the offline gate before they trust the online metrics. Visit datamining.cloud or write to contact@datamining.cloud.
About DataMining Founded by PhD researchers from the Institute of Software, School of Computer Science, Peking University, DataMining builds original task data and executable evaluation environments for frontier models and agents, with reproducible rubric specifications and bootable Harbor Task verifiers. Benchmarks named here are public third-party benchmarks; we align capability requirements without copying tasks. Visit datamining.cloud or write to contact@datamining.cloud.