Zero-Human Evaluation Loops

The Story

A platform engineering team at a mid-market SaaS company decides to automate their internal tool development. They've read about autonomous coding agents. They set one up: the agent receives a feature spec, writes code, runs the existing test suite, fixes failures, and commits when tests pass. The team removes code review from the workflow. Velocity triples.

Six weeks later, the codebase is measurably worse. The agent passes every test, but the tests only cover happy paths and syntax correctness. Edge cases around multi-tenant data isolation -- the kind a senior engineer catches in review by asking "what happens if tenant B's webhook fires during tenant A's migration?" -- are never tested because nobody encoded that concern into the harness. The agent optimized exactly what the harness measured. The harness measured exactly what was easy to formalize. The judgment that lived in the senior engineer's head never made it into the system.

A second team takes a more disciplined approach. They're building a regulatory document drafting system for a healthcare compliance firm. They follow OpenAI's self-evolving agents cookbook: a baseline agent generates drafts, an LLM judge scores them against a rubric, feedback flows back into the agent's context, and the system iterates autonomously. The loop converges -- scores improve steadily over cycles. The team presents the results to their compliance director.

The compliance director finds two problems in the first three documents she reviews. One misapplies a recent CMS rule change. Another uses language that's technically accurate but would trigger a specific audit flag the firm has learned to avoid through years of practice. The LLM judge scored both documents highly because the rubric measured structural completeness, citation accuracy, and readability. The rubric didn't measure regulatory nuance because nobody knew how to formalize "this phrasing triggers audits" into a scoring function.

A third team has the harness and the rubric. Their autonomous agent handles document processing workflows -- extracting data from uploaded files, matching it against internal records, and generating reconciliation reports. In isolation, it works. End-to-end scores are strong. They deploy it connected to live systems: Okta for authentication, Jira for ticketing, Slack for notifications, Google Drive for document storage.

The agent fails at the boundaries. Okta session tokens expire mid-workflow and the agent doesn't re-authenticate -- it retries the same expired token indefinitely. Jira's API rate-limits the agent during peak hours, and the retry logic creates duplicate tickets. A Slack notification fails silently because the webhook URL changed during a workspace migration that nobody told the agent about. Each failure is invisible to the evaluation harness because the harness tested the agent's core logic in a sandboxed environment. The real-world dependencies were never part of the test.

Three teams, three levels of sophistication, one shared lesson: zero-human loops are only as good as the world model they're allowed to practice inside. The first team had no real harness. The second had a harness that couldn't encode domain expertise. The third had a harness that couldn't simulate the messy real world. Each loop ran fast. None ran in the right direction.

The Core Idea

T32 is the point where evals stop being a quality checkpoint and become the operating system of autonomous work.

Every topic in this series has assumed a human somewhere in the loop -- designing the rubric, reviewing the annotation queue, making the release decision, interpreting the dashboard. That assumption doesn't disappear in T32. It transforms. The human moves upstream. Instead of reviewing individual outputs, the human defines the specifications, builds the validation harness, encodes the policies, and designs the scenarios that the autonomous loop will run against. Then the loop runs -- agents generating, testing, iterating -- with no per-output human review.

This is what some teams now call a "dark factory" -- borrowed from lights-out manufacturing, where the factory floor runs without human workers present. The term is useful but misleading if you hear "no humans." What it actually means is: no humans on the production line. The humans are in the control room, designing the line, calibrating the sensors, defining the tolerances, and watching the aggregate metrics.

Think of it like a lights-out manufacturing plant. The assembly line runs without anyone on the floor. Robots build, sensors measure, quality gates reject defective units. No human touches the product. But humans designed every robot's movement, calibrated every sensor's tolerance, defined every quality gate's threshold, and engineered the fallback when something the line wasn't designed for shows up. "No humans on the line" is not the same as "no human judgment in the system." It means the judgment was compiled into the machinery before the lights went off.

The Architecture

The architecture that makes this work has converged across multiple teams toward a shared pattern. OpenAI's "Testing Agent Skills Systematically with Evals" reduces an agent eval to a simple, powerful unit: prompt, captured run, a small set of checks, and a score. That's the minimal atom of a zero-human loop -- not vibe-based review, but executable checks attached to traces and artifacts.

Anthropic extends that atom into long-running autonomous work. Their harness design article describes a three-agent structure -- planner, generator, evaluator -- that improved multi-hour autonomous coding sessions by making subjective or complex quality criteria concrete enough to grade. That's the strongest public technical case for why T32 is plausible: the loop is not one model writing code alone. It's a structured multi-agent architecture where one agent plans, another generates, and a third evaluates -- and the evaluator's rubric carries the judgment that a human reviewer would otherwise provide.

StrongDM's public Software Factory shows the most fully realized version of the pattern. The architecture is Seed → Validation Harness → Feedback Loop, with tokens as fuel. The seed is the intent specification. The harness is the proxy for reality -- scenario-based validation that tests not just whether the code runs but whether it behaves correctly under the conditions that matter. The feedback loop feeds evaluation results back into the next generation cycle so the system compounds correctness over iterations. Their Digital Twin Universe makes high-speed iteration economically practical: thousands of replayable scenarios per hour, dangerous failure modes tested without hitting live APIs.

The Governance Layer Most Teams Skip

The harness runs the loop. But what governs the harness? In enterprise settings, zero-human loops that scale need policies-as-code -- version-controlled, automatically enforced rules that travel with the application. OpenAI's agentic governance cookbook makes this explicit: organizations should define policies as code, apply guardrails automatically to every AI call, evaluate defenses with precision and recall, and build systems with handoffs, observability, and oversight from day one.

This is the part many "dark factory" narratives skip: zero-human loops that work in enterprise settings are usually more governed than manual review processes, not less. Manual review is governance by attention -- someone looks at the output and decides whether it's acceptable. Automated governance is governance by infrastructure -- policies enforce constraints automatically, traces make every decision auditable, escalation paths route exceptions to humans, and monitoring surfaces anomalies before they compound.

What "Zero-Human" Actually Means

"Zero-human" is a spectrum, not a binary. The public evidence in 2026 consistently describes reduced-touch, high-governance autonomy rather than truly unattended operation.

OpenAI's self-evolving agents example keeps human domain experts in the loop for high-stakes regulatory work, even as detailed correction decreases. Anthropic's harness article reports progress from autonomous sessions that needed constant intervention to multi-hour sessions that needed only structural guidance -- not zero intervention, but dramatically less. StrongDM's factory architecture is the most automated public example, but even it is designed around human-authored seeds and human-designed validation scenarios.

The honest framing is: zero-human per item, not zero-human in the system. A factory that produces 10,000 units per day without a human touching any individual unit still employs engineers who designed the line, technicians who maintain the machines, and quality managers who audit the aggregate. The same is true for agent loops. The human contribution shifts from artisanal (reviewing each output) to architectural (designing the harness, encoding the policies, calibrating the scenarios). That shift is real and valuable. Calling it "zero-human" overstates what happens and understates what's required.

Where This Hits In Practice

The team that automated generation but not validation. The agent writes fast. The tests are shallow. The harness checks syntax and basic correctness but not the things a senior reviewer catches -- architectural soundness, security implications, tenant isolation, edge-case behavior. The fix isn't slower agents. It's deeper harnesses. The harness must include customers, integrations, and economics, not just code correctness.

The team where domain expertise lives in people's heads, not in the harness. The autonomous loop converges on a local optimum defined by the rubric, but the rubric doesn't capture the tacit knowledge that domain experts carry -- the audit-triggering phrasing, the regulatory nuance, the "we learned this the hard way" heuristics. The fix is a formalization pipeline: before removing human reviewers, extract their judgment into scenarios, edge cases, and rubric dimensions. If you can't formalize it, the human stays in the loop for that specific dimension.

The team whose harness doesn't simulate the real world. The agent works beautifully in a sandboxed environment and breaks at every real-world boundary -- expired tokens, rate limits, webhook changes, API versioning. The fix is environmental fidelity: StrongDM's Digital Twin approach -- behavioral clones of dependencies that let the agent practice against realistic failure modes without hitting live services.

The enterprise where accountability is unclear once humans leave the loop. In regulated industries, "who approved this?" is a legal question, not a process question. If no human reviewed the output, the answer has to be: "The policy-as-code framework approved it, the evaluation harness verified it, the progressive deployment system staged it, and the monitoring system confirmed it -- all of which were designed and governed by these named individuals under this governance process." That chain of accountability must be designed before the first human is removed from the review loop, not after.

Connecting The Dots

From T24 (The Flywheel): T24 described the flywheel where failures become datasets and regressions become gates. T32 is what happens when that flywheel gets strong enough to run with very little per-item human touch. The difference is not conceptual -- it's architectural maturity. The flywheel's four steps become the heartbeat of the dark factory. The factory just runs the flywheel faster.

From T25 (Progressive Deployment): Progressive deployment and kill switches become more important, not less, once the production loop is more autonomous. Policy, observability, and guardrails are the scaffolding that lets autonomy scale safely. T25's staged rollout isn't a nice-to-have in T32 -- it's the safety mechanism that prevents a fast-running loop from compounding errors at scale.

From T27 (Build vs Buy): Build-versus-buy becomes sharper here. Once you want policy-as-code, trace-based evals, hidden scenarios, simulation environments, and automatic retraining, you're no longer picking a test harness -- you're choosing an operating model.

From T30 and T31 (Limits + Eval Awareness): Every limit from T30 becomes more consequential when there's no human reviewer to catch what the harness misses. T31's evaluation awareness is a direct threat to T32: if the agent optimizes for the harness rather than for real outcomes, the loop runs fast and confidently in the wrong direction.

Remember This

Every topic from T1 to T31 has built toward a measurement system. T32 is what happens when that system gets strong enough to run without waiting for a human to inspect each output.

The honest language matters. "Zero-human" doesn't mean no humans in the system. It means no humans per item. The humans move upstream -- from reviewing outputs to designing the factory that produces them. That upstream work is harder, not easier. Formalizing what a senior engineer catches in code review, or what a compliance expert notices in a regulatory draft, or what a domain specialist flags from experience -- that formalization is the real engineering challenge of T32. The agent is the easy part. The harness is the hard part.

The question T32 asks every team is: have you compiled enough judgment into your infrastructure that the loop can run without per-output human oversight? If the answer is "we think so," run the loop alongside human review for a quarter and compare. If the harness catches everything the humans catch, the humans can move fully upstream. If it doesn't, you know exactly which dimensions still need formalization. That's not a shortcut. It's the only safe path to the dark factory.