The Living System

Models Change Every Quarter. Your Architecture Has To Survive It.

In March 2025, a mid-size health insurance company ran a production agent fleet handling prior authorization reviews. The system worked. Clinicians submitted authorization requests, the agent checked them against clinical guidelines, flagged approvals and denials with supporting evidence, and routed edge cases to human reviewers. The team had spent eight months building it.

Then their model provider shipped a new version.

The changelog said what changelogs always say: improved reasoning, better instruction following, reduced hallucination rates. The team read the changelog, saw no breaking changes, and promoted the new model to production on a Thursday afternoon.

By Monday morning, the denial rate for prior authorizations had increased 23%. Not because the clinical guidelines changed. Not because the agent's tools broke. Not because the harness assembled context differently. The new model interpreted the same clinical evidence with subtly different reasoning patterns.

The 23% spike triggered an internal review. The state insurance commissioner's office received 14 patient complaints in a single week — a 4x increase. The engineering team rolled back to the previous model version, but the rollback took 11 hours because nobody had documented exactly which model version was running before the upgrade.

Those 11 hours cost the company an estimated $340,000 in manual review labor, regulatory response effort, and expedited reprocessing of flagged cases.

Designing for the Model You Don't Have Yet

Everything you've built across this deep dive — the context pipelines, the harness, the agent architecture, the governance matrices, the reliability investments — sits on top of a model that you don't control and that changes on someone else's schedule.

Think of it like a hospital that replaces surgeons every few years. The operating procedures, instrument trays, safety checklists, and peer review process aren't designed around a specific surgeon's style. They're designed around the ROLE. When a new surgeon arrives, the hospital evaluates whether they meet the role's requirements — in a structured assessment, not in the operating room.

Discipline 1: Model-agnostic architecture. The harness should never contain a line of code that says "because this is Claude 3.5 Sonnet, do X." Your system prompt describes the TASK, not the MODEL. Your tool definitions specify contracts, not model-specific formatting quirks. Your eval criteria measure task performance, not model-specific behavior.

Discipline 2: Version pinning. You decide when to upgrade. Pin to a specific model identifier, not "latest." The version pin creates a stability guarantee: the model running today is the model running yesterday, unless someone deliberately changed it.

Discipline 3: Eval regression suites. The immune system. Regular evals measure "is the product good?" Model upgrade evals measure "did the product CHANGE?" Behavior anchors catch flipped decisions. Distribution checks catch aggregate drift. Regression-specific metrics catch subtle changes.

Discipline 4: Role abstraction. The harness doesn't call "Claude." It calls "the clinical review agent" — a role defined by capabilities, constraints, and expected behavior. The specific model is a configuration parameter.

The Migration That Breaks Everything Quietly

The model upgrade as a product event. In mature organizations, a model upgrade is treated with the same rigor as a database migration. It has a runbook. Rollback is pre-configured and tested before the upgrade begins. The PM owns the go/no-go decision because regression suite results are product quality data, not engineering metrics.

Multi-model production environments. Large enterprises increasingly run multiple models simultaneously — different models for different agent roles, different models for different regions. The living system's architecture supports this natively because role abstraction already decouples the agent from a specific model.

Compliance and audit continuity. Regulators don't audit models. They audit systems. The living system's version pinning provides the audit trail. The regression suite provides the evidence. The role abstraction provides the framework. An auditor can trace any production decision back to the model version, the eval results that qualified that version, and the governance matrix that constrained it.

Remember This

1. A production AI system sitting on a model that changes without your knowledge isn't stable — it's drifting. Your architecture must assume the model will change and design for controlled upgrades, not accidental drift.

2. Four disciplines make the living system work: model-agnostic architecture, version pinning, eval regression suites, and role abstraction. Skip any one, and the system becomes fragile in a specific, predictable way.

3. The eval suite is the immune system. Without it, model upgrades are gambles. With it, they're controlled experiments. Monthly regression runs, minimum. No exceptions.