Compounding Feedback Loops as Your Moat

The opening scene

Two SaaS companies. Same vertical. Similar AI products. Q1 2025.

Both ship customer support agents in March. Both pick the same frontier model. Both have engineering teams of similar pedigree. The PRDs read alike. The first month’s resolution rates are within two points of each other — call it 38% and 36%.

By Q1 2026, those numbers have gone in opposite directions. The first company is at 71% resolution and climbing about three points a quarter. The second company is at 31% and declining slowly. Same model. Same engineering quality. Same PRD on day one.

What happened in the year between?

The first company built a closed feedback loop the same week they shipped. Every conversation the agent had — successful, failed, escalated, abandoned — produced a structured trace. The traces fed an outcome-attribution layer that tagged each conversation with whether the customer’s actual problem got resolved (not whether the agent claimed it did). Once a week, an automated pipeline scored the traces, surfaced the bottom 5% of failure clusters, and proposed harness edits — a tool boundary tightened here, a system prompt clarified there, a routing rule added for a recurring miss. A small validation harness ran each proposed edit against a golden eval set before it shipped. Once it passed, it shipped. The loop ran 3.4 times a week on average.

The second company shipped the model and called it done. They have a quarterly business review where someone presents a slide of agent metrics. The metrics are flat to declining. The team’s read is “the model isn’t smart enough.” The remediation plan is “wait for the next model release.” When the next model release arrived in October 2025, they swapped it in. Resolution went up two points. Six weeks later it was back to where it had been.

The difference between these two companies is not their model. The model was the same. The difference is that one of them built a structure where production sharpens production, and the other one didn’t.

That is what this post is about.

A 60-second answer

A compounding feedback loop is a system in which the output of an AI agent becomes structured input that mechanically improves the next version of the agent, on a cadence faster than any competitor without your deployment footprint can reproduce.

Four properties make it real:

• Output becomes input. Each interaction produces a signal that feeds the next interaction.
• Attribution to outcome. Each signal has a real-world result attached, not just a log line.
• Mechanical learning, not organisational learning. The loop runs on automation, not on humans reviewing dashboards quarterly.
• Velocity as the moat indicator. Self-Optimisation Rounds per week, the metric defined in BONUS-T01 §7. Above 3.2 a week is world-class. Below 1 a week means the loop is barely running.

If your top AI feature lacks any one of those, you do not have a loop. You have a snapshot of capability that decays at the speed of model releases by your competitors.

Why this is the dominant 2026 strategic insight

In L1-T08 we walked through the inference cost layers and the gap between frontier closed models and the open frontier had narrowed to a four-times multiplier on the same Intelligence Index — DeepSeek V4 at roughly $1,071 of compute per benchmark run versus Claude Opus 4.7 at roughly $4,811. That gap closes further every quarter. By the time L3-T02 ships, “which model” is a procurement decision, not a strategy decision.

Capability is now rentable.

What is not rentable is the loop where your production traffic, your customer outcomes, and your operational telemetry combine into a private signal that nobody else has. That signal is the only thing left that compounds. Everything else — the model, the cloud, the engineers, the design language — is a check away from a competitor.

This is why every serious AI company in 2026 is structurally obsessed with three questions:

• What is the loop?
• How fast does it run?
• What proprietary asset does it produce that a competitor with the same model cannot produce?

If a PM cannot answer those three questions about their top AI feature, the feature is not a moat. It is a starting position.

Living Software — and the three layers it compounds across

There is a name for the structure those three questions point at. The practitioner community in 2026 has converged on it: Living Software. Software systems that improve continuously through their own use, treating user interactions as first-class training signal rather than as logs to be archived. The frame is not poetic; it is architectural. Logs are stored and forgotten. Signal is processed, attributed, and fed back. The difference shows up at month twelve of a product’s life, then violently at month twenty-four.

Living Software has a structure. It runs on three nested feedback layers, each with its own cadence, its own owner, and its own moat property. Most teams build one of the three, name it “the eval flywheel,” and stop. The teams that compound build all three.

Micro — the per-trace layer

Every individual user interaction produces signal. Did the agent succeed? Did the user accept the suggestion, override it, iterate on it, escalate to a human? Each trace is a labelled training event. The Micro layer feeds the eval suite from L2-T06 and the eval flywheel from AI Evals L3-T24. This is the layer most teams have. The two layers above it are where the leverage lives.

Meso — the per-workflow layer

Patterns across many traces in the same workflow. Which workflow types succeed? Which fail consistently? Which user populations exhibit which patterns? The Meso layer feeds prompt-pattern improvements, retrieval-index updates, and guardrail tuning. The failure-mode genealogy from AI Evals L3-T22 is the operating manual at this layer. Without Meso, every fix is a one-off; with Meso, a single pattern fix lifts hundreds of conversations at once.

Macro — the per-organization layer (Workspace DNA)

Patterns across an organisation’s full use of the AI over months and years. Preferred phrasings. Internal terminology. Recurring edge cases. Workflow shortcuts that don’t exist anywhere else. The accumulated learning is organisation-specific — this is what the practitioner community calls Workspace DNA. A competitor swapping in the same model on day one starts from zero on Workspace DNA. The customer experiences this as “the new AI doesn’t get us” — even when the new AI is technically equivalent.

Workspace DNA is the customer-side compounding moat. It does not appear on a feature comparison sheet. It is invisible in a pilot. It is decisive in a renewal.

The 85-billion-workflow proof point

ServiceNow is the most-studied public example of organisation-scale Workspace DNA done structurally. Every workflow that runs through the platform produces signal that improves AI capability for every customer. The cumulative corpus — reported around 85 billion workflows by early 2026 — is signal that no competitor can buy, license, or shortcut. The differentiation is not the model. Every competitor has the same model access. The differentiation is the workflow signal that compounds. That is the public reference architecture for Living Software at platform scale. (ServiceNow — AI Workflows at Scale.)

Duolingo runs the same pattern at consumer scale — hundreds of millions of language-learning interactions a day, each labelled by user behaviour, each feeding the pedagogy model. Competing apps with the same underlying model can’t replicate the accumulated learning. Anthropic’s Claude Code team runs the simplest form: dogfooding. The team uses the product daily; their feedback flows directly into prompt patterns and harness improvements. Three different scales. Same architectural principle.

Fresh data is non-negotiable

A 2025 Stanford HAI study on hallucination patterns reported a stark structural finding: systems running on stale training and retrieval data hallucinate roughly 35% more than systems with continuous data refresh. The mechanism is mundane and brutal: as user behaviour, terminology, prices, policies, and external context evolve, an AI trained on stale data starts confidently producing answers that no longer match reality. The user does not see “stale training.” The user sees “the AI is wrong more often than it used to be.” By the time the team diagnoses it, three months have passed and trust has been spent.

Living Software solves this structurally. The architecture is always learning from current usage. Stale-data drift cannot accumulate because the data is continuously refreshed at all three layers — Micro daily, Meso weekly, Macro monthly. Frozen software cannot solve this with a quarterly retrain; the drift is faster than the cycle.

(Stanford HAI — Hallucination Patterns and Data Freshness.)

The four properties of a real compounding feedback loop

This is the spine of the post. Read it like a checklist for any AI feature you own.

1. Output becomes input

The cheap version of this sentence is “we collect data.” Almost every company says that. It is almost always meaningless.

The real version is harder. Data sharpens the agent’s behaviour for the next user, in a way the data sitting in a warehouse cannot. That requires a path from the agent’s output, through structured logging, through outcome attribution, through a learning mechanism, back into the agent’s behaviour — without a human in the middle copying numbers from one dashboard into another.

A test: pick the most recent improvement to your AI feature. Trace it backward. If the answer is “the engineer noticed something on a call and decided to fix it,” that is organisational learning. It is fine. It is not a loop. If the answer is “the system surfaced the failure cluster, proposed an edit, validated it, and shipped it,” that is a loop.

ServiceNow’s context engine processes around 95 billion workflows a year and trains on roughly 7 trillion transactions. That number is not impressive because it is large. It is impressive because every one of those workflows carries an outcome — completed, escalated, reverted — that the engine can use to update its behaviour for the next workflow. The size matters only because the structure underneath is right.

2. Attribution to outcome

The loop measures what worked, not just what happened. This is the property that separates a flywheel from an analytics dashboard.

A typical AI feature ships with logging that captures what the agent said and what tool it called. That tells you what happened. It does not tell you whether the customer’s problem got solved.

To get attribution, three things have to be present.

First, the outcome has to be observable in the system itself, not pulled from a survey three weeks later. A support agent’s outcome is “did the customer reopen this ticket within seven days?” not “did the customer rate the conversation 5 stars?” — because the second is sparse and the first is dense.

Second, the outcome has to be linked back to the specific decisions inside the trace, not just attached to the conversation overall. Knowing the conversation failed is not enough. You need to know which decision in the conversation pushed it into the failure space.

Third, the attribution layer has to be cheap enough to run on every interaction, not just a sampled fraction. Sampling is fine for analytics; it is fatal for a learning loop, because the bottom 5% of conversations — the ones the loop should be learning from — are precisely the ones least likely to land in a sample.

Duolingo’s Birdbrain personalisation system sits on billions of learner-performance data points. The reason that asset is uncopyable is not the volume. It is that every data point carries a behaviour-outcome pair: the learner saw this exercise, in this state, and the result was a correct answer, an incorrect answer, an abandonment, or a retention session three weeks later. A competitor can rent the model. They cannot rent the structured behaviour-outcome pairs.

3. Mechanical learning, not organisational learning

This is where most teams quietly fail. They build the data, they build the attribution, and then they put a quarterly review meeting on the calendar to discuss it.

A loop that runs four times a year is not a loop. It is a project plan with a learning step.

The 2026 pattern, codified in LangChain’s Better Harness recipe, runs on a different cadence entirely. The recipe is short. Mine traces continuously. Cluster failures automatically. Diagnose root causes with structured analysis. Propose harness edits — system prompt changes, tool boundary tweaks, routing changes. Validate each proposed edit against a held-out eval set. Ship the ones that pass. Repeat on a weekly or sub-weekly cadence.

Andrej Karpathy’s Autoresearch, the framework Aakash Gupta calls “the Karpathy Loop” in his March 2026 PM guide, is the same shape pointed at codebases instead of agents. Tobi Lutke pointed Autoresearch at Shopify’s templating engine and got 53% faster rendering from 93 automated commits. Ninety-three commits a human team could have written, in theory, over months. The system wrote them in days because the loop ran on automation, not on a sprint cadence.

The pattern matters more than the tool. A loop that depends on an engineer remembering to look at a dashboard on the second Tuesday of every month will run twelve times a year. A loop that runs on automation runs 150 times a year. The first cannot beat the second over any time horizon, because the second compounds.

4. Velocity as the moat indicator

If the loop is real, it has a velocity. That velocity is measurable. The metric is Self-Optimisation Rounds per week — the number of complete trace-mine → diagnose → edit → validate cycles the loop completes in seven days. We defined this in BONUS-T01 §7 as one of the five harness metrics every AI PM should be able to read off the wall.

Empirically, in the deployments visible across vertical SaaS, customer support, and developer tooling in 2026:

• Above 3.2 rounds a week is world-class. The system is genuinely learning faster than the model release cycle.
• Between 1 and 3 rounds a week is healthy. The loop is running, but probably bottlenecked by validation throughput.
• Below 1 round a week means the loop is barely running. Almost all “we have a feedback loop” claims fall here when measured.
• Zero means there is no loop. There is a backlog and a roadmap.

A PM who can quote their feature’s Self-Optimisation Rounds per week is operating at a different altitude from a PM who can only quote feature accuracy. The first is talking about a moat. The second is talking about a snapshot.

The other four BONUS-T01 harness metrics — Context Durability, Intervention Rate, Autonomy Rate, Cost per Output — are the outputs the loop is designed to improve. They get better because the loop runs. If they are flat over a quarter, the loop is not running, regardless of what the dashboard says.

Three verified loop archetypes

Most real loops in 2026 fall into one of three archetypes. They overlap; the strongest companies run all three. The mistake is to assume you have one when you have a degraded version of it.

Archetype 1 — The Data Flywheel

The pattern: proprietary behaviour-outcome pairs at scale, fed back into the model or the harness around the model, producing personalisation or accuracy a competitor cannot reproduce without your deployment footprint.

The verified examples.

ServiceNow’s context engine handles around 95 billion workflows a year, with roughly 7 trillion transactions training the engine. Bill McDermott’s framing in the Q1 2026 earnings call was that the engine becomes structurally more useful per workflow as the corpus grows, because the routing and resolution patterns are increasingly specific to what enterprise IT actually does, not what general models think enterprise IT does.

Duolingo’s Birdbrain has been operating since the mid-2010s and now sits on billions of learner-performance data points. The system personalises lesson difficulty in real time based on the specific mistake patterns of the specific learner, in the specific language, at the specific time of day. A new entrant could ship the same lesson catalogue tomorrow. They could not ship the personalisation, because the personalisation is the function of the data, and the data was produced by years of users learning with the product.

Why this is uncopyable. The competitor with the same model and the same engineers cannot replicate the asset by writing a check, because the asset is produced by the customers using the product over time. This is the property L2-T02 (Building Compounding Moats) called flywheel-grade — the value of the asset to user N+1 is a function of what users 1 through N produced.

Where PMs get it wrong. Believing the asset is “the data we have.” The asset is the attribution layer that links behaviour to outcome. Without that, the data is content. With that, the data is fuel.

Archetype 2 — The Harness Hill-Climb

The pattern: the agent improves its own harness on a cadence. The system mines its own traces, diagnoses its own failures, proposes its own edits, validates them on its own eval suite, and ships the ones that pass. Humans set the policy and review the deltas; the system does the work.

The verified examples.

Karpathy’s Autoresearch, 42K+ stars at time of writing, is the canonical reference implementation. The shape is: pick a target codebase, define a fitness function, let an agentic system propose changes, validate each change against the fitness function, accept the ones that improve it. Karpathy’s framing — and Aakash Gupta’s elaboration of “the Karpathy Loop” for PMs — is that the loop is the unit of competitive advantage, not the agent.

Tobi Lutke’s experiment with Autoresearch on Shopify’s templating engine is the cleanest published example of the loop running on production code. The result was 53% faster rendering from 93 automated commits. The number that matters is not the speedup. It is that 93 commits shipped without 93 human-engineer-days going into them.

LangChain’s Better Harness recipe, published in April 2026, is the operational playbook for running the same pattern on agent harnesses rather than codebases. The four steps — trace mine → diagnose → harness edit → validate — are the steps of any serious 2026 agent team’s weekly cadence.

Why this is uncopyable. Not because the technique is secret. The technique is documented in public. The advantage is that the loop only works if you have a meaningful eval suite, a meaningful trace corpus, and a meaningful production deployment to mine. A competitor without those cannot run the loop, regardless of how clearly they understand the recipe.

Where PMs get it wrong. Treating “we’ll add automation later” as a deferred decision. Once the manual cadence is in place, the team’s calendar fills with the manual cadence and the loop never gets built. The right time to add automation is when the manual loop is running but slow, not “after we get something shipped.”

Archetype 3 — The Dogfooding Forge

The pattern: the company runs on its own AI product at production scale. Failures surface internally before customers see them. Each internal failure becomes a fix that ships before any customer hits the same failure. The internal organisation is the largest, fastest, most demanding QA team the product has.

The verified example.

ServiceNow’s self-IT autonomy — the company resolves over 90% of its own internal IT requests autonomously, on the same product they sell. The number is not the headline. The headline is that 90% autonomy on a population of tens of thousands of internal users, day after day, produces a failure surface area orders of magnitude larger than any external customer, and every one of those failures is observable, instrumented, and feeds the same loop the customer-facing version runs on.

McDermott’s articulation in 2026 has been blunt: the internal IT operation is a forge. It hammers the product. The product comes out harder.

Why this is uncopyable. The competitor can use the product. They cannot use it at the scale and intensity of the company that built it. The forge runs hottest where the producer is also the heaviest user.

Where PMs get it wrong. Treating dogfooding as a culture initiative — “everyone has to use the product on Friday” — rather than as an instrumented loop. If dogfooding does not produce traces, attribution, and harness edits on the same cadence as customer traffic does, it is morale, not a moat.

What kills feedback loops

Four traps. Each looks reasonable from the inside. Each is the reason most “we have a feedback loop” claims do not survive scrutiny.

The compounding test

L2-T02 (Building Compounding Moats) defined a real moat as one that passes at least two of three properties: flywheel, network effect, trust accumulation. The reason compounding feedback loops merit a post of their own is that they are flywheel-grade by construction.

If the four properties hold — output becomes input, attribution to outcome, mechanical learning, velocity above one round a week — the system is producing a private asset whose value to the next user is a function of what previous users produced. That is the definition of a flywheel.

If the loop also runs at scale, network effects begin to emerge: the agent gets better in the verticals where you have the most customers, which makes you the default vendor in those verticals, which gives you more customers in those verticals.

If the loop ships with the governance from Trap 4 — provenance, holdout, rollback — trust accumulation starts to compound, because each cycle that ships safely strengthens the customer’s belief that automation in this product is safer here than at competitors.

A loop without governance scores one of three. A loop with governance, at scale, scores three of three. Most “AI moats” claimed in 2024 and 2025 scored zero.

Remember this

1. Living Software treats user interactions as first-class training signal. Three layers: Micro (per-trace), Meso (per-workflow), Macro (per-organization). Each feeds the next.
2. Workspace DNA is the organisation-specific moat. The longer the AI runs in a given organisation, the more value-additive it becomes — and the harder it is to replace. Invisible in pilots. Decisive in renewals.
3. Fresh data is non-negotiable. Roughly 35% more hallucinations on stale-data systems (Stanford HAI, 2025). Continuous refresh is the structural fix; quarterly retrains can’t catch up.
4. ServiceNow’s 85-billion-workflow signal is the public proof. Differentiation isn’t the model. It’s the accumulated workflow signal that no competitor with the same model can shortcut.
5. Feedback architecture is product, not ops. The PM owns the strategic intent. Engineering owns the implementation. Delegating it as plumbing is how the L3 leverage point gets missed.

In practice — six steps

1. Audit your feedback architecture across the three layers. Micro: are user interactions labelled and flowing into the eval suite? Meso: are workflow-level patterns analysed and feeding harness improvements? Macro: is Workspace DNA being captured at the organisation level? A missing layer is a missing moat.
2. Build the Micro layer first. The eval flywheel from AI Evals L3-T24. Production traces flow into annotation queues. Reviewed traces flow into the eval dataset. CI updates trigger new evals. This is table stakes.
3. Build the Meso layer next. Failure-mode genealogy (AI Evals L3-T22). Pattern detection across traces. Workflow-type analysis. One Meso fix lifts hundreds of conversations — the leverage step most teams skip.
4. Then build the Macro layer. Per-organisation terminology dictionaries, workflow shortcuts, integration patterns. The PM owns what should this AI feel like to this customer at month 18 that it did not at month 1; engineering owns the storage, retrieval, and personalisation mechanics.
5. Invest in continuous data refresh. Retrieval indices that update daily. Eval datasets that absorb new failure modes monthly. Prompt patterns reviewed quarterly against current ground truth. Stale-data drift is the silent killer; the cadence is the cure.
6. Dogfood the product. The simplest form of Living Software architecture is using the product daily, the way Anthropic’s Claude Code team does. Your team’s own usage produces signal that improves the product for every customer. The cheapest Macro layer you will ever build.

How this connects to the rest of the system

A few cross-links worth holding in mind, with the substance carried in-line so the link is reference rather than dependency.

The Agent Tax framing from L1-T03 (Agentic PMF Standard) connects directly. The Agent Tax is the structural cost of operating the agent — inference, supervision, intervention, trust recovery. A flywheel shrinks the tax over time because each loop cycle reduces intervention and improves autonomy. A non-flywheel grows the tax because the agent is frozen at deployment while customer expectations move. Same agent, opposite trajectories, depending on whether the loop is real.

L2-T01 (4D Strategic Framework) — Discover, Decide, Develop, Deploy — works as a loop, not a line. The fourth D feeds back into the first; what production reveals reshapes what discovery looks for. Without that closure, 4D is a Gantt chart.

L2-T05 (Inference FinOps) is the cost discipline that pairs with this post. The loop produces the data that drives the optimisation cadence — model substitution decisions, prompt size reductions, caching policies. Without the loop, FinOps is a one-time exercise. With the loop, FinOps becomes a weekly rhythm.

L2-T06 (Evals as the New PRD) is the measurement substrate the loop runs on. The held-out eval set in the Better Harness recipe is the eval suite from L2-T06. Without a real eval suite, the loop has no validation step, which means it has no shipping criterion, which means it is not a loop.

L2-T10 (Shipping with Proof + Adoption Mechanics) — the Magnifying Glass thesis — is the compounding mechanism in action at the customer-facing layer. The loop converts exposed defects into roadmap items at the cadence of the loop’s velocity, which is precisely how shipping with proof compounds.

BONUS-T01 §7 carries the depth on Self-Optimisation Rounds as a metric.

L3-T06 (The Self-Improving Moat), forthcoming, pairs directly with this post. This post is the operational anatomy of the loop. L3-T06 covers the strategic question of how to communicate the moat to a board that wants to see “AI strategy” in twenty slides.