Why Most AI Products Fail

The story

Consider a product team at a mid-market software company that built an AI assistant for their customer support workflow. Six months in, the assistant is shipping. Adoption is climbing. The most engaged customers — the ones who use it daily, ask it complex questions, run multi-step workflows — are loud advocates. NPS is high. Sales is using the assistant in pitches. Marketing is writing case studies.

The CFO asks for a margin breakdown by user. The slide that comes back ends the team's celebration. The most engaged 10% of users — the cohort the team had been optimizing for — have negative gross margin. Their inference cost per session is so high that the per-seat subscription doesn't cover the AI calls they generate. The middle of the curve is barely profitable. Only the tail (low-engagement users) actually contributes margin. The team had been optimizing for the cohort that was killing the business.

This is the Margin Death Spiral — the first of the three traps. The team did everything SaaS PM training prescribed: optimize for engagement, expand power users, build for retention. The playbook was correct for SaaS. It was catastrophic for an AI product where engagement is the primary cost driver.

Cursor's $7,225 invoice — a developer's monthly bill on a single AI dev tool — is the public version. Replit's reported negative 14% gross margin on $252M ARR is the public version at scale. These are not edge cases. They're the predictable consequence of building AI products with SaaS assumptions intact.

Consider a second team that pitches their AI initiative to leadership as an "FTE replacement." The deck says: "This will save us 15 full-time roles in customer support, worth $1.8M/year." Leadership greenlights the budget. The team ships. Six months later, the savings haven't materialized — but the workflow has changed in ways nobody scoped. The AI handles the first-tier triage. The remaining tickets that escalate to humans are harder than the average pre-AI ticket because the easy ones are now resolved automatically. The remaining humans need more training, not less. The team didn't replace 15 FTEs. They replaced 12 FTEs and upgraded the remaining 3 — net savings ~40% of the projection. Worse, the projection itself was the basis for the budget approval, which means leadership now feels deceived, even though every individual claim was honest.

This is the FTE Fallacy — trap number two. AI rarely replaces headcount cleanly. It rearranges workflows, changes the mix of work that humans do, and shifts the skill profile of the remaining team. Teams that frame their AI initiatives as straight FTE replacements set themselves up to be measured against a target that doesn't exist. Klarna's experience — the public "AI replaced 853 FTEs" headline — has a less-publicized footnote: the company subsequently re-hired humans for a meaningful portion of the work because the AI handled volume but not nuance.

Consider a third team that runs an "AI ROI" review every quarter. The dashboard shows time-saved estimates, productivity lift surveys, and qualitative testimonials. Leadership feels good. Executives quote the numbers in earnings calls. Then a board member asks: "What does the time-saved figure actually mean? Are people working fewer hours? Did we reduce headcount? Did we ship more?" The team can't answer. The "time saved" was self-reported, lightly aggregated, and never tied to a P&L line item. The ROI looks great on the slide. It evaporates under examination.

This is Measurement Theater — trap number three. AI initiatives produce a uniquely fertile environment for measurement theater because the outputs are qualitative, the timeframes are long, and the counterfactual ("what would have happened without AI") is hard to construct. Teams substitute evidence-shaped numbers — surveys, anecdotes, time-saved estimates — for evidence-grade numbers. The dashboard satisfies leadership for two quarters. By quarter three, the credibility is gone, the budget pressure starts, and the AI initiative gets quietly defunded — not because it failed, but because nobody could prove it succeeded.

A 2025 study reported that ~95% of enterprise AI pilots fail to ship into profitable production. That number is consistent with what I see across the field. The 5–17% that succeed don't have better models, better prompts, or better evals. They have honest measurement, honest economics, and honest workflow framing. They avoid all three traps from the start.

The core idea

The three traps share a common root: they all substitute SaaS reflexes for AI reality. Each trap is a place where SaaS PM training produces exactly the wrong move.

The concept — visualized

Trap 1: The Margin Death Spiral

The mechanic: in SaaS, engagement is unambiguously good — engaged users retain longer, expand more, refer more. The marginal cost of additional engagement is near zero. Every PM dashboard rewards engagement.

In AI, engagement is the primary cost driver. Each additional session generates inference calls, retrieval calls, tool calls. The most engaged users — the ones the SaaS playbook would optimize for — are the ones whose unit economics most often collapse. Cursor's power users were the ones whose monthly bills hit four figures. Replit's negative-margin segment was the engaged one.

The signature of the Margin Death Spiral: engagement metrics climbing on the dashboard while gross margin per user is invisible or hidden in aggregate numbers. The team thinks they're winning. The CFO eventually shows them they're losing.

The fix isn't to discourage engagement. It's to measure cost per outcome alongside engagement and treat the two as a paired metric from day one. A team running the harness metrics from L1-T01 — Context Durability, Intervention Rate, Self-Optimization Rounds — already has the diagnostic. Teams without it discover the problem when the CFO does.

Trap 2: The FTE Fallacy

The mechanic: leadership wants ROI numbers. The fastest way to produce a defensible number is to multiply average FTE cost by the number of "replaced" roles. The math is clean. The story is compelling. The math is also wrong.

AI rarely replaces FTEs cleanly. It changes the mix of work humans do. The easy work gets automated. The remaining work is harder on average and requires higher-skilled humans. The hiring profile shifts. The training cost shifts. The remaining team's productivity may even drop because they're now handling only the hard cases.

Klarna's public arc is the canonical version. The 2024 announcement: "AI replaced 853 FTEs." The 2025 reality: a meaningful re-hiring of humans because the AI handled volume but not nuance. The savings were real. They were also smaller than the headline, and the headline was the basis for budget decisions.

The fix is to frame AI initiatives as workflow rearrangements rather than headcount replacements. Measure: change in cycle time, change in escalation rate, change in skill mix of remaining team, change in training cost per remaining FTE. These metrics are harder to fit on a slide. They're also closer to the truth.

Trap 3: Measurement Theater

The mechanic: AI initiatives produce qualitative-feeling outputs. The natural metrics — "users say it saves them 5 hours/week" — are easy to gather and impossible to verify. Leadership wants ROI. Teams substitute evidence-shaped numbers (surveys, anecdotes, time-saved estimates) for evidence-grade numbers (P&L attribution, controlled comparisons, holdout cohorts).

The signature: a quarterly review that goes well for two quarters and falls apart in quarter three. The dashboards never produced anything a CFO could actually book. The AI investment became a faith-based line item. When budget pressure arrived, faith ran out.

The fix is to ground every AI claim in evidence-grade numbers from the start: a holdout cohort that didn't get the AI feature, a P&L line item the team commits to moving, a controlled comparison run by a finance partner who isn't the AI team. Measurement + Adoption = AI ROI is the L1-T10 framework that closes this trap. The version this chapter teaches is the diagnostic: if you can't say what P&L line moves, you don't have an ROI; you have a survey.

Where this hits in production

The 5–17% that ship into profitable production share three behaviors. First, they treat margin as a Day-1 metric, not a Day-180 cleanup. Second, they frame their initiative as workflow rearrangement, with explicit accounting for the work that gets harder alongside the work that gets easier. Third, they ground every ROI claim in an evidence-grade number that a finance partner co-owns. None of these behaviors are technical. All of them are PM operating-model decisions.

The 95% that fail share a single pattern. They installed the SaaS playbook and ran it. They optimized for engagement. They framed the initiative as FTE replacement. They reported time-saved estimates. The math works in the deck. It collapses in the second-year P&L review.

Cursor's response is instructive. When the margin reality became public, Cursor changed the pricing model, but more importantly, they changed the harness. Context compression, semantic caching, loop pruning — the techniques from Eval Economics (AI Evals L3-T26) — drove cost per outcome down while preserving quality. The fix wasn't "charge more." The fix was "make each successful outcome cheaper to produce." That's the harness mastery from L1-T01 paying off.

Replit's −14% gross margin is the case study for what happens without that mastery. $252M ARR sounds impressive. Negative gross margin means every dollar of revenue costs more than a dollar to produce. SaaS companies don't survive that pattern. The fix is the same as Cursor's: re-engineer the harness to drive cost per outcome down. The cost of not doing it is a business that scales itself into bankruptcy.

The trap

The trap is acknowledging the traps and then running the SaaS playbook anyway. Senior PMs read this chapter, agree with every word, and then walk into the Monday roadmap review and prioritize engagement features without a margin gate, frame the AI initiative as FTE replacement to win budget, and ship a quarterly review built on time-saved surveys because that's what leadership expects.

The fix is organizational, not personal. Make margin a Day-1 metric on every AI dashboard. Reframe the initiative narrative away from FTE replacement before the budget conversation. Co-own ROI claims with a finance partner before they appear in a deck. These are operating-model decisions. They have to be made once and held against the gravitational pull of the SaaS playbook on every roadmap meeting that follows.

The second trap is treating the three traps as separable. They aren't. Margin Death Spiral feeds FTE Fallacy (engagement-driven cost makes the FTE math worse), which feeds Measurement Theater (the team stops measuring real outcomes because real outcomes don't support the deck). The fix is the system — harness mastery from L1-T01, evidence-grade ROI from L1-T10, eval flywheel from the AI Evals series — applied together.

Remember this

In practice

Step 1: Audit your top three AI initiatives for the three traps. For each, answer: (a) Do we measure cost per outcome alongside engagement? (b) Have we framed the initiative as FTE replacement, or as workflow rearrangement? (c) Is our ROI grounded in P&L attribution, or in time-saved surveys? Any "no" is a trap. Fix it before the next quarterly review.

Step 2: Make margin a Day-1 metric. Add gross margin per user (or per workflow) to the AI product dashboard. Pair it with engagement on every chart. If engagement is climbing while margin is dropping, that's the Margin Death Spiral and the team needs harness re-engineering immediately, not next quarter.

Step 3: Reframe FTE-replacement claims as workflow rearrangements. Before any AI initiative goes to budget, replace "saves N FTEs" with: cycle time delta, escalation rate delta, skill-mix change for remaining team, training cost delta. The numbers are harder to put on a slide. The numbers are also defensible six months later.

Step 4: Co-own ROI claims with a finance partner. No AI ROI claim ships to leadership without a finance counterpart who validated the methodology and committed to the P&L line. This is the single highest-leverage move against Measurement Theater. The finance partner's skepticism is the team's protection.

Step 5: Run a quarterly trap audit. Once per quarter, ask: have any of the three traps crept back in? They will. Engagement-only dashboards drift back. FTE-replacement language sneaks back into the all-hands deck. Time-saved surveys reappear because the holdout cohort got noisy. The audit catches the drift before it compounds.

The practice — visualized

References