Inference FinOps for PMs

The team that thought it had cost discipline

A vertical SaaS PM ships a feature in October 2025 with a cost section in the PRD. The four cost layers from L1-T08 are filled in. The five PM-owned levers are named. The CFO signs off. The feature ships. The PM moves on to the next sprint.

Four months later the inference bill is up 47% and revenue from the feature is up 22%. The gross margin on the feature is compressing and nobody on the team has had a meeting about it for ninety days. The cost section in the PRD is now ninety days old. None of the levers have been reviewed since launch.

Engineering says “we’ll optimise after Q2 closes.” Product says “the feature is working — adoption is up.” Finance flags the line item in the next QBR. The PM spends the following sprint on an emergency cost-reduction project. The team finds 35% in immediate savings — model swap, prompt-cache rollout, batch-API migration — work that should have happened gradually over the four months.

This is the failure mode the L1-T08 PRD section can’t prevent on its own: the PRD is a Day-1 artefact. The cost trajectory is a continuous problem. Without an operational discipline that runs quarterly, the PRD’s cost section becomes a snapshot of a moving system, and the system moves away from the snapshot every week the team isn’t looking at it.

The discipline that closes the gap is Inference FinOps for PMs. FinOps is the cloud-financial-operations practice that emerged across 2018–2024 to bring engineering, finance, and product into one cadence around cloud cost. The AI inference variant is younger, sharper, and entirely PM-owned at the feature layer.

Think of FinOps as SRE for cost. Before SRE, reliability was everyone’s and no-one’s problem; after SRE, it became a measured discipline with explicit error budgets and named owners. FinOps does the same to inference spend: a quantified business metric with a PM owner, an engineering implementer, and a finance reporting line.

This post is the operating manual.

The frame to carry into every monthly business review

Cost-in-the-PRD is the entry artefact. Inference FinOps is the operational practice. The PRD section without the practice is a slide. The practice without the section is firefighting. Together, they compound — every quarter, the PM compresses cost faster than usage grows, which means gross margin expands while capability expands.

The frame:

This sentence describes the entire economic thesis of operational AI cost management in 2026. Token prices have been falling roughly 28× per year on average across the frontier (the underlying mechanic for the Inference Treadmill thesis I’ll detail in L1-T07). Total inference spend in mature deployments still grows 3–4× per year — because usage grows faster than unit cost falls. The math is brutal if you’re passive. The math is your friend if you operate against it.

FinOps is the operating layer that turns the math from brutal into beneficial.

The four FinOps disciplines

FinOps for AI is not a single dashboard or a single workflow. It’s four interconnected practices, each with a specific PM artefact and a specific cadence.

Discipline 1 — Attribution

Every dollar of inference cost mapped to a feature, a team, and a customer cohort. Not “we spent $X on Anthropic this month.” “Feature A consumed $Y for customer cohort Z, with this Cost per Output, with this P90 distribution.”

Without attribution, you can’t optimise. The team that doesn’t know which feature is driving cost can’t run the levers at the right place. Attribution is the substrate that makes every other FinOps discipline possible.

The PM artefact: the Attribution Map — a tagged inference graph where every API call, batch job, and cached read carries metadata for feature, team, customer cohort, and use case. Most engineering implementations of this exist as a tagging discipline — every request includes attribution metadata, the bill aggregates by tags, the dashboard surfaces the breakdown.

Cadence: ongoing. Tagging is a code discipline; it doesn’t have a cadence beyond “always on.”

Discipline 2 — Metering

Usage instrumented at the granularity that feeds attribution. Token counts per request, tokens per cache hit vs miss, batch-versus-real-time, per-customer rate of consumption.

Metering is the data layer underneath attribution. Without metering, you have a bill but no analytics on it. With metering, you have an inference economy you can model, forecast, and optimise.

The PM artefact: the Metering Spec — a one-page document listing every metric the system instruments and the granularity at which each is captured. Tokens per request, tokens per user, tokens per feature, cache hit rate, P50/P90/P99 latency, retry rate, error rate.

Cadence: review quarterly. Metering needs additions as new features ship. The spec is the artefact that prevents observability gaps from accumulating.

Discipline 3 — Budgeting

Cost budgets set per feature, with alert thresholds that trigger conversations before they become incidents.

The standard FinOps alerting framework that’s stabilised across 2025–2026 is the 60 / 80 / 95 / 110 structure (CloudZero FinOps in the AI Era 2026 Survey Report):

• 60% of budget consumed — informational; logged, no action required
• 80% of budget consumed — alert; trigger a conversation between PM and engineering on whether the trajectory is acceptable
• 95% of budget consumed — escalation; block expansion of the feature until an optimisation sprint runs
• 110% of budget consumed — incident; engineering + product + finance alignment required, the feature is now operating outside its cost envelope

The 80% threshold is the most important one. It’s the conversation point — not the panic point. A PM who hits 80% mid-quarter has the option to either reforecast the budget upward (because revenue is matching the trajectory) or to trigger an optimisation sprint (because revenue isn’t). Either is a valid response. The wrong response is to ignore the alert and let the trajectory cross 95% with no plan.

The PM artefact: the Feature Budget — a per-feature monthly cost budget, with named owner, with alerts wired to the four thresholds. Most companies don’t have this; they have a global infrastructure budget. The feature-level budget is the artefact that makes optimisation actionable.

Cadence: budgets review quarterly. Alert handling is real-time; the trigger is the alert, not the calendar.

Discipline 4 — Optimisation Cadence

A quarterly review across the five PM-owned cost levers from L1-T08, with named owners, named experiments, named expected savings.

This is where FinOps stops being a dashboard exercise and starts being a roadmap input. Every quarter, the PM and engineering review the levers, identify the highest-ROI optimisation, run it as a sprint, ship the savings, and update the budgets. The lever-by-lever cadence is the engine that turns the treadmill into a margin machine.

The five levers, with their FinOps cadence and the verified 2026 anchors:

The five PM-owned levers — operational cadence

Lever 1 — Model choice

Cadence · every six months for major features

The frontier moves. New models drop costs by significant multiples on regular intervals. DeepSeek V4 Pro running the Artificial Analysis Intelligence Index at $1,071 versus Claude Opus 4.7 at $4,811 (Artificial Analysis) is the canonical 2026 example — for many task surfaces, the cheaper model is acceptable.

A team that hasn’t reviewed model choice in twelve months is paying the gap between today’s frontier and last year’s choice. The review is a one-week sprint: pull the eval set, run candidates, compare cost-quality, decide. Most teams find at least one feature where a model swap saves 30–60% with acceptable quality delta.

Lever 2 — Distillation

Cadence · review when a feature crosses 100K MAU or annual spend exceeds $200K

Train a smaller model to mimic the larger one’s behaviour on your specific tasks. Lose 5–10% accuracy. Gain 5–8× cost reduction. A 2026 healthcare deployment moved from a frontier model to a 13B distilled model and cut annual inference cost from $400K to $120K — same task surface, accuracy delta inside their tolerance (TechAhead).

Distillation is the heaviest lever — it requires real engineering investment — but it’s also the most durable. A distilled model becomes infrastructure for the team. The cadence question isn’t “should we?” — it’s “which feature is large enough to justify the engineering cost?” The 100K MAU / $200K-spend threshold is roughly the inflection point.

Lever 3 — Caching

Cadence · quarterly review per major feature

Anthropic’s prompt caching documentation describes cache read tokens at 0.1× base input price — a 90% discount on repeated context, with cache writes at 1.25–2× base (Anthropic prompt caching). DeepSeek V4-Flash’s cache-hit pricing at $0.028 vs $0.14 per million input tokens is a 5× discount on the same pattern (DeepSeek V4 pricing).

Most teams don’t realise their cache hit rate. The PM’s quarterly cadence question: “For each major feature, what fraction of input tokens are repeated context, and what’s our actual cache hit rate against that potential?” The gap between potential and actual is usually large in the first quarter and shrinkable to <10% by the third.

The caching lever is uniquely PM-owned because the question of what context is repeated is a UX question, not an engineering question. The PM has to specify which context to cache. The engineer implements. The lever doesn’t pull itself.

Lever 4 — Batching

Cadence · ongoing for non-real-time workloads; default to batch where possible

For workloads that don’t need real-time response, both major API providers offer Batch APIs with substantial discounts. OpenAI Batch API offers a 50% discount on inputs and outputs (OpenAI pricing). Anthropic Batch API offers a 50% discount on both input and output tokens (Anthropic pricing). The widely-cited 80% number from earlier 2025 reporting was specific to the OpenAI o3 model price drop in June 2025, not the Batch API itself.

A clean rule for the PM: any workload that doesn’t need a response in <30 seconds should be on a Batch API by default. Overnight summarisation, weekly reports, bulk classification, periodic enrichment — all batch candidates. The 50% discount compounds across a portfolio of features.

For real-time workloads, the equivalent lever is continuous batching at the inference-server layer. PagedAttention and continuous batching push production GPU utilisation from 16–30% to 60–80%+ — meaning the same hardware serves 2–4× the requests at the same cost (vLLM project). Most modern inference stacks (vLLM, TGI, TensorRT-LLM) implement this by default; the PM’s role is to verify that the team is on a modern serving stack and that GPU utilisation is in the 60%+ range.

The lever-pulling rule of thumb: if your real-time GPU utilisation is below 50%, you’re paying for hardware you’re not using. The fix is a serving-stack review, not a model swap.

Lever 5 — Eval-driven gating

Cadence · quarterly review of model-routing rules

Not every query needs the expensive model. A PM who has a real eval set can route easy queries to a cheap model and reserve the expensive one for the queries it actually moves the needle on.

Combine quantisation and prompt pruning techniques and the savings stack: prompt pruning and quantisation can produce 60–70% token reduction on suitable workloads (Mirantis inference cost analysis, March 2026). Layered with model-tier routing (cheap model for easy queries, expensive model for hard queries), the per-feature cost can compress significantly without quality regression.

The gating lever is the most under-pulled in 2026. Teams build evals as QA artefacts (which I’ll cover in detail in L2-T16) and miss that the eval is also a routing artefact. The eval IS the routing logic. Without it, every query gets the most expensive path “to be safe.” Safe for the engineer. Expensive for the company.

Bonus lever — Loop pruning for agentic workflows

Cadence · trace review every sprint for any agentic feature

Once a feature graduates from single-turn inference to a multi-step agent, a sixth lever opens up: loop pruning. Agentic workflows routinely repeat reasoning, re-call the same tool, or pass excess context between steps. Each loop costs tokens, and the cost compounds turn-over-turn. Anthropic’s harness-design guidance documents the canonical patterns — planner/generator/evaluator splits that reduce planning loops, structured handoff artefacts that shrink context passing, and explicit early-termination signals that prevent over-recursion (Anthropic — Harness Design for Long-Running Apps). Disciplined trace review on the highest-volume agent typically yields 50–70% token reduction on multi-step patterns without quality regression.

The Cursor cost-cliff response in mid-2025 was Inference FinOps in public: context compression (loop pruning), prompt caching, and model routing shipped together with the pricing redesign (Cursor — Pricing Update). The pricing got the headlines; the cost-side work is what made the new economics defensible.

The Inference Treadmill — and how FinOps inverts it

Here’s the underlying economic dynamic that makes FinOps load-bearing for the AI PM in 2026.

Per-token frontier prices have fallen by roughly 28× per year on average across 2023–2026 (the broader pattern that L1-T07: AI Economics 101 unpacks). At first glance, this should mean inference costs collapse. They don’t, in mature deployments. Total inference spend grows 3–4× per year, because usage grows faster than unit cost falls.

The math is the treadmill: you run faster, the floor moves backward, your absolute distance increases.

What FinOps does — and only what FinOps does — is invert the dynamic. With the four disciplines and the quarterly lever cadence operating, the team compresses cost faster than usage grows. Each quarter:

• New model dropped → 30–50% on the model swap
• Cache rollout → 30–60% on repeated context
• Batch migration for eligible workloads → 50% on those workloads
• Distillation on the highest-spend feature → 5–8× on that feature
• Routing rules tightened → 20–40% on overall mix

You don’t pull every lever every quarter. You pull one lever per feature per quarter — the highest-ROI one for that feature’s current state. Compounded across a portfolio of features and a year of cadences, the savings dwarf usage growth. Gross margin expands. The treadmill becomes a margin machine.

The PMs who don’t operate this way ride the treadmill backward. The PMs who do compound the savings into roadmap leverage — because saved cost becomes available capital for the next bet.

The FinOps dashboard — what the PM owns end-to-end

The artefact that makes FinOps operational is the dashboard. Most companies have a global infrastructure dashboard. The AI PM needs a feature-level FinOps dashboard with at minimum:

The dashboard is the artefact. The four disciplines are the practice. The five levers are the moves. Together they’re an operating system.

The four traps in Inference FinOps

If the four disciplines are the diagnosis, here are the four habits to break specifically — the FinOps mistakes most AI PMs make by default and which will, quietly and expensively, undo their margin.

The sentence to carry

If you remember one frame from this post, make it that one. The four disciplines (attribution, metering, budgeting, optimisation cadence) are the structural practice. The five levers (model choice, distillation, caching, batching, eval gating) are the moves. The dashboard is the artefact. Compounded together, they’re the operating system that turns the AI cost story from “expensive forever” into “expanding capability inside expanding gross margin.”