Benchmarks ≠ Evals

The Story

New York City reported its MyCity Business chatbot at 95% accuracy — and The Markup found it telling business owners they could legally take their workers' tips, reject housing-voucher holders, and refuse cash payments. Every one of those answers contradicted city law.

Launched October 2023 as a "once-in-a-generation" front door to more than 2,000 official city webpages, the system was grounded on government sources. By the measures the team tracked, things looked fine.

Those measures averaged across thousands of benign queries — parking hours, business registration, permit steps. The chatbot handled those well. It failed on the questions that carried legal weight: worker protections, housing discrimination, consumer rights. For a government compliance tool, one wrong answer in that category outweighs hundreds of correct answers about parking.

The team was measuring the wrong thing — not because they were careless, but because they were using the AI equivalent of a standardized test when the job called for a product-specific exam. The standardized test was passing. The product-specific exam was never written.

The Core Idea

When you hire someone, you might glance at their SAT scores or college GPA. Those numbers tell you something real — this person performed well under standardized conditions against a broad knowledge base. But they tell you almost nothing about whether this specific person will succeed at this specific job, with these specific customers, under these specific constraints.

AI benchmarks work the same way.

MMLU, HumanEval, HellaSwag — these measure what a model can do in a controlled lab setting. Model providers publish these numbers because they're standardized, comparable, and look impressive in a press release.

But your product doesn't operate in a lab. It operates with specific prompts you've written, specific documents you've retrieved, and specific failure consequences you're trying to avoid. A model that scores in the 95th percentile on reasoning benchmarks can still produce catastrophically wrong answers for your users.

There's a subtler version of this problem that catches experienced teams. Many tool vendors sell pre-built scoring functions — checks that claim to measure things like "helpfulness" or "faithfulness." They sound useful because they promise instant quality measurement without requiring you to define quality yourself.

Here's why they mislead: "Helpful" means completely different things for different users. For a luxury property buyer, a helpful response is polished and assumes financial sophistication. For a first-time homebuyer, a helpful response avoids jargon and explains next steps patiently. A generic helpfulness scorer treats both the same, scores both highly if the response is fluent, and catches neither's actual failure modes.

The shift from which model has the highest score to which model performs best on my product's specific requirements changes three things: how you define quality (a matrix, not a number), how you improve (prompts, retrieval, guardrails — not just model shopping), and how you compound advantage. Six months in, a team running daily evals understands their product's quality landscape with a precision a competitor starting from scratch cannot replicate by installing a different model.

Prompts are disposable. The discipline of knowing what good looks like for your users is what compounds.

Where this hits in production

Multi-tenant correctness. In enterprise B2B, many products serve multiple customers (tenants) from the same system. The same question can have different correct answers for different customers because of differing policies. Enterprise correctness is a matrix. A benchmark score is just one number.

Regulated industries reward refusal. In healthcare or finance, the correct answer is sometimes "I can't answer that." Public benchmarks reward answering. Enterprise evals need to reward disciplined refusal.

In Practice: The Air Canada Chatbot

In February 2024, a tribunal ruled against Air Canada. Their chatbot told a customer he could claim a bereavement discount retroactively. The airline's actual policy said the opposite. Total damages: CAD $812.

The chatbot likely performed perfectly well on general fluency benchmarks. What it wasn't tested on was strict policy adherence for specific, high-stakes edge cases.

If the team had measured only the general dimensions, they would have shipped with confidence. The product-specific dimensions would have stopped the launch. That gap is the entire argument of this topic.