Series 03 / 04 · A · E · O

AI EVALS &
OBSERVABILITY

Benchmarks lie.
Judges drift.
The eval is the PRD.

Start with Topic 01 — Benchmarks ≠ Evals → See the 35-chapter map ↓

36 chapters 4 levels

Evidence · Selected

Three receipts. One discipline.

95%

NYC MyCity chatbot.

The benchmark passed. Production told small-business owners they could legally take workers' tips. They got sued.

0.95⁷ = 70%

Compound reliability.

A 95%-accurate step chained seven times succeeds 70% of the time. Five points per step nearly doubles end-to-end success.

$812

Air Canada's chatbot.

A BC tribunal made the airline liable for what its AI said. The eval the team didn't run was the one that would have caught it.

L1 · FOUNDATION10 chapters

Act 01The Discipline of Honesty

By the end: you can defend "we shipped because the eval said so," and walk into any AI quality conversation asking the questions nobody else is asking.

T01 Benchmarks ≠ Evals Why a 95% leaderboard and a sued chatbot are the same number. READ T02 Non-Determinism Same prompt. Different answer. The variance is the product. READ T03 The Quality Owner Engineering ships. Providers train. Only the PM defines good. READ T04 Failure Anatomy Read 100 outputs. Find the three patterns running 80% of failures. READ T05 Golden Datasets The answer key every eval depends on. Build once, curate forever. READ T06 The Three Gulfs Specification. Generalisation. Comprehension. Every AI failure lives in one. READ T07 Two Axes of Checking Code or judge. Online or offline. The 2×2 that decides every test. READ T08 Traces & Observability If you can't see every step, you can't fix the one that broke. READ T09 The Tool Landscape Braintrust, LangSmith, Arize, Patronus. When to pick which, and why. READ T10 Your First Eval Suite From zero to a running suite this week. The L1 capstone. READ

L2 · PRACTICE10 chapters

Act 02The Practitioner's Loop

Build judges that hold up. Evaluate RAG, chat, and agents. Survive the math: 0.85¹⁰ ≈ 20%.

T11 Machine Rubrics Write the rubric an AI judge can actually follow. Twice in a row. READ T12 Judging the Judge The judge is a model. Verify it before you trust a number from it. READ T13 Smarter Judges CoT. Pairwise. Calibration. The techniques that make judges hold up. READ T14 The RAG Triad Context relevance. Groundedness. Answer relevance. Skip one, fly blind. READ T15 Conversation Evals Single-turn scores hide the multi-turn collapse. Evaluate the whole arc. READ T16 Agent Evals Pass^k. Trajectory evaluation. The math that breaks naive autonomy. READ T17 Trace Debugging Turn "wrong answer" into the specific step, the specific span, the fix. READ T18 Pipeline Evals Evals in CI/CD. Bad changes blocked before they reach a user. READ T19 Production Watch Catch drift in hours, not quarters. Online evals that page you, not users. READ T20 Human-in-the-Loop Design the human review that scales. Sample, route, label, return. READ

L3 · ARCHITECTURE10 chapters

Act 03The Frontier

Stop treating evaluation as a testing activity. Run it as organisational infrastructure — flywheels, governance, P&L.

T21 Beyond Pass/Fail When binary scoring lies. Deep rubrics that survive a real review. READ T22 Root Cause Analysis Three causes hide 80% of failures. The contract-review agent that proved it. READ T23 Shadow Testing A/B and shadow deploys. Prove quality on real traffic before you cut over. READ T24 The Eval Flywheel Failures feed the dataset. The dataset feeds the rubric. The rubric raises the bar. READ T25 Agent Governance Progressive deployment. Kill switches. The bar an autonomous agent must clear. READ T26 Eval Economics Every eval has a price. Make evaluation a P&L line, not a hobby. READ T27 Build vs Buy When the platform earns its line item. When you build the harness yourself. READ T28 Red Teams Break your AI before adversaries do. The cheapest line of defence you have. READ T29 Evals as Strategy The eval is the requirement. Whoever owns the eval owns the product. READ T30 The Limits What evaluation can't tell you. Knowing that is the senior move. READ

L4 · LEADING EDGE6 chapters

Act 04The Open Questions

Open questions, not settled science. A serious practitioner knows where the map ends.

B01 Gaming the Eval When the model learns it's being tested and behaves better on the test. READ B02 Dark Factories Fully automated eval loops. Zero humans in the loop. What breaks first. READ B03 Physical World Evals Robots. Cars. Drones. How do you evaluate AI that touches atoms? READ B04 Student > Teacher When the model is smarter than the judge, who grades whom? READ B05 Tool Ecosystem Evals MCP, A2A, agent-to-agent calls. Evaluating AI that talks to other AI. READ B06 Production-Grade Trace Scoring Why a 0.93 dashboard still ships a product nobody uses — and the one rule that fixes it. READ

The best AI teams ship faster because they know exactly what 'good' looks like — and they measure it every day.

Start with Topic 01 → Map all 36 chapters ↓

Fin · Series 03 of 04 · Thank you for reading

Continue the syllabus. · Three next-hops, in order.

AI EVALS & OBSERVABILITY

Three receipts. One discipline.

AI EVALS &
OBSERVABILITY