Ravi Teja Palanki

AI Evals · 36 episodes

T01Benchmarks ≠ Evals T02Non-Determinism T03The Quality Owner T04Failure Anatomy T05Golden Datasets T06The Three Gulfs T07Two Axes of Checking T08Traces & Observability T09The Tool Landscape T10Your First Eval Suite T11Machine Rubrics T12Judging the Judge T13Smarter Judges T14The RAG Triad T15Conversation Evals T16Agent Evals T17Trace Debugging T18Pipeline Evals T19Production Watch T20Human-in-the-Loop T21Beyond Pass/Fail T22Root Cause Analysis T23Shadow Testing T24The Eval Flywheel T25Agent Governance T26Eval Economics T27Build vs Buy T28Red Teams T29Evals as Strategy T30The Limits B01Gaming the Eval B02Dark Factories B03Physical World Evals B04Student > Teacher B05Tool Ecosystem Evals B06Production-Grade Trace Scoring

My AI Learnings ▾

The Five Series

Harness EngineeringThe runtime that makes agents reliable.Agentic StackHow the agent stack actually works.AI EvalsMeasuring what LLMs actually do.AI PM OSThe operating system for AI PMs.

Frontier Companies ▸Signal vs noise across the AI economy.

Anthropic OpenAI Google / Alphabet DeepSeek xAI / SpaceXAI Microsoft Meta

Companions

My Key TakeawaysDistilled lessons from the work.My PresentationsDecks and keynotes.

Connect With Me ↗Connect ↗

My Profile My AI Learnings

AI Evals · Episodes

T01Benchmarks ≠ Evals T02Non-Determinism T03The Quality Owner T04Failure Anatomy T05Golden Datasets T06The Three Gulfs T07Two Axes of Checking T08Traces & Observability T09The Tool Landscape T10Your First Eval Suite T11Machine Rubrics T12Judging the Judge T13Smarter Judges T14The RAG Triad T15Conversation Evals T16Agent Evals T17Trace Debugging T18Pipeline Evals T19Production Watch T20Human-in-the-Loop T21Beyond Pass/Fail T22Root Cause Analysis T23Shadow Testing T24The Eval Flywheel T25Agent Governance T26Eval Economics T27Build vs Buy T28Red Teams T29Evals as Strategy T30The Limits B01Gaming the Eval B02Dark Factories B03Physical World Evals B04Student > Teacher B05Tool Ecosystem Evals B06Production-Grade Trace Scoring

The Five Series

Harness Engineering Agentic Stack AI Evals AI PM OS

Frontier Companies

Anthropic OpenAI Google / Alphabet DeepSeek xAI / SpaceXAI Microsoft Meta

Companions

My Key Takeaways My Presentations Connect With Me ↗

Shadow Testing — AI Evals