- 01.Why comparing eval tools by feature count is the wrong approach -- and why deployment model and operating-model fit should be the first filter.
- 02.How the market splits into three archetypes (custom build, open-source-first, managed/hybrid) that answer fundamentally different organizational questions.
- 03.The trap that catches growing teams: starting with a script, accidentally building an internal eval company, and only then realizing the build-vs-buy question was never asked.
The Story
A healthcare AI company starts small. Two engineers write a Python script that runs their LLM's outputs through custom scorers and logs results to a CSV. It works. They add more scorers, a dashboard, dataset versioning, trace logging, annotation workflows, and role-based access control.
Eighteen months later, three engineers spend half their time maintaining the evaluation infrastructure. The eval system has its own backlog, its own bugs, its own feature requests. The company has accidentally built an eval platform inside the company -- and it's not a very good one.
A second team adopts DeepEval -- a code-first, open-source framework with 50+ built-in metrics and tight CI integration. The engineering team loves it. Six months later, the product org asks to see results. QA wants annotation workflows. Clinical reviewers need structured audit trails. DeepEval is excellent at what it's designed for -- code-first testing for developers. But a framework and a platform solve different problems.
A third team picks the most polished managed platform, onboards quickly, and starts running experiments within a week. Three months later, security review surfaces a problem: the platform stores trace data in a region that doesn't comply with data residency requirements. The team is now choosing between migrating platforms, negotiating custom deployment, or accepting a compliance gap.
The Core Idea
Build-vs-buy in evaluation is not a technology question. It's a question about where your scarce engineering time should go. Every hour spent building dataset versioning, trace storage, annotation workflows, and experiment comparison UIs is an hour not spent on the AI product itself.
The real decision is between three archetypes, each with different cost structures, control tradeoffs, and organizational fits.
- Custom scorers + proprietary logic
- Custom trace storage + datasets
- Custom review workflows + RBAC
- Custom monitoring + experiments
- Framework: DeepEval -- code-first, 50+ metrics, local, CI-tight
- Platform: Langfuse -- obs + prompts + evals + OTel
- Platform: Phoenix -- tracing + evals + datasets
- Self-hostable. You run the infra.
- LangSmith -- trace + eval + deploy + RBAC
- Braintrust -- obs + eval + hybrid data control
- Confident AI -- platform layer above DeepEval
- Arize AX -- enterprise Phoenix + VPC
| Platform | Cloud Regions | Hybrid | Full Self-Host | Air-Gap | Enterprise Auth |
|---|---|---|---|---|---|
| LangSmith | US, EU | Yes | Yes | -- | RBAC, SSO, SCIM |
| Braintrust | Managed | Yes (data) | Partial | -- | Projects, API keys |
| Langfuse | US, EU, HIPAA | -- | Yes | -- | Enterprise admin |
| Phoenix | -- | -- | Yes | Yes | Spaces, API keys |
| Arize AX | Managed | -- | Yes (VPC) | -- | Enterprise |
Where This Hits in Production
Make deployment model the first filter, not the last. Teams that compare features first and deployment model last end up picking a platform that works beautifully for three months until security review blocks the rollout.
Match the archetype to your team's actual composition. Small engineering-led teams get fastest leverage from code-first testing. Mid-size teams where product, QA, and engineering all need access benefit from integrated platforms. Enterprise teams should weight deployment model, RBAC, SSO, and audit capabilities heavily.
The hybrid model is increasingly the enterprise answer. Braintrust's architecture -- customer-controlled data infrastructure, vendor-managed UI -- is a pattern designed for organizations that want managed productivity without sending sensitive data to a third-party cloud. This "split the control plane from the data plane" pattern is likely to become the standard enterprise deployment model.
Common Mistake
Accidental platform engineering.
A team starts with a script and a CSV. Over months they add dashboards, versioning, annotation, access controls, monitoring -- until they've built an internal eval platform nobody planned, nobody budgeted, and nobody is responsible for maintaining.
If you've built more than custom scorers and a simple runner, you're building infrastructure. Decide deliberately whether that's where engineering time should go.Remember This
Build-vs-buy in evaluation is not a technology question. It's about where scarce engineering time creates the most value -- in the eval infrastructure or in the AI product. For most teams, the product is the answer.
Three archetypes: custom build (maximum control, maximum maintenance), open-source-first (framework for developers, platform for operations), managed/hybrid (compressed workflow, vendor dependency). The choice depends on where your competitive advantage sits.
Not all open-source tools are the same kind of tool. DeepEval is a framework (it runs tests). Langfuse and Phoenix are platforms (they run operations). Comparing them on the same dimension produces meaningless rankings.