Series 1 of 4 · Agentic Stack · Level 3 · Topic 23

Agent Anatomy

Agentic Systems
Back to Agentic Stack
In This Note You Will Learn
  • 01.The five structural components every agent is built from -- perception, reasoning, action, memory, and the loop.
  • 02.Why tools (action) matter more than models (reasoning) for product differentiation -- and the PM decision ranking that follows.
  • 03.How to determine which components your agent actually needs versus which ones add complexity without earning their keep.

Five Components, Three That Actually Matter

In early 2025, a healthcare technology company decided to build an "AI agent" for diagnostic assistance. The vision was ambitious: review patient records, cross-reference symptoms against medical literature, suggest differential diagnoses, and recommend next steps. They built all five agent components. It took nine months and $2.3 million.

The team they lost the contract to built something simpler. Three components: perception, reasoning, action. No persistent memory. No iterative loop. It took four months, it shipped, and physicians used it because it was fast, transparent, and didn't try to "think" more than the physician wanted it to.

The anatomy of an agent is five components. The anatomy of a GOOD agent is the minimum set of components the task actually demands.

Planner, Executor, Memory, Critic, Router

Every agent is built from five structural components that map to cognitive functions: perceive, think, act, remember, iterate.

An agent's anatomy is five components -- perception, reasoning, action, memory, and the loop -- where the PM's design decision is which components this specific agent needs, not whether to implement all five.

-- The definition
Fig 1. The Agent's Building Blocks
Every Agent Has Five Parts.
Most Need Three.
Core (non-negotiable)
Optional (justify before adding)
👁
Perception
Core
Takes in the environment. Reads files, parses inputs, observes state. This is your context pipeline from Levels 1 & 2.
🧠
Reasoning
Core
Decides the next action. The model's core contribution. Increasingly commoditized across frontier providers.
✍️
Action
Core
Executes on the world. Tool calls, API requests, file writes. The tool set IS the product -- different tools, different agent.
🗃️
Memory
Add when justified
Stores context across iterations. Working, episodic, and semantic. Adds cost at every step plus storage infrastructure.
🔄
The Loop
Add when justified
Reason, act, observe, repeat. Multiplies every other component's cost. The structural feature that makes agents agents.
5-component agent at 10 iterations = 50 cost events per task.
3-component single-pass = 3 cost events. Match to the job.

The ranking that changes product decisions. PM decisions ranked by impact on agent quality: (1) Tools -- the right tool set with a mediocre model outperforms the wrong set with a frontier model. (2) Context -- everything from Levels 1 and 2. (3) Loop duration. (4) Data access. (5) Model choice. The model is the least differentiating factor. Tool set and context quality are what separate products.

The Components You Skip Are the Ones That Bite

The "full-stack agent" fallacy. Each component adds infrastructure cost, maintenance burden, and failure surface. Memory requires storage, relevance ranking, staleness detection, and loading strategy. The loop requires iteration limits, divergence detection, and graceful termination. Each unnecessary component is engineering you maintain without product value.

Tool sprawl is feature creep for agents. Model accuracy on tool selection degrades significantly beyond 15-20 available tools. A model with 50 tools spends more reasoning effort choosing WHICH tool than executing the task. Curate the minimum tool set per task type.

!

The Trap

Building all five components because "that's what an agent is."

Academic definitions include all five. This creates an implicit assumption that a "real" agent needs all five. Teams implement persistent memory for stateless tasks. They build iterative loops for agents that need one well-reasoned pass.

Apply the minimum viable agent test. Can this task be accomplished with perception + reasoning + action alone? If yes, ship that. Prove users need memory or iteration before building it.

The

.3M Lesson

Two teams built the same clinical decision support product. One overengineered. One shipped.

Fig 2. The Simpler Agent Won
5 Components vs. 3 Components.
Same Task. Opposite Outcomes.
5-Component Agent

Full-Stack Approach

9 months · $2.3M · Lost the contract
Full memory system. Multi-iteration loops. Every bell, every whistle.
Perception
Reasoning
Action
Memory (cross-visit)
Loop (iterative refinement)
3-Component Agent

Minimum Viable Agent

4 months · Shipped · Physicians chose it
Perceive the case. Reason through guidelines. Recommend action. Done.
Perception
Reasoning
Action
50 cost events per task vs. 3. The simpler system won because it was faster and physicians could see what it did.
Component Necessity Check: 4.5 of 5 components justified — memory earns partial inclusion for firm preferences. Good agents are audited, not assumed.

Remember This

1. Five components: perception, reasoning, action, memory, loop. Most production agents only need three. Add the other two only when the task demands them.

2. PM decision ranking by impact: tools > context > loop duration > data access > model choice. The model is least differentiating in most production systems.

3. The anatomy IS the diagnostic framework. When an agent fails, identify WHICH component failed. Fix the failing component, not the entire system.

References

1. Building Effective Agents -- Anthropic Engineering Blog

2. OpenAI Agents SDK -- OpenAI Documentation

3. Harvey AI -- Harvey AI

4. The Anatomy of an AI Agent -- Anthropic Documentation

Up Next // Topic 24

Architecture Patterns

Five production-tested patterns form an explicit hierarchy. Start with the simplest. Move to complex only when the simpler one demonstrably fails.
Previous Topic Back to the Deep Dive