AI AGENT ENGINEERING

Production AI Agent Architecture: Tools, Permissions, Evals and Control

A production AI agent is not a chatbot with more prompts. It is a software system that can read business context, decide what to do, call tools and sometimes change external state. That means the hard part is not only model quality. The hard part is defining what the agent may see, what it may change, how every action is validated, and what happens when the model is uncertain, slow or wrong.

Updated 2026-09-08 · Engineering guide · Vadym Dmytruk

Start with tool contracts, not prompts

Before choosing a model, define the tools the agent can call. Each tool should have a narrow responsibility, typed inputs, explicit outputs and predictable failure behavior. A “create customer” tool, for example, should not also silently change billing or permissions. Narrow contracts reduce accidental side effects and make failures easier to trace.

Tool contracts should be versioned like APIs. Validate arguments server-side, normalize external errors, enforce timeouts and retries, and capture tool-call telemetry. Prompt instructions can guide behavior, but authorization must live outside the model. The system should remain safe even when the model produces an unexpected call.

Separate read access from write access

An agent that can search documents is fundamentally lower risk than an agent that can send money, modify CRM records or publish content. Treat these capabilities as separate permission classes. Read-only tools can often run automatically, while write tools may require additional checks, budgets or approval.

A useful pattern is a capability policy layer between the model and the tool executor. The policy layer knows the user, tenant, workflow state and risk level. It decides whether the requested action is allowed, requires confirmation or must be rejected. This keeps business policy deterministic instead of burying it in natural-language instructions.

Use structured outputs for every machine decision

Free-form text is fine for explanations. It is weak as a machine interface. Production agents should emit structured outputs whenever another system consumes the result: JSON with a strict schema, explicit enums, bounded arrays and required fields.

Schema validation creates a measurable boundary between model reasoning and application logic. If the output is invalid, the system can retry, switch to a fallback path or escalate to a human. This also makes model upgrades safer because the application depends on a contract rather than a particular writing style.

Evaluate workflows, not demo prompts

A few successful demos do not prove an agent is reliable. Build an evaluation set from representative business tasks, edge cases and known failure modes. Measure tool selection, argument accuracy, policy violations, completion rate, latency and escalation rate.

The strongest evals include both offline tests and production traces. Offline tests protect releases; production traces show what users actually ask. Each important failure should become a regression case so the system gets harder to break over time.

Design explicit human-in-the-loop points

Human review should not be added everywhere or nowhere. Put it where the cost of a wrong action is high: spending money, deleting data, sending external messages, changing permissions or committing contractual information. Low-risk analysis can often remain automatic.

The approval UI should show the proposed action, relevant context and expected effect. A user should be able to approve, edit or reject it. This is more useful than a generic “Are you sure?” dialog because it creates an operational control point and a clear audit trail.

Build for model and provider failure

Models can rate-limit, time out, refuse requests or degrade on certain tasks. Provider abstraction, timeouts and deterministic fallbacks are part of the architecture. The workflow should know when to retry, when to use a smaller model, and when to switch to a non-AI path.

Observability should include model, prompt version, token use, latency, tool calls, validation failures and final outcome. Without these traces, a production incident becomes guesswork. With them, teams can compare providers, tune cost and understand where quality actually breaks.

Explore related engineering services

AI Product DevelopmentAI Agent DevelopmentRAG SystemsMobile App DevelopmentSaaS DevelopmentProduct SecurityWebRTC DevelopmentBackend & Cloud