Agent Evaluation

Catch regressions in response quality, tool use, latency, and memory.

Changes to models, instructions, tools, and knowledge can introduce regressions. Agno evals turn response criteria and expected tool use into executable cases. Run them during development, gate CI with their exit code, and evaluate selected production outputs through hooks.

evals.py

import sys

from agno.agent import Agent
from agno.eval import Case, cli
from agno.tools.calculator import CalculatorTools

calculator = Agent(
    id="calculator",
    model="openai:gpt-5.5",
    tools=[CalculatorTools()],
    instructions="Use the calculator tools for every calculation.",
)

CASES = (
    Case(
        name="factorial_uses_calculator",
        agent=calculator,
        input="What is 10 factorial?",
        criteria="States that 10 factorial equals 3,628,800.",
        expected_tool_calls=("factorial",),
    ),
)

if __name__ == "__main__":
    sys.exit(cli(CASES))

Create a virtual environment, install the OpenAI integration, and set OPENAI_API_KEY before running the suite:

uv venv --python 3.12
uv pip install -U "agno[openai]"

uv run python evals.py --json-output tmp/evals.json

Each case runs the component once and applies the configured judge and reliability checks to the same output. The CLI returns a nonzero exit code when a case fails, so the suite can gate CI.

Evaluation types

Type	Measures	Guide
Accuracy	Correctness against an expected answer	Accuracy evals
Agent as judge	Custom quality criteria scored by an evaluator model	Agent-as-judge evals
Reliability	Expected tool calls and arguments	Reliability evals
Performance	Runtime latency and memory use	Performance evals

Where evals run

Stage	Pattern
Local development	Run one case while changing an agent.
CI	Run tagged eval suites and keep the JSON report.
Production	Evaluate selected outputs with a synchronous or background post-hook.
AgentOS	Store eval results in a configured database and manage them through the AgentOS API.

Next steps

Task	Guide
Build an eval suite	Eval suites
Add evals to an agent platform	Agent platform evals
Inspect the API surface	Agent API