> ## Documentation Index
> Fetch the complete documentation index at: https://phidatainc.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Evals

> Lock in agent behavior with regression tests.

Evals are regression tests for your agents. Rerun the same prompts against the same agents and behavior drift becomes visible.

`/improve-agent` generates probes from an agent's instructions to find new weaknesses. Evals preserve known behavior as repeatable cases.

## Cases

Cases live in `evals/cases.py`. Each case sends one input to an agent (`agent=`) and optionally checks two things:

* **judge**: `AgentAsJudgeEval` scores the response against `criteria` (binary pass/fail) using an LLM.
* **reliability**: `ReliabilityEval` checks which tools fired against `expected_tool_calls`.

A case looks like this:

```python evals/cases.py theme={null}
from os import getenv

# The WebSearch agent calls parallel_search when PARALLEL_API_KEY is set, web_search otherwise.
_WEB_SEARCH_TOOL = "parallel_search" if getenv("PARALLEL_API_KEY") else "web_search"

CASES: tuple[Case, ...] = (
    Case(
        name="web_search_recent_anthropic_research",
        agent=web_search,
        input="What did Anthropic publish about agent research recently?",
        tags=("live",),
        timeout_seconds=120,
        criteria=(
            "Answers the question by citing at least one real Anthropic URL "
            "(anthropic.com domain). The response is grounded in fetched content "
            "rather than refusing to answer."
        ),
        expected_tool_calls=(_WEB_SEARCH_TOOL,),
    ),
    # add more cases here
)
```

A case can use either check or both. If both are set, the agent runs once and feeds the same response into both.

Add tags to group your cases into suites. The template uses three tags: `smoke`, `release`, and `live`. This case uses the `live` tag because its answer depends on the open web.

## Run the suite

The suite runs on the host, calls the model, and logs results to your local Postgres through `eval_db`. Start the platform first (`docker compose up -d`) and make sure `.env` has your `OPENAI_API_KEY`.

<Steps>
  <Step title="Create a virtual environment">
    The eval suite runs on the host and needs a local virtual environment:

    ```bash theme={null}
    ./scripts/venv_setup.sh
    ```

    Activate it:

    ```bash theme={null}
    source .venv/bin/activate
    ```
  </Step>

  <Step title="Run the eval suite">
    ```bash theme={null}
    python -m evals --tag smoke     # fast suite
    python -m evals                # full suite
    ```

    Other options:

    ```bash theme={null}
    python -m evals -v             # stream the agent run with full panels
    python -m evals --name <case>  # single case while iterating
    ```
  </Step>
</Steps>

Each case prints its response and the verdicts for the checks it defines. The run ends with an `Eval Summary` table.

Results write to Postgres via `eval_db`. The eval history shows up on [os.agno.com](https://os.agno.com) alongside your sessions and traces, so you can see when a case started failing and what changed.

## Diagnose failures with your coding agent

Run `/create-evals` to add coverage for an agent. The skill maps the agent's behavior, proposes cases, writes them to `evals/cases.py`, and verifies the new cases.

Open your coding agent and run:

```text theme={null}
/eval-and-improve
```

The coding agent runs the suite, triages every failure (bad criteria, real regression, flaky LLM judge), and proposes in-scope fixes. It edits the agent or the case and re-runs until the suite is green.

## When to run evals

| Trigger                               | Frequency       |
| ------------------------------------- | --------------- |
| Before deploying a change to an agent | Every time      |
| As part of CI                         | Every PR        |
| Against production                    | On a daily cron |
| After bumping a model version         | Every time      |

The template registers a daily `run-evals` schedule in the disabled state because it uses model calls. Enable it from the AgentOS UI when you want the `smoke`-tagged cases to run daily. See [Scheduler](/agent-os/scheduler/overview) for the cron API.

## What good cases look like

* **Specific.** "Returns a JSON object with `ticker` and `price`" beats "Returns the right answer".
* **Stable.** Avoid prompts whose correct answer changes daily. Use phrasing like "describes a real, recent..." instead of locking in a specific result.
* **Scoped to one behavior.** One case per behavior makes failures easy to read.
* **Anchored to tools.** `expected_tool_calls` catches the failure mode where the agent confidently makes things up instead of calling a tool.

## Next

[Next steps →](/agent-platform/next-steps)