> ## Documentation Index
> Fetch the complete documentation index at: https://phidatainc.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Observability

> Trace agent, team, and workflow runs across models, tools, and steps.

A production agent run can cross models, tools, team members, and workflow steps. Platform teams use traces to explain an unexpected answer, locate a slow call, and follow a failure to its source. AgentOS instruments those runs with OpenTelemetry, stores trace data in your configured database, and renders the same trace tree in the Control Plane.

```python theme={null}
from agno.os import AgentOS

agent_os = AgentOS(
    agents=[agent],
    db=db,
    tracing=True,
)
```

`tracing=True` instruments supported agent, team, and workflow operations. Each instrumented run produces spans for model calls, tool executions, team coordination, and workflow steps. AgentOS writes the aggregate trace to `agno_traces` and individual spans to `agno_spans`.

## Where trace data goes

The built-in exporter writes trace records to the database you configure:

| Configuration                                    | Trace destination                                         |
| ------------------------------------------------ | --------------------------------------------------------- |
| `AgentOS(db=trace_db, tracing=True)`             | `trace_db`                                                |
| `AgentOS(tracing=True)` with component databases | First database found on a native agent, team, or workflow |
| `AgentOS(tracing=True)` with no database         | Tracing is skipped and AgentOS logs a warning             |
| Custom OpenTelemetry exporter                    | The exporter destination you configure                    |

Tracing does not change where models and tools send data. An agent can still call external model providers, tools, or exporters. Trace attributes can contain prompts, tool arguments, and model output, so apply the same access and retention controls you use for other sensitive application data.

<Frame caption="Traces stored in your own database, viewed in a SQL client">
  <img src="https://mintcdn.com/phidatainc/JUdd69R_GtzOgSkJ/images/traces-in-db.png?fit=max&auto=format&n=JUdd69R_GtzOgSkJ&q=85&s=529b890c9d21fd1bc9d99138d0584b01" alt="agno_traces and agno_spans tables in a database client" style={{ borderRadius: "0.5rem" }} width="2038" height="480" data-path="images/traces-in-db.png" />
</Frame>

## What gets captured

Each record in `agno_spans` stores the span name, parent, status, timestamps, duration, and an `attributes` JSON object. The aggregate row in `agno_traces` stores the run, session, user, and component IDs when present, plus overall status, start and end times, and duration.

Traces follow OpenInference semantic conventions, so you can query them directly:

```sql theme={null}
-- Top 10 slowest span types by average duration
SELECT
    name,
    AVG(duration_ms) AS avg_ms,
    COUNT(*) AS calls
FROM ai.agno_spans
GROUP BY name
ORDER BY avg_ms DESC
LIMIT 10;
```

`PostgresDb` creates its tables in the `ai` schema by default (override with `PostgresDb(db_schema=...)`). Qualify the table as `ai.agno_spans` or add `ai` to your `search_path`.

## In the AgentOS UI

The [control plane](https://os.agno.com) renders the same traces visually. Click a run to see the full tree: LLM hops, tool calls with their inputs and outputs, and sub-agent traces. Filter by user, session, or time range.

<Frame caption="The same traces, rendered in the AgentOS control plane">
  <img src="https://mintcdn.com/phidatainc/JUdd69R_GtzOgSkJ/images/traces-in-os.png?fit=max&auto=format&n=JUdd69R_GtzOgSkJ&q=85&s=6db05953914dad56d8183af285f8f3fc" alt="Trace tree in the AgentOS UI" style={{ borderRadius: "0.5rem" }} width="2932" height="1314" data-path="images/traces-in-os.png" />
</Frame>

## Multi-database tracing

Traces are high-volume and write-heavy, with a different cost profile, retention, and access pattern than sessions. For production, route them to a dedicated database by pointing the AgentOS `db` at it:

```python theme={null}
from agno.agent import Agent
from agno.db.postgres import PostgresDb
from agno.models.openai import OpenAIResponses
from agno.os import AgentOS

# Each agent keeps its own database
agent_db = PostgresDb(db_url="postgresql+psycopg://primary/...")

# Dedicated database for traces
trace_db = PostgresDb(db_url="postgresql+psycopg://traces/...")

agent = Agent(name="Research Agent", model=OpenAIResponses(id="gpt-5.2"), db=agent_db)

agent_os = AgentOS(
    agents=[agent],
    db=trace_db,   # All traces are written here
    tracing=True,
)
```

The AgentOS `db` is where traces land. Keeping it separate isolates trace write volume from your agent data and gives you independent retention and scaling.

See [Multi-DB tracing](/agent-os/tracing/usage/tracing-with-multi-db-scenario) for the `setup_tracing()` variant with batch tuning.

## External providers

You can configure OpenTelemetry exporters for Langfuse, LangSmith, Arize, Logfire, MLflow, The Context Company, or another OpenTelemetry endpoint.

See the [Observability section](/observability/overview): [Langfuse](/observability/langfuse), [LangSmith](/observability/langsmith), [Arize](/observability/arize), [Logfire](/examples/integrations/observability/logfire-via-openinference), [MLflow](/observability/mlflow), [The Context Company](/observability/the-context-company).

## Developer Resources

* [Tracing overview](/tracing/overview)
* [Tracing in AgentOS](/agent-os/tracing/overview)
* [Query traces from your database](/tracing/db-functions)