> ## Documentation Index
> Fetch the complete documentation index at: https://phidatainc.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Human routing and eval

> Route low-confidence fields for approval and track extraction accuracy against a golden set.

Document processing pipelines need a review path for low-confidence fields and a way to measure extraction quality over time. Add both to the same extraction agent.

## Per-field confidence

Wrap each field in a confidence carrier so a downstream check can decide what needs review. The schema is identical to the one in [data labeling](/use-cases/data-labeling/structured-extraction#per-field-confidence); the routing logic is the part that lives here.

```python theme={null}
from typing import Literal, Optional

from agno.agent import Agent
from agno.media import File
from agno.models.openai import OpenAIResponses
from pydantic import BaseModel


Confidence = Literal["high", "medium", "low"]


class ConfidentField(BaseModel):
    value: Optional[str] = None
    confidence: Confidence


class Invoice(BaseModel):
    invoice_number: ConfidentField
    vendor: ConfidentField
    invoice_date: ConfidentField
    total: ConfidentField


agent = Agent(
    model=OpenAIResponses(id="gpt-5.5"),
    instructions=(
        "Extract invoice fields. For each field, report confidence: "
        "high (explicit on the document), medium (inferred from structure), "
        "low (guessed, partly obscured, or ambiguous). Be conservative."
    ),
    output_schema=Invoice,
)

invoice = agent.run(
    "Extract this invoice.",
    files=[File(filepath="scan-low-quality.pdf")],
).content
# Invoice(invoice_number=ConfidentField(value='1042', confidence='high'),
#         vendor=ConfidentField(value='Acme Corp', confidence='high'),
#         invoice_date=ConfidentField(value=None, confidence='low'),
#         total=ConfidentField(value='1296.0', confidence='medium'))
```

## Route on low confidence

Walk the extracted fields, find values below the confidence threshold, and route the document in application code.

```python theme={null}
def low_confidence_fields(invoice: Invoice) -> list[str]:
    return [
        name
        for name, field in invoice.model_dump().items()
        if field.get("confidence") == "low"
    ]


flagged = low_confidence_fields(invoice)
if flagged:
    send_to_human_queue(invoice, flagged)
else:
    write_to_database(invoice)
```

Treat confidence as an agent-provided routing signal. Application code sets the threshold and chooses the action.

## Gate the next action with `requires_confirmation`

Wrap a downstream action, such as a database write or ERP push, in a tool that requires approval. Every call pauses the run until a human confirms it. This places approval at the system boundary.

```python theme={null}
from agno.agent import Agent
from agno.db.sqlite import SqliteDb
from agno.models.openai import OpenAIResponses
from agno.tools import tool


@tool(requires_confirmation=True)
def post_to_erp(invoice_id: str, vendor: str, total: float) -> str:
    """Post an extracted invoice to the AP ledger."""
    # ...real ERP call...
    return f"Posted {invoice_id} for {vendor}: {total}"


db = SqliteDb(db_file="tmp/extraction.db")

writer = Agent(
    model=OpenAIResponses(id="gpt-5.5"),
    tools=[post_to_erp],
    db=db,
    instructions="Given a parsed invoice, post it to the ERP with post_to_erp.",
)

run = writer.run(
    f"Post this invoice: {invoice.model_dump_json()}"
)

if run.is_paused:
    for requirement in run.active_requirements:
        if requirement.needs_confirmation:
            # Surface this to a reviewer UI; here we approve directly.
            print(f"Approve: {requirement.tool_execution.tool_name}")
            requirement.confirm()

    run = writer.continue_run(
        run_id=run.run_id,
        session_id=run.session_id,
        requirements=run.requirements,
    )
```

The pause is persisted in `db`. A different process can reconstruct the same Agent configuration and continue with the stored `run_id` and `session_id`. See [human approval](/hitl/approval) for async variants and listing pending approvals from the database.

## Accuracy against a golden set

A representative golden set compares extraction behavior across prompt or model changes.

```python theme={null}
from agno.agent import Agent
from agno.db.sqlite import SqliteDb
from agno.eval.accuracy import AccuracyEval
from agno.media import File
from agno.models.openai import OpenAIResponses

db = SqliteDb(db_file="tmp/extraction.db")

agent = Agent(
    model=OpenAIResponses(id="gpt-5.5"),
    instructions="Extract invoice fields. Null if missing.",
    output_schema=Invoice,
)

run = agent.run(
    "Extract this invoice.",
    files=[File(filepath="golden/invoice-001.pdf")],
)

evaluation = AccuracyEval(
    db=db,
    name="invoice-extraction-golden",
    model=OpenAIResponses(id="gpt-5.5"),
    input="Extract this invoice.",
    expected_output=(
        "Invoice number 1042, vendor Acme Corp, dated 2026-04-12, "
        "total 1296.00 USD."
    ),
)

result = evaluation.run_with_output(
    output=run.content.model_dump_json(), print_results=True
)
# AccuracyResult(avg_score=9.0, ...)
assert result is not None
print(result.avg_score)
```

`AccuracyEval.run()` executes its configured agent but does not accept document files. Run document extraction first, then pass its serialized output to `run_with_output`. `AccuracyEval` uses a model judge to compare that output with `expected_output`; it is a semantic score, not an exact field-by-field metric.

```python theme={null}
results = []
for doc in golden_set:
    run = agent.run("Extract this invoice.", files=[File(filepath=doc.path)])
    eval_ = AccuracyEval(
        db=db,
        name=f"invoice-{doc.id}",
        model=OpenAIResponses(id="gpt-5.5"),
        input="Extract this invoice.",
        expected_output=doc.expected_description,
    )
    results.append(
        eval_.run_with_output(
            output=run.content.model_dump_json(), print_results=False, print_summary=False
        )
    )
```

Passing `db=db` stores each evaluation result. Compare model-judge scores across repeated runs with a fixed judge configuration, and add deterministic field checks for invoice numbers, dates, totals, and line items. See the [evals cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/09_evals/accuracy) for database logging and the team variant.

## Review controls

| Pattern                        | What it answers                                 | When it fires                                    |
| ------------------------------ | ----------------------------------------------- | ------------------------------------------------ |
| Confidence routing             | "Which fields on this document need a human?"   | Every run, per document                          |
| Approval-gated tools           | "Should we let the agent take the next action?" | At a specific tool boundary                      |
| AccuracyEval over a golden set | "How does a model judge compare these outputs?" | After a prompt or model change, or on a schedule |

Confidence routing and approval-gated tools act on one document. `AccuracyEval` records a model-judge signal for comparing configurations.

## Next steps

| Task                             | Guide                                                                       |
| -------------------------------- | --------------------------------------------------------------------------- |
| Schedule the eval to run nightly | [Batch and durability](/use-cases/document-processing/batch-and-durability) |
| Approve from an external UI      | [Human approval](/hitl/approval)                                            |
| Add a two-labeler review step    | [Quality pipeline](/use-cases/data-labeling/quality-pipeline)               |

## Developer Resources

* [Approval cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/02_agents/11_approvals)
* [Accuracy eval cookbook](https://github.com/agno-agi/agno/tree/main/cookbook/09_evals/accuracy)
* [Human approval](/hitl/overview)