Scenario Testing

Simulate conversations and evaluate agent behavior with the Scenario testing framework.

Use the Scenario framework for agentic simulation-based testing. Scenario simulates conversations between agents, user simulators, and judges so you can test and evaluate agent behavior in a controlled environment.

Tip: For more on using Scenario with Agno, see the Scenario documentation.

Prerequisites

uv pip install -U agno openai langwatch-scenario pytest pytest-asyncio
export OPENAI_API_KEY=your_openai_api_key

Basic Scenario Testing

scenario_testing.py

import pytest
import scenario
from agno.agent import Agent
from agno.models.openai import OpenAIResponses

# Configure Scenario defaults (model for user simulator and judge)
scenario.configure(default_model="openai/gpt-4.1-mini")

@pytest.mark.agent_test
@pytest.mark.asyncio
async def test_vegetarian_recipe_agent() -> None:
    # 1. Define an AgentAdapter to wrap your agent
    class VegetarianRecipeAgentAdapter(scenario.AgentAdapter):
        agent: Agent

        def __init__(self) -> None:
            self.agent = Agent(
                model=OpenAIResponses(id="gpt-5.2"),
                markdown=True,
                debug_mode=True,
                instructions="You are a vegetarian recipe agent.",
            )

        async def call(self, input: scenario.AgentInput) -> scenario.AgentReturnTypes:
            response = self.agent.run(
                input=input.last_new_user_message_str(), # Pass only the last user message
                session_id=input.thread_id, # Pass the thread id, this allows the agent to track history
            )
            return response.content

    # 2. Run the scenario simulation
    result = await scenario.run(
        name="dinner recipe request",
        description="User is looking for a vegetarian dinner idea.",
        agents=[
            VegetarianRecipeAgentAdapter(),
            scenario.UserSimulatorAgent(),
            scenario.JudgeAgent(
                criteria=[
                    "Agent should not ask more than two follow-up questions",
                    "Agent should generate a recipe",
                    "Recipe should include a list of ingredients",
                    "Recipe should include step-by-step cooking instructions",
                    "Recipe should be vegetarian and not include any sort of meat",
                ]
            ),
        ],
    )

    # 3. Assert and inspect the result
    assert result.success

Usage

See Basic for the full setup steps.