Website Tools - Web Scraping and Content Analysis

Scrape and summarize a web page with WebsiteTools, which registers its read_url function when no knowledge base is attached.

Use WebsiteTools to let an agent fetch and summarize a page. With no knowledge base attached, the toolkit registers one read_url function.

website_tools.py

"""
Website Tools - Web Scraping and Content Analysis

This example demonstrates how to use WebsiteTools for web scraping and analysis.
Shows enable_ flag patterns for selective function access.
WebsiteTools is a small tool (<6 functions) so it uses enable_ flags.
"""

from agno.agent import Agent
from agno.tools.website import WebsiteTools

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------


agent = Agent(
    tools=[WebsiteTools()],  # All functions enabled by default
    description="You are a comprehensive web scraping specialist with all website analysis capabilities.",
    instructions=[
        "Help users scrape and analyze website content",
        "Provide detailed summaries and insights from web pages",
        "Handle various website formats and structures",
        "Ensure respectful scraping practices",
    ],
    markdown=True,
)

# Example usage

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    print("=== Basic Web Content Search Example ===")
    agent.print_response(
        "Search web page: 'https://docs.agno.com/introduction' and summarize the key concepts",
        markdown=True,
    )

Run the Example

1. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   

   uv venv --python 3.12
   .venv\Scripts\activate
   
2. Install dependencies

   uv pip install -U agno beautifulsoup4 openai
   
3. Export your OpenAI API key

   Mac/Linux

   export OPENAI_API_KEY="your_openai_api_key_here"
   

   $Env:OPENAI_API_KEY="your_openai_api_key_here"
   
4. Run the example

   Save the code above as website_tools.py, then run:

   python website_tools.py
   

Full source: cookbook/91_tools/website_tools.py