LLMs.txt Tools with Knowledge Base

Bulk-load every page linked from an llms.txt index into a PgVector knowledge base, then answer questions with semantic search.

Demonstrates loading all documentation from an llms.txt file into a knowledge base for retrieval-augmented generation (RAG).

llms_txt_tools_knowledge.py

"""
LLMs.txt Tools with Knowledge Base
=============================

Demonstrates loading all documentation from an llms.txt file into a knowledge base
for retrieval-augmented generation (RAG).

The agent reads the llms.txt index, fetches all linked documentation pages,
and stores them in a PgVector knowledge base for semantic search.
"""

from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.models.openai import OpenAIResponses
from agno.tools.llms_txt import LLMsTxtTools
from agno.vectordb.pgvector import PgVector

# ---------------------------------------------------------------------------
# Setup Knowledge Base
# ---------------------------------------------------------------------------

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    vector_db=PgVector(
        table_name="llms_txt_docs",
        db_url=db_url,
    ),
)

# ---------------------------------------------------------------------------
# Create Agent
# ---------------------------------------------------------------------------

agent = Agent(
    model=OpenAIResponses(id="gpt-5.4"),
    knowledge=knowledge,
    search_knowledge=True,
    tools=[LLMsTxtTools(knowledge=knowledge, max_urls=20)],
    instructions=[
        "You can load documentation from llms.txt files into your knowledge base.",
        "When asked about a project, first load its llms.txt into the knowledge base, then answer questions.",
    ],
    markdown=True,
)

# ---------------------------------------------------------------------------
# Run Agent
# ---------------------------------------------------------------------------
if __name__ == "__main__":
    agent.print_response(
        "Load the documentation from https://docs.agno.com/llms.txt into the knowledge base, "
        "then tell me how to create an agent with Agno",
        markdown=True,
        stream=True,
    )

Run the Example

1. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   

   uv venv --python 3.12
   .venv\Scripts\activate
   
2. Install dependencies

   uv pip install -U agno "psycopg[binary]" beautifulsoup4 openai pgvector sqlalchemy
   
3. Export your OpenAI API key

   Mac/Linux

   export OPENAI_API_KEY="your_openai_api_key_here"
   

   $Env:OPENAI_API_KEY="your_openai_api_key_here"
   
4. Run PgVector

   macOS / Linux

   docker run -d \
     -e POSTGRES_DB=ai \
     -e POSTGRES_USER=ai \
     -e POSTGRES_PASSWORD=ai \
     -e PGDATA=/var/lib/postgresql \
     -v pgvolume:/var/lib/postgresql \
     -p 5532:5432 \
     --name pgvector \
     agnohq/pgvector:18
   

   docker run -d `
     -e POSTGRES_DB=ai `
     -e POSTGRES_USER=ai `
     -e POSTGRES_PASSWORD=ai `
     -e PGDATA=/var/lib/postgresql `
     -v pgvolume:/var/lib/postgresql `
     -p 5532:5432 `
     --name pgvector `
     agnohq/pgvector:18
   
5. Run the example

   Save the code above as llms_txt_tools_knowledge.py, then run:

   python llms_txt_tools_knowledge.py
   

Full source: cookbook/91_tools/llms_txt_tools_knowledge.py