LLMs.txt Reader

Fetch an llms.txt index and convert its linked documentation pages into documents.

LLMsTxtReader creates a document for an llms.txt overview, then fetches up to max_urls linked pages.

llms_txt_reader.py

from agno.agent import Agent
from agno.knowledge.knowledge import Knowledge
from agno.knowledge.reader.llms_txt_reader import LLMsTxtReader
from agno.models.openai import OpenAIResponses
from agno.vectordb.pgvector import PgVector

db_url = "postgresql+psycopg://ai:ai@localhost:5532/ai"

knowledge = Knowledge(
    name="LLMs.txt Docs",
    vector_db=PgVector(table_name="llms_txt_docs", db_url=db_url),
)

agent = Agent(
    model=OpenAIResponses(id="gpt-5.2"),
    knowledge=knowledge,
    search_knowledge=True,
)

if __name__ == "__main__":
    knowledge.insert(
        url="https://docs.agno.com/llms.txt",
        reader=LLMsTxtReader(
            max_urls=10,
            allowed_hosts=["docs.agno.com"],
        ),
    )
    agent.print_response("What is Agno?", markdown=True)

Run the Agent

1. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   

   uv venv --python 3.12
   .venv\Scripts\activate
   
2. Install dependencies

   uv pip install -U agno beautifulsoup4 openai pgvector psycopg sqlalchemy
   
3. Export the API key

   export OPENAI_API_KEY=your_openai_api_key_here
   
4. Run PgVector

   macOS / Linux

   docker run -d \
     -e POSTGRES_DB=ai \
     -e POSTGRES_USER=ai \
     -e POSTGRES_PASSWORD=ai \
     -e PGDATA=/var/lib/postgresql \
     -v pgvolume:/var/lib/postgresql \
     -p 5532:5432 \
     --name pgvector \
     agnohq/pgvector:18
   

   docker run -d `
     -e POSTGRES_DB=ai `
     -e POSTGRES_USER=ai `
     -e POSTGRES_PASSWORD=ai `
     -e PGDATA=/var/lib/postgresql `
     -v pgvolume:/var/lib/postgresql `
     -p 5532:5432 `
     --name pgvector `
     agnohq/pgvector:18
   
5. Run the agent

   python llms_txt_reader.py
   

Reader Parameters

Parameter	Type	Default	Description
max_urls	int	20	Maximum number of linked URLs to fetch from the file
timeout	int	60	HTTP timeout in seconds
proxy	Optional[str]	None	Optional HTTP proxy URL
skip_optional	bool	False	Skip entries under the ## Optional section
chunking_strategy	Optional[ChunkingStrategy]	FixedSizeChunking()	Strategy for chunking content
allowed_hosts	Optional[List[str]]	None	Hostnames the reader is allowed to fetch from. See Restricting URL Fetches.

LLMsTxtReader also accepts the base Reader constructor parameters.

allowed_hosts applies to the index, linked pages, and redirects. LLMsTxtReader.async_read() fetches the linked pages concurrently.

Next Steps

Task	Guide
Restrict outbound requests	Restricting URL Fetches
Crawl pages without an llms.txt index	Website Reader
Configure content chunking	Chunking Overview