Web Search Reader

WebSearchReader searches the web via DuckDuckGo and reads result pages into documents.

WebSearchReader reads data from web search results.

Parameters

Parameter	Type	Default	Description
chunk	bool	True	Whether to chunk the documents
chunk_size	int	5000	Size of each chunk when chunking is enabled
encoding	Optional[str]	None	Text encoding used when reading content
search_timeout	int	10	Timeout for search operations in seconds
request_timeout	int	30	Timeout for HTTP requests in seconds
delay_between_requests	float	2.0	Delay between requests in seconds
max_retries	int	3	Maximum number of retries for failed requests
user_agent	str	"Mozilla/5.0..."	User agent string for HTTP requests
search_engine	Literal["duckduckgo"]	"duckduckgo"	Search engine to use. DuckDuckGo is the only supported engine
search_delay	float	3.0	Delay between search requests in seconds
max_search_retries	int	2	Maximum retries for search operations
rate_limit_delay	float	5.0	Delay when rate limited in seconds
exponential_backoff	bool	True	Whether to use exponential backoff for retries
chunking_strategy	Optional[ChunkingStrategy]	None	Strategy for chunking content. When None, uses SemanticChunking with the reader's chunk_size
allowed_hosts	Optional[List[str]]	None	Hostnames the reader is allowed to fetch from. See Restricting URL Fetches.

WebSearchReader also accepts the base Reader constructor parameters, including max_results for the number of search results to read.

Methods

read()

Searches the web for the query (up to max_results search results), reads each result page, and returns a list of documents.

Parameter	Type	Default	Description
query	str	Required	Search query to run

async_read()

Async variant of read(). Accepts the same parameters.