vLLM Embedder

Generate embeddings with vLLM, either in-process or against a vLLM server.

VLLMEmbedder runs in two modes. Local mode loads the model in-process with no server or API key. Remote mode connects to a running vLLM server through its OpenAI-compatible API, selected by setting base_url.

vllm_embedder.py

from agno.knowledge.embedder.vllm import VLLMEmbedder

# Local mode: vLLM loads the model in-process
embedder = VLLMEmbedder(
    id="sentence-transformers/all-MiniLM-L6-v2",
    dimensions=384,
    enforce_eager=True,
    vllm_kwargs={
        "disable_sliding_window": True,
        "max_model_len": 256,
    },
)
embedding = embedder.get_embedding("The quick brown fox jumps over the lazy dog.")

print(embedding[:5])
print(len(embedding))

# Remote mode: connect to a running vLLM server
remote_embedder = VLLMEmbedder(
    id="sentence-transformers/all-MiniLM-L6-v2",
    dimensions=384,
    base_url="http://localhost:8000/v1",
    api_key="your-api-key",  # optional, also read from VLLM_API_KEY
)

Local mode downloads the model from Hugging Face on first use. Larger models need matching GPU memory; intfloat/e5-mistral-7b-instruct (4096 dimensions) needs roughly 14GB of VRAM.

Run the Example

1. Set up your virtual environment

   Mac

   uv venv --python 3.12
   source .venv/bin/activate
   

   uv venv --python 3.12
   .venv\Scripts\activate
   
2. Install dependencies

   uv pip install -U agno vllm openai
   
3. Run the example

   python vllm_embedder.py
   

Developer Resources

• VLLMEmbedder reference
• vLLM OpenAI-compatible server
• Embedders overview