Retrieval-Augmented Generation (RAG) is the gold standard for empowering LLMs with domain-specific enterprise data. However, default naive RAG architectures suffer from slow latency (2-5 seconds) and poor relevance. In this post, we breakdown the technical optimizations used by AxonFlow AI to achieve under 400ms end-to-end response times.
1. The Latency Bottleneck Breakdown
A standard RAG query passes through four main stages:
- Embedding Query (50ms): Converting user text into vector representations.
- Vector Search (40ms): Performing HNSW distance calculations against millions of document vectors in Qdrant.
- Re-ranking (120ms): Passing top 50 matches through a cross-encoder (Cohere Rerank v3) for semantic precision.
- LLM Generation (150ms to first token): Streaming response tokens to the client frontend over Server-Sent Events (SSE).
2. Optimized Python Retrieval Middleware
from qdrant_client import QdrantClient
import cohere
qdrant = QdrantClient(url="https://qdrant.axonflow.in", api_key="...")
co = cohere.Client("...")
async def hybrid_search(query_text: str, top_k: int = 5):
# 1. Vector Search
query_vector = await embed_fast(query_text)
hits = qdrant.search(
collection_name="enterprise_kb",
query_vector=query_vector,
limit=30
)
# 2. Cross-Encoder Re-Ranking
documents = [hit.payload['text'] for hit in hits]
rerank_results = co.rerank(
query=query_text,
documents=documents,
top_n=top_k,
model="rerank-english-v3.0"
)
return [documents[r.index] for r in rerank_results.results]
3. Key Production Takeaways
By shifting to hybrid search (dense vectors + sparse BM25 indices) and streaming response tokens immediately via SSE, we reduced perceive latency from 3.2 seconds down to 380ms for enterprise healthcare and legal deployments.