Building Sub-400ms Production RAG Pipelines

Retrieval-Augmented Generation (RAG) is frequently demonstrated in hackathons using naive LangChain splits and vanilla cosine similarity. However, when deployed into regulated enterprise environments across millions of multi-page PDF documents, naive RAG collapses: query latency climbs past 4 seconds, hallucinations proliferate, and context windows choke on irrelevant boilerplate.

In this engineering breakdown, we detail the production RAG architecture engineered by AxonFlow AI powering systems like Chatlly RAG Assistant and Catalyst Bio, achieving a verified 380ms P99 retrieval latency across 2.4M vectorized documents.

1. The Naive RAG Failure Modes at Enterprise Scale

When enterprise users query legal contracts or clinical trial reports, three systemic failures emerge:

• Semantic Blur: Dense vector embeddings (e.g. OpenAI text-embedding-3-small) compress 8,000 tokens into a fixed 1,536-dimensional float vector. Exact keywords like chemical formula codes (e.g., CYP3A4*22) or clause numbers get lost in the semantic average.
• Chunk Boundary Truncation: Fixed 500-character chunking with 50-character overlap frequently severs multi-row financial tables or split sentences across chunk seams.
• Re-ranking Latency Bottlenecks: Cross-encoder re-ranking models (such as BGE-Reranker-Large or Cohere Rerank) provide state-of-the-art accuracy but introduce 700ms–1,200ms of CPU/GPU overhead when fed too many candidates.

2. The Multi-Stage Hybrid Retrieval Pipeline

To break the 400ms barrier, we decouple retrieval into three pipelined phases: fast parallel candidate generation, reciprocal rank fusion (RRF), and targeted cross-encoder scoring:

[User Query]
     │
     ├───► [BM25 Sparse Lexical Search (Qdrant payload index)] ──► Top 40 Candidates
     │
     └───► [Dense Vector HNSW Search (cosine similarity)]      ──► Top 40 Candidates
                                                                          │
                                                                          ▼
                                             [Reciprocal Rank Fusion (k=60)]
                                                                          │
                                                                          ▼
                                                         Top 18 Unique Chunks
                                                                          │
                                                                          ▼
                                           [Quantized Flash-Reranker ONNX (18ms)]
                                                                          │
                                                                          ▼
                                                       Top 5 Context Chunks
                                                                          │
                                                                          ▼
                                                     [Streaming LLM Generation]

3. High-Throughput Python Asynchronous Retrieval Implementation

Below is a production extract from our core RAG retrieval worker utilizing asynchronous concurrency over Qdrant and fast ONNX runtime re-ranking:

import asyncio
from qdrant_client import AsyncQdrantClient
from qdrant_client.http import models

class EnterpriseRetriever:
    def __init__(self, qdrant_url: str, collection: str):
        self.client = AsyncQdrantClient(url=qdrant_url, timeout=2.0)
        self.collection = collection

    async def hybrid_retrieve(self, query_text: str, dense_vector: list[float], limit: int = 5):
        # Fire sparse lexical and dense semantic queries in true parallel
        sparse_task = self.client.scroll(
            collection_name=self.collection,
            scroll_filter=models.Filter(
                must=[models.FieldCondition(key="content", match=models.MatchText(text=query_text))]
            ),
            limit=40,
            with_payload=True
        )
        
        dense_task = self.client.search(
            collection_name=self.collection,
            query_vector=dense_vector,
            limit=40,
            with_payload=True
        )

        sparse_res, dense_res = await asyncio.gather(sparse_task, dense_task)
        
        # Reciprocal Rank Fusion (RRF) calculation
        scores = {}
        k = 60.0
        
        for rank, hit in enumerate(sparse_res[0]):
            scores[hit.id] = scores.get(hit.id, 0.0) + (1.0 / (k + rank + 1))
            
        for rank, hit in enumerate(dense_res):
            scores[hit.id] = scores.get(hit.id, 0.0) + (1.0 / (k + rank + 1))
            
        ranked_ids = sorted(scores.keys(), key=lambda x: scores[x], reverse=True)[:limit]
        return ranked_ids

4. Chunking Tables and Hierarchical Markdown

Instead of arbitrary token slicing, our document parser utilizes AST-based markdown chunking. Headers (#, ##, ###) are preserved as metadata paths prepended to each chunk. HTML and Markdown tables are never sliced; they are converted into compact JSON records and attached to single parent chunks with contextual breadcrumbs.

5. Benchmarked Results Across 2.4M Production Embeddings