Naive Retrieval-Augmented Generation (RAG) implementation—converting text chunks into dense vector embeddings and performing a top-K cosine similarity lookup—frequently fails in production. Standard dense vector retrieval suffers from distinct failure modes: it struggles with exact keyword matching (part numbers, technical jargon, error codes), fails to handle sparse domain-specific nomenclature, and often returns semantically similar but contextually irrelevant chunks that dilute the Language Model’s (LLM) context window.

To deliver enterprise-grade accuracy with sub-300ms retrieval SLAs, production architectures must evolve from single-vector lookup models into a multi-stage hybrid retrieval pipeline. This guide breaks down the architectural blueprint for building a resilient RAG pipeline that combines sparse full-text search (BM25 / PostgreSQL tsvector), dense vector search (pgvector with HNSW indices), Reciprocal Rank Fusion (RRF), and Cross-Encoder re-ranking models.

The Three-Stage Retrieval Architecture

High-precision retrieval requires progressive filtering: broad, fast candidate generation followed by compute-intensive contextual precision scoring.

┌────────────────────────────────────────────────────────────────────────┐
│                        User Query / Prompt                             │
└───────────────────────────────────┬────────────────────────────────────┘
                                    │
                                    ▼
       ┌────────────────────────────┴────────────────────────────┐
       │             Parallel Candidate Retrieval                │
       ├────────────────────────────┬────────────────────────────┤
       │   Dense Vector Search      │    Sparse Keyword Search   │
       │  (pgvector / OpenAI)       │     (BM25 / tsvector)      │
       └─────────────┬──────────────┴──────────────┬─────────────┘
                     │ Rank Top 50                 │ Rank Top 50
                     └──────────────┬──────────────┘
                                    │
                                    ▼
       ┌─────────────────────────────────────────────────────────┐
       │     Stage 2: Reciprocal Rank Fusion (RRF) Merge         │
       │            Unified Candidate Set (Top 30)               │
       └────────────────────────────┬────────────────────────────┘
                                    │
                                    ▼
       ┌─────────────────────────────────────────────────────────┐
       │     Stage 3: Cross-Encoder Re-Ranking (Cohere / local)  │
       │           Precision Context Scoring (Top 5)             │
       └────────────────────────────┬────────────────────────────┘
                                    │
                                    ▼
       ┌─────────────────────────────────────────────────────────┐
       │             LLM Context Window Injection                │
       └─────────────────────────────────────────────────────────┘

Stage 1: Parallel Candidate Generation

The pipeline splits incoming user queries into two concurrent execution paths:

1. Dense Retrieval: Generates vector embeddings using models like text-embedding-3-large or bge-large-en-v1.5 and queries a vector database to capture high-level semantic intent.
2. Sparse Retrieval: Uses standard BM25 algorithms or PostgreSQL full-text search engines to match exact lexical tokens, numerical identifiers, and proper nouns.

Stage 2: Reciprocal Rank Fusion (RRF)

Vector distances and BM25 scores operate on vastly different numerical scales and distributions. Instead of attempting to normalize raw scores, Reciprocal Rank Fusion combines the results strictly based on their positional ranks:

$$RRF_Score(d \in D) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$

Where $M$ represents the retrieval systems (dense and sparse), $r_m(d)$ is the rank of document $d$ in system $m$, and $k$ is a smoothing constant (typically set to $60$).

Stage 3: Cross-Encoder Re-Ranking

While bi-encoders produce static embeddings for fast vector lookup, they compress document meaning into fixed dimensions independent of the query. A Cross-Encoder processes the user query and candidate chunk jointly through deep attention layers, calculating an absolute relevancy score. Because Cross-Encoders are computationally expensive, we execute them only against the top 20–30 candidate chunks isolated by RRF.

Database Architecture: Hybrid PostgreSQL Schema

PostgreSQL equipped with pgvector allows teams to consolidate vector storage, full-text indexes, and business data inside a single ACID-compliant database.

-- Enable vector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Document chunks table with hybrid index support
CREATE TABLE document_chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    document_id UUID NOT NULL,
    tenant_id UUID NOT NULL,
    content TEXT NOT NULL,
    metadata JSONB DEFAULT '{}'::jsonb,
    -- Full-Text Search Vector
    tsv TSVECTOR GENERATED ALWAYS AS (to_tsvector('english', content)) STORED,
    -- Dense Embedding Vector (OpenAI text-embedding-3-small default size)
    embedding VECTOR(1536) NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW()
);

-- HNSW Index for ultra-fast dense similarity lookup
CREATE INDEX idx_chunks_embedding_hnsw
ON document_chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- GIN Index for rapid sparse lexical lookup
CREATE INDEX idx_chunks_tsv ON document_chunks USING gin(tsv);

-- Tenant Isolation Composite Index
CREATE INDEX idx_chunks_tenant ON document_chunks(tenant_id);

Hybrid RRF Execution Function

This stored SQL function performs parallel dense and sparse searches for a specific tenant, computes RRF scores, and returns merged candidate context.

CREATE OR REPLACE FUNCTION match_document_chunks_hybrid(
    query_text TEXT,
    query_embedding VECTOR(1536),
    match_tenant_id UUID,
    candidate_limit INT DEFAULT 50,
    rrf_k INT DEFAULT 60
)
RETURNS TABLE (
    id UUID,
    content TEXT,
    metadata JSONB,
    rrf_score FLOAT
)
LANGUAGE plpgsql
AS $$
BEGIN
    RETURN QUERY
    WITH dense_matches AS (
        SELECT
            dc.id,
            ROW_NUMBER() OVER (ORDER BY dc.embedding <=> query_embedding) AS rank
        FROM document_chunks dc
        WHERE dc.tenant_id = match_tenant_id
        ORDER BY dc.embedding <=> query_embedding
        LIMIT candidate_limit
    ),
    sparse_matches AS (
        SELECT
            dc.id,
            ROW_NUMBER() OVER (ORDER BY ts_rank_cd(dc.tsv, plainto_tsquery('english', query_text)) DESC) AS rank
        FROM document_chunks dc
        WHERE dc.tenant_id = match_tenant_id
          AND dc.tsv @@ plainto_tsquery('english', query_text)
        ORDER BY ts_rank_cd(dc.tsv, plainto_tsquery('english', query_text)) DESC
        LIMIT candidate_limit
    )
    SELECT
        dc.id,
        dc.content,
        dc.metadata,
        COALESCE(1.0 / (rrf_k + dm.rank), 0.0) +
        COALESCE(1.0 / (rrf_k + sm.rank), 0.0) AS rrf_score
    FROM dense_matches dm
    FULL OUTER JOIN sparse_matches sm ON dm.id = sm.id
    JOIN document_chunks dc ON dc.id = COALESCE(dm.id, sm.id)
    ORDER BY rrf_score DESC
    LIMIT candidate_limit;
END;
$$;

Production Execution Pipeline (TypeScript / Node.js)

The following service layer coordinates candidate retrieval from PostgreSQL, formats contexts, calls Cohere's Re-Rank API, and returns top-ranked context fragments ready for LLM prompt injection.

import { Pool } from 'pg';
import { CohereClient } from 'cohere-ai';
import OpenAI from 'openai';

interface ChunkResult {
  id: string;
  content: string;
  metadata: Record<string, any>;
  rrf_score: number;
  re_rank_score?: number;
}

export class ProductionRAGPipeline {
  private pgPool: Pool;
  private cohere: CohereClient;
  private openai: OpenAI;

  constructor() {
    this.pgPool = new Pool({ connectionString: process.env.DATABASE_URL });
    this.cohere = new CohereClient({ token: process.env.COHERE_API_KEY });
    this.openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
  }

  async retrieveContext(
    userQuery: string,
    tenantId: string,
    topKFinal: number = 5
  ): Promise<ChunkResult[]> {
    // Step 1: Generate dense query vector
    const embeddingResponse = await this.openai.embeddings.create({
      model: 'text-embedding-3-small',
      input: userQuery,
    });
    const queryVector = JSON.stringify(embeddingResponse.data[0].embedding);

    // Step 2: Execute Hybrid Database Retrieval (RRF)
    const sqlQuery = `
      SELECT id, content, metadata, rrf_score
      FROM match_document_chunks_hybrid($1, $2::vector, $3::uuid, 30, 60);
    `;
    const { rows: candidateChunks } = await this.pgPool.query<ChunkResult>(sqlQuery, [
      userQuery,
      queryVector,
      tenantId,
    ]);

    if (candidateChunks.length === 0) {
      return [];
    }

    // Step 3: High-Precision Re-Ranking via Cohere Cross-Encoder
    const reRankResponse = await this.cohere.rerank({
      model: 'rerank-english-v3.0',
      query: userQuery,
      documents: candidateChunks.map((chunk) => chunk.content),
      topN: topKFinal,
    });

    // Step 4: Map reranked indices back to target chunks
    const finalContexts: ChunkResult[] = reRankResponse.results.map((result) => {
      const originalChunk = candidateChunks[result.index];
      return {
        ...originalChunk,
        re_rank_score: result.relevanceScore,
      };
    });

    return finalContexts;
  }
}

Comparing Retrieval Strategy Performance

Selecting the right architecture requires balancing accuracy against compute latency and cost trade-offs.

Operational Best Practices

1. Chunking Dynamics: Use parent-child context chunking. Generate vector embeddings for small child chunks (e.g., 200 tokens) to keep dense retrieval precise, but pass the larger parent chunk (e.g., 1000 tokens) to the LLM context window to retain narrative continuity.
2. Metadata Filtering Prior to Search: Always inject tenant constraints, access control list (ACL) security filters, or date range restrictions directly into the SQL WHERE clause prior to vector distance calculations to avoid cross-tenant data leakage.
3. In-Memory Caching: Cache dense vector embeddings of frequent user queries in Redis to bypass embedding API generation costs and slice 50ms off latency.

How BrickTry Accelerates & Powers This

Building enterprise-grade, hybrid-search RAG pipelines requires navigating embedding model choices, tuning index performance, implementing tenant isolation, and configuring Cross-Encoder APIs. BrickTry accelerates the entire lifecycle of building modern AI applications through integrated execution environments and engineering pods.

┌────────────────────────────────────────────────────────────────────────┐
│                        BRICKTRY PLATFORM ENGINE                        │
├────────────────────────────────────────────────────────────────────────┤
│  ┌─────────────────────────┐         ┌──────────────────────────────┐  │
│  │   BrickTry Lab          │         │  AI Dev Pairing Engine       │  │
│  │   - Instant WASM/Vite   │◄───────►│  - AST Security Auditing     │  │
│  │   - Local pgvector Dev  │         │  - Automated Migration Gen   │  │
│  └────────────┬────────────┘         └──────────────┬───────────────┘  │
│               │                                     │                  │
│               └──────────────────┬──────────────────┘                  │
│                                  ▼                                     │
│  ┌──────────────────────────────────────────────────────────────────┐  │
│  │                 Senior Full-Stack Engineering Pods               │  │
│  │     - HNSW Index Optimization   - RLS Multi-Tenant Audits       │  │
│  └───────────────────────────────┬──────────────────────────────────┘  │
└──────────────────────────────────┼─────────────────────────────────────┘
                                   ▼
┌────────────────────────────────────────────────────────────────────────┐
│                   100% Owned Enterprise Codebase                       │
└────────────────────────────────────────────────────────────────────────┘

• Interactive Lab Sandbox (/lab): Instantly test, benchmark, and visualize hybrid retrieval models inside BrickTry’s zero-setup, in-browser container execution environment. Prototype SQL tsvector queries, measure HNSW distance metrics, and validate Cohere re-ranking latencies before deploying a single line of production code.
• AI-Human Dev Pairing: BrickTry’s autonomous AI scaffolding generates the vector migrations, client wrappers, and TypeScript integration pipelines. Simultaneously, dedicated senior full-stack engineers review context-chunking boundaries, optimize SQL indexes, and ensure robust error handling.
• Automated AST Security Auditing: RAG pipelines risk exposing confidential context or succumbing to context-poisoning vulnerabilities. BrickTry’s static analysis engine scans vector queries and JSON payload interfaces to enforce Row-Level Security (RLS) patterns and guard against context injection.
• 100% Source Code & Infrastructure Ownership: You retain full ownership of all generated code, GitHub repositories, Dockerized services, and database schema scripts. Build custom RAG architectures on BrickTry without vendor lock-in or proprietary API dependencies.

Build, Test, and Scale This on BrickTry

BrickTry pairs you with autonomous AI scaffolding supervised by dedicated senior full-stack software engineers in an interactive in-browser development sandbox. Test, build, and deploy production-grade software with 100% source code ownership and zero vendor lock-in.

Launch Interactive Requirement Builder →