RAG & Vector Stores

A pre-trained LLM has two limitations RAG addresses. First, its knowledge is frozen at training cutoff. GPT-4o cannot tell you about an event last week unless it was retrained or given context. Second, its knowledge is everyone's knowledge: there is no Acme Inc. employee handbook in the GPT-4 training set, and there should not be.

You have three options. Fine-tune the model on your data (expensive, slow, and the model still hallucinates with confidence on the things it didn't quite learn). Stuff the data into every prompt (works for tiny corpora, breaks for anything over ≈100k tokens). Or retrieve the relevant subset at query time, paste it into the prompt, and let the model do its in-context thing. Option three is RAG.

Lewis et al. 2020 named the technique. The mechanics predate the paper: question-answering systems have been doing "find the passage, then read it" since at least DrQA in 2017. What 2020-era RAG added was end-to-end training of the retriever and the generator together. What 2023-era production RAG dropped was the end-to-end training. Modern systems use a frozen embedding model and a frozen LLM, glued together with a database.

The unspoken claim in every "RAG works" demo is that the retrieval step actually retrieved the right thing. When it does not, the model hallucinates with the retrieved-but-wrong context as fuel. Retrieval failure is the dominant failure mode of production RAG, by a wide margin.

An embedding model is a neural net that takes a text and returns a vector in $\mathbb{R}^d$. The vector is supposed to be "close to" the vectors of texts with similar meaning. Where the model puts those vectors is determined by the training objective: contrastive learning on pairs of (query, relevant passage), often with hard negatives mined from BM25 false positives.

Concrete numbers as of 2026:

• OpenAI text-embedding-3-small: 1536 dims, ≈MTEB 62.3, $0.02/1M tokens, 8192 token context.
• OpenAI text-embedding-3-large: 3072 dims, ≈MTEB 64.6, $0.13/1M tokens.
• Cohere embed-english-v3.0: 1024 dims, ≈MTEB 64.5, optimized for English retrieval.
• BGE-M3: 1024 dims, open weights, multilingual, supports dense + sparse + multi-vector in one model. Probably the strongest open option.
• Voyage voyage-3-large: 1024 dims, leaderboard topper for English retrieval as of late 2025.
• Nomic nomic-embed-text-v1.5: 768 dims with Matryoshka representations — trained so the most important information is packed into the leading dimensions, like nested dolls, which is why you can truncate to 64/128/256/512 and still keep most of the meaning. Apache-2 licensed.

MTEB (the Massive Text Embedding Benchmark) is the public leaderboard those scores come from: it ranks embedding models across a spread of tasks — classification, clustering, retrieval, summarization — so one number stands in for general-purpose quality. It moves monthly. The headline numbers cluster within 2-3 points; the choice between top-tier embeddings rarely changes a downstream retrieval system by more than a few percent on standard benchmarks. The choice between top-tier and middle-tier usually does. The choice between embedding and not-embedding (using BM25 instead) usually does, and often goes the wrong way: dense embeddings can lose to BM25 on out-of-distribution domains.

Library path:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-m3")
texts = ["The quick brown fox jumps.", "A speedy auburn fox leaps."]
embeddings = model.encode(texts, normalize_embeddings=True)  # (2, 1024)

From-scratch path (the cosine-similarity computation itself, by hand):

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Cosine sim between two vectors. -1 to +1."""
    return (a @ b) / (np.linalg.norm(a) * np.linalg.norm(b))

def cosine_similarity_batch(query: np.ndarray, corpus: np.ndarray) -> np.ndarray:
    """One query, many corpus vectors. Returns (n_corpus,)."""
    # Normalize once at corpus build time; here for clarity:
    q_n = query / np.linalg.norm(query)
    c_n = corpus / np.linalg.norm(corpus, axis=1, keepdims=True)
    return c_n @ q_n

Three distance functions show up in every vector database. They are not interchangeable, but for normalized embeddings they reduce to each other in predictable ways.

Cosine similarity is $\frac{a \cdot b}{\|a\| \|b\|}$. It measures angle, not magnitude. Range: $[-1, 1]$.

Dot product (inner product) is $a \cdot b$. Range: unbounded. Equal to cosine if both vectors are unit-norm.

L2 distance (Euclidean) is $\|a - b\| = \sqrt{\sum_i (a_i - b_i)^2}$. For unit-norm vectors, $\|a - b\|^2 = 2 - 2 (a \cdot b)$, so L2 ordering is the inverse of dot-product ordering.

The practical rule: most modern embedding models output unit-norm vectors (or are trained with normalized cosine objectives). For those, cosine, dot, and L2 give the same ranking. Pick whichever your vector database is fastest at; usually dot product, since it skips the normalization step at query time.

Caveat: not every embedding model is normalized. OpenAI's text-embedding-3-* outputs vectors that are approximately unit-norm but not exactly. Cohere's are normalized. BGE varies by checkpoint. If you switch metric without normalizing, results can degrade silently. Always normalize at index time and query time, both, even when the model "should" do it.

def normalize(v: np.ndarray) -> np.ndarray:
    """Project a vector onto the unit sphere. Safe for zero vectors."""
    norm = np.linalg.norm(v, axis=-1, keepdims=True)
    return v / np.maximum(norm, 1e-12)

A vector database is a system that stores $(id, vector, metadata)$ tuples and answers "give me the $k$ tuples whose vectors are nearest to this query vector". The hard work is in approximate nearest neighbor (ANN) search: exact nearest-neighbor over a billion 1024-dim vectors is too slow, so you trade accuracy for speed.

The dominant ANN algorithms:

• HNSW (Hierarchical Navigable Small Worlds): a multi-layer graph built so most nodes can be reached from any other in a minimum number of hops (the "small world" property), with each layer a sparser subset of the one below. Search descends layer by layer, starting from broad, coarse approximations and narrowing at each level. Recall ≈95-99% at 10-50× speedup over brute force. The default in Qdrant, Weaviate, Pinecone, ChromaDB.
• IVF + PQ (Inverted File + Product Quantization): partition the vector space into clusters (IVF), then compress each vector with product quantization (PQ) — replacing the raw floats with a short code, the trick that lets FAISS hold billions of vectors in memory. Lower memory, slightly lower recall. The default for FAISS in production.
• DiskANN / Vamana: HNSW-style graph but designed to live on disk. The choice when your index is billions of vectors and does not fit in RAM. Microsoft's Vamana paper from 2019; FreshDiskANN added insert/delete support.
• ScaNN (Google): anisotropic vector quantization tuned for inner-product search. Strong recall@10 at high QPS.

The vector database market is crowded. The five names you will see most:

• Pinecone: managed, simple API, expensive at scale.
• Weaviate: open source, schema-first, GraphQL API.
• Qdrant: open source, Rust, growing fast in 2025-2026.
• Chroma: open source, embedded-first, the "SQLite of vector DBs".
• FAISS: not a DB, a library. Most "vector database" features (HTTP API, metadata filtering, replication) are missing. Use it when you control deployment.

What none of these solve: the embedding model is the bottleneck, not the database. If your embeddings put "Python the language" and "python the snake" near each other, swapping HNSW for IVF will not fix it.

model = SentenceTransformer("BAAI/bge-m3")
doc_emb = model.encode(docs, normalize_embeddings=True)
q_emb = model.encode([query], normalize_embeddings=True)
hits = util.semantic_search(q_emb, doc_emb, top_k=k)[0]  # [{'corpus_id','score'}, ...]

def add(self, docs):
    self.docs.extend(docs)
    vecs = self.embed_fn(docs)
    norms = np.linalg.norm(vecs, axis=-1, keepdims=True) + 1e-9
    normed = vecs / norms
    self.vecs = normed if self.vecs is None else np.vstack([self.vecs, normed])

def search(self, query, k=10):
    q = self.embed_fn([query])
    q = q / (np.linalg.norm(q, axis=-1, keepdims=True) + 1e-9)
    sims = (self.vecs @ q.T).squeeze(-1)
    order = np.argsort(-sims)[:k]
    return [(int(i), float(sims[i])) for i in order]

Chunking is the step where you take a long document and break it into pieces small enough to embed. There is no clean answer to "how big should a chunk be?", because the right answer depends on the embedding model's context, the question style, and the document structure.

The dominant strategies, in increasing order of sophistication:

• Fixed-size: split every N tokens (typical: 256-1024), with overlap (typical: 10-20%). Fast, dumb, often fine.
• Recursive character splitting (LangChain default): try to split on \n\n, fall back to \n, then ., then space, then character. Tries to keep semantically meaningful units together.
• Sentence-based: use a sentence tokenizer (spaCy, NLTK), then group sentences into chunks of N tokens.
• Markdown / structural: split on #, ##, ### headers. Best for documents with explicit structure.
• Semantic: embed every sentence, find chunk boundaries where the cosine similarity between adjacent sentences drops below a threshold. The "smart" strategy; results are not consistently better than recursive in benchmarks.
• Late chunking (Jina, 2024): embed the whole document with a long-context embedding model, then average-pool over chunks. Preserves long-range context. Strong on tasks where chunks need to know about the rest of the doc.
• Contextual retrieval (Anthropic, 2024): prepend each chunk with a 50-100 token LLM-generated summary of "what this chunk is about, in the context of the full document". Embedding the augmented chunk improves retrieval recall by ≈30% on Anthropic's internal eval. The cost is one LLM call per chunk at index time.

The honest practical advice: start with recursive character splitting at 512 tokens with 50-token overlap. Measure recall@10 against a held-out QA set. If it is bad, try contextual retrieval. If that is still bad, the problem is probably not chunking.

# Recursive splitter, the simplified version
def recursive_split(text: str, chunk_size: int = 512, separators: list = None) -> list[str]:
    seps = separators or ["\n\n", "\n", ". ", " "]
    if len(text) <= chunk_size:
        return [text]
    for sep in seps:
        if sep in text:
            parts = text.split(sep)
            chunks, cur = [], ""
            for p in parts:
                if len(cur) + len(p) + len(sep) <= chunk_size:
                    cur = cur + (sep if cur else "") + p
                else:
                    if cur: chunks.append(cur)
                    cur = p
            if cur: chunks.append(cur)
            # Recurse only on chunks still too big
            return [c for ck in chunks for c in recursive_split(ck, chunk_size, seps[1:])]
    return [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]

BM25 (Best Matching 25) is the term-matching score that powered Lucene-based search engines for 25 years before anyone said "embedding". It scores a (query, document) pair by how many query terms appear in the document, weighted by inverse document frequency (IDF — how rare a term is across the whole corpus, so matching a rare word counts for more than matching "the") and length-normalized:

With $k_1 \approx 1.5$ and $b \approx 0.75$ as standard. The thing it gets right that dense retrievers get wrong: exact-match recall on rare terms. If your query is "Q4 2024 revenue figures for product XYZ-42", BM25 finds the documents containing "XYZ-42". A dense embedding model, depending on training data, may put XYZ-42 in a "product-name-shaped" cluster that fires on any product name.

Hybrid search is the sum (or fusion) of BM25 and dense scores. Two combination strategies:

• Linear weighted sum: $\text{score} = \alpha \cdot \text{BM25} + (1-\alpha) \cdot \text{dense}$, with $\alpha$ tuned on held-out queries.
• Reciprocal Rank Fusion (RRF; Cormack et al. 2009): rank-based, robust. $\text{RRF}(d) = \sum_r \frac{1}{k + \text{rank}_r(d)}$, with $k \approx 60$. RRF does not require score normalization between systems and consistently beats linear blending in benchmarks.

# RRF in five lines
def rrf(rankings: list[list[str]], k: int = 60) -> list[tuple[float, str]]:
    """rankings: list of ranked doc-id lists, one per retriever."""
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
    return sorted(((s, d) for d, s in scores.items()), reverse=True)

The retrieval pipeline so far is a bi-encoder: query and document are embedded independently, similarity is one dot product per candidate. That is cheap but coarse. A cross-encoder takes the (query, document) pair together through a transformer and outputs a single relevance score. It is much more accurate at telling apart "kinda relevant" from "exactly the answer". It is also 100-1000× slower per pair.

The canonical pipeline: retrieve top-100 with bi-encoder, re-rank to top-10 with cross-encoder. The bi-encoder's job is recall; the cross-encoder's job is precision. The big names:

• Cohere rerank-3.5: API-based, multilingual, $2/1k searches.
• BGE-reranker-v2-m3: open weights, multilingual, runs on a single GPU.
• mxbai-rerank-large-v1: open weights, English-focused, currently top of the BEIR rerank leaderboard.
• MS MARCO MiniLM-L-6: tiny (22M params), surprisingly strong, the de facto baseline since 2021.

Quantitative effect: on BEIR, adding a re-ranker typically gives +5 to +15 points of nDCG@10 over a strong dense retriever alone. The marginal cost is usually worth it.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("BAAI/bge-reranker-v2-m3")
candidates = ["doc text 1", "doc text 2", "doc text 3"]
pairs = [(query, d) for d in candidates]
scores = reranker.predict(pairs)   # higher = more relevant
ranked = sorted(zip(scores, candidates), reverse=True)

A user query is not always a good search query. "Why is my Postgres slow?" might match documentation about MySQL because both contain the word "database" with high frequency, while the actually-relevant Postgres-specific docs use terms like VACUUM, autovacuum, pg_stat_statements. The query needs to be rewritten before it hits the retriever.

The two patterns:

• Query expansion: ask an LLM to rewrite the query, generate variants, or expand acronyms. Run retrieval on all variants and fuse the results with RRF.
• HyDE (Hypothetical Document Embeddings; Gao et al. 2022): ask the LLM to write a fake answer to the query, embed that fake answer, and use it as the search vector. The intuition is that the fake answer has the vocabulary and length of a real answer, which makes it embed closer to the real answer than the question would.

HyDE works because of an asymmetry in embedding space: question-shaped texts and answer-shaped texts cluster in different regions even when they share topic, and the answer-shaped clusters are denser. Caveat: HyDE adds latency (one extra LLM call) and amplifies hallucinations when the LLM's guess is wrong. It helps most on hard domains the embedding model has not seen, and hurts on simple factual lookups.

def hyde_search(query: str, llm, retriever) -> list[str]:
    """Generate a fake answer, embed it, retrieve real docs."""
    fake = llm(f"Write a brief, plausible answer to: {query}")
    return retriever.search(fake, k=10)

Multi-hop questions are the ones a single retrieval step cannot answer. "Which film directed by the brother of the lead actor in Inception won an Oscar?" needs at least three lookups: lead actor of Inception (Leonardo DiCaprio), brother (Adam Farrar... wait, that is wrong — actually Christopher Nolan directed it, so this is a bad example, but you get the shape). The point: chained retrieval where each step depends on the last.

Two architectures:

• Iterative RAG: agent loop where the LLM emits a new search query each turn until it has enough. Most production multi-hop is this.
• GraphRAG (Microsoft, 2024): build a knowledge graph from the corpus at index time, retrieve subgraphs at query time. Performs well on "global" questions ("summarize the major themes of this corpus") and worse than vanilla RAG on "local" factual questions.

The honest take on GraphRAG: it works well in the original paper's benchmarks. Practitioners report mixed results when reproducing on their own data. The construction cost (one LLM pass over every document) is real. Whether it is worth it depends on whether your users ask synthesis questions versus lookup questions.

CAG (cache-augmented generation) is the technique that, for small enough corpora, fits the entire corpus into the model's context using KV-cache pre-loading. Chan et al. 2024 ("Don't Do RAG: When Cache-Augmented Generation is All You Need") argued that for corpora under ≈100k tokens, CAG matches or beats RAG with lower latency and no retrieval failures.

The pitch is real. Frontier models (Gemini 1.5, Claude 3.5/4, GPT-4.1) handle 1M+ tokens. Cache once, query many times, get RAG-like behavior with zero infrastructure. For the right corpus (a single product manual, a single legal contract, a code repository), it works.

CAG does not replace RAG when:

• The corpus exceeds the practical context budget (most corpora).
• The corpus changes faster than you can re-prefill.
• You need provenance (RAG gives you "this answer came from chunk X"; CAG gives you "this answer came from the model").
• Cost matters at scale (per-query token costs scale with context length).

The honest framing: RAG and CAG are points on a spectrum. RAG is "retrieve $k$ chunks, put them in context". CAG is "set $k$ = all of them". The dial between them is mostly economic.

RAG has two failure modes: retrieving the wrong context, and generating wrong answers from the right context. Evaluation needs to measure both.

Retrieval metrics (per query, average over a labeled QA set):

• Recall@k: fraction of relevant documents in the top-k. The blunt instrument; the one you optimize for in early iteration.
• MRR (Mean Reciprocal Rank): $\frac{1}{n} \sum_q \frac{1}{\text{rank}_q}$, where $\text{rank}_q$ is the position of the first relevant document for query $q$. Penalizes low-ranked relevant hits.
• nDCG@10: normalized discounted cumulative gain. Unlike recall, it uses graded relevance (a document can be perfectly, partly, or not relevant) and discounts each hit by the log of its rank, so a relevant result at position 1 counts for more than the same result at position 9. The standard metric on the BEIR retrieval benchmarks (see Going Further).
• Hit Rate@k: did at least one relevant document appear in top-k. Coarser than recall but easier to interpret.

Generation metrics:

• Faithfulness: does the generated answer contain claims that are supported by the retrieved context? Often LLM-judged with a binary "yes / no, this sentence is unsupported".
• Answer relevance: is the answer responsive to the question? LLM-judged.
• Context precision: of the chunks retrieved, what fraction were actually used in the answer?

The standard frameworks in 2026: RAGAS (built around the four metrics above), TruLens (similar plus trace-level logging), DeepEval (broader LLM-eval that includes RAG metrics). They all use LLM-as-judge under the hood, with the failure modes that implies (next chapter).

# A minimal recall@k computation
def recall_at_k(retrieved: list[str], relevant: set[str], k: int) -> float:
    if not relevant: return 0.0
    hits = sum(1 for d in retrieved[:k] if d in relevant)
    return hits / len(relevant)

A production RAG system has roughly the following layers. The choices at each layer compose multiplicatively; small wins everywhere stack.

The most common mistake: optimizing the embedding model before measuring retrieval. The second most common: switching frameworks (LangChain to LlamaIndex to vanilla Python) instead of fixing the eval.