RAG Basics & Vector Databases

Concept#

A model's training data is frozen at a point in time and has no idea about your company's internal documentation, this morning's product changelog, or a customer's specific account history. Retrieval-Augmented Generation (RAG) solves this without retraining or fine-tuning anything: instead of hoping the model already knows the answer, you retrieve a small, relevant slice of a much larger knowledge base at request time and hand it to the model as context, then ask it to answer using that context.

RAG has two entirely separate phases that run at different times: ingestion (offline, run once or on a schedule, whenever source documents change) and query-time retrieval (online, run on every single user question).

Phase 1, Ingestion: documents become searchable vectors#

const doc = fs.readFileSync('handbook.md', 'utf-8');
// 40,000 characters, far too large for one embedding call

import fs from "fs";
 
const doc = fs.readFileSync("handbook.md", "utf-8"); // e.g. 40,000 characters
 
// 1. Chunk the document, far too large to embed as a single unit
const chunks = splitIntoChunks(doc, { size: 500, overlap: 50 });
// → ~85 overlapping chunks of ~500 characters each
 
// 2. Embed each chunk, a SEPARATE model from the one that will generate answers
const vectors = await Promise.all(
  chunks.map((chunk) => embeddingModel.embed(chunk.text)),
);
// → each chunk.text becomes a fixed-length float vector (e.g. 1536 dimensions)
 
// 3. Store vector + original text + metadata in a vector database
await vectorDb.upsert(
  chunks.map((chunk, i) => ({
    id: chunk.id,
    vector: vectors[i],
    metadata: { source: chunk.source, text: chunk.text },
  })),
);

Chunking with overlap (overlap: 50 above) exists to prevent a fact from being silently split across a chunk boundary, without overlap, a sentence that happens to straddle exactly where one 500-character chunk ends and the next begins can become unretrievable, because neither half alone is similar enough to a relevant query to surface in search. The embedding model is deliberately a different model from the one that later generates answers, its only job is producing a numeric vector that captures a chunk's meaning, not generating text.

Phase 2, Query time: retrieve, then generate#

const question = 'What is our refund window for annual plans?';
const queryVector = await embeddingModel.embed(question);

// 1. Embed the user's question, with the SAME embedding model used at ingestion
const question = "What is our refund window for annual plans?";
const queryVector = await embeddingModel.embed(question);
 
// 2. Similarity search against the vector database
const results = await vectorDb.query({ vector: queryVector, topK: 4 });
// → the 4 nearest stored vectors, ranked by similarity (commonly cosine distance)
 
// 3. Build context from ONLY the retrieved chunks, not the whole knowledge base
const context = results.map((r) => r.metadata.text).join("\n---\n");
 
// 4. Generate a grounded answer using client.messages.create()
const response = await client.messages.

This is the entire point of RAG: pull in a small, relevant slice of a much larger knowledge base for THIS specific question, instead of stuffing the entire knowledge base into every request (which would be both prohibitively expensive in tokens and, past a certain size, would simply exceed the context window). The final answer is grounded in the retrieved chunk's actual current wording rather than the model's frozen training-data memory, which is exactly what lets a RAG system stay current with a knowledge base that changes daily, with zero retraining.

The single most important invariant: same embedding model, both times#

Using the same embedding model at ingestion and at query time is not a minor detail, mixing embedding models silently breaks similarity search entirely, because different embedding models produce vectors in entirely different, mutually meaningless vector spaces. A query vector from model A compared against stored vectors from model B will not return relevant results, and, critically, this failure produces no error at all; it just quietly returns poor-quality or irrelevant chunks, which is a much harder bug to diagnose than an outright crash.

Instructing the model to answer only from context#

The system prompt in the query-time example above deliberately constrains Claude to the retrieved context. Without this instruction, the model may blend its own training-data knowledge with the retrieved context, which can reintroduce exactly the staleness problem RAG exists to solve, a well-designed RAG system prompt typically also tells the model what to say when the retrieved context doesn't actually answer the question, rather than letting it guess.

Try It#

Predict what happens in this scenario before checking the solution.