Introduction to RAG

Retrieval-Augmented Generation (RAG) is a technique that combines LLMs with external knowledge retrieval. Instead of relying solely on what the model learned during training, RAG retrieves relevant documents and includes them in the prompt.

The Problem RAG Solves

LLMs have two fundamental limitations:

Knowledge cutoff: Training data ends at a specific date
Hallucination: Models confidently generate incorrect information

RAG addresses both by grounding the model's responses in real, current data.

Without RAG:
User: "What's our company's refund policy?"
LLM:  "Most companies offer 30-day refunds..." (generic, potentially wrong)

With RAG:
User: "What's our company's refund policy?"
[Retrieves: refund-policy.pdf → "14-day money-back guarantee..."]
LLM:  "Your company offers a 14-day money-back guarantee..." (accurate!)

How RAG Works

                    ┌─────────────────┐
User Question ────→│  1. Embed Query  │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │ 2. Search Vector│
                    │    Database     │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │ 3. Retrieve Top │
                    │    Documents    │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │ 4. Combine with │
                    │   Original Query│
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │ 5. Send to LLM  │
                    └────────┬────────┘
                             │
                    ┌────────▼────────┐
                    │  6. LLM Responds│
                    │  with Context   │
                    └─────────────────┘

The RAG Pipeline Steps

Indexing Phase (done once, ahead of time):
- Load your documents (PDFs, web pages, databases)
- Split them into chunks
- Generate embeddings for each chunk
- Store in a vector database
Query Phase (happens per user query):
- Convert the user's question into an embedding
- Search the vector database for similar chunks
- Retrieve the top-k most relevant chunks
- Add them to the LLM prompt as context
- Generate a response grounded in the retrieved documents

What Are Embeddings?

Embeddings are numerical representations of text that capture meaning. Similar texts have similar embeddings.

python

from openai import OpenAI

client = OpenAI()

def get_embedding(text):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=text
    )
    return response.data[0].embedding  # List of 1536 floats

# Similar texts → similar embeddings
emb1 = get_embedding("How do I return a product?")
emb2 = get_embedding("What is your refund policy?")
emb3 = get_embedding("Tell me about the weather")

# emb1 and emb2 will be close (similar meaning)
# emb3 will be far from both (different topic)

Simple RAG Implementation

Here's a minimal RAG system using Python:

python

from openai import OpenAI
import numpy as np

client = OpenAI()

# Step 1: Your knowledge base
documents = [
    "Our refund policy allows returns within 14 days of purchase.",
    "Premium plans cost $49/month and include priority support.",
    "Free shipping on orders over $50 within the United States.",
    "Business hours are Monday to Friday, 9 AM to 6 PM IST.",
    "Contact support at support@example.com or call +91 98765 43210."
]

# Step 2: Create embeddings for all documents
def create_index(docs):
    embeddings = []
    for doc in docs:
        emb = get_embedding(doc)
        embeddings.append(emb)
    return np.array(embeddings)

doc_embeddings = create_index(documents)

# Step 3: Search for relevant documents
def search(query, top_k=3):
    query_emb = np.array(get_embedding(query))

    # Cosine similarity
    similarities = np.dot(doc_embeddings, query_emb) / (
        np.linalg.norm(doc_embeddings, axis=1) * np.linalg.norm(query_emb)
    )

    top_indices = similarities.argsort()[-top_k:][::-1]
    return [(documents[i], similarities[i]) for i in top_indices]

# Step 4: Generate answer with context
def rag_query(question):
    # Retrieve relevant documents
    results = search(question, top_k=3)
    context = "\n".join([doc for doc, score in results])

    # Generate answer
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": """Answer the user's question based ONLY on the
                provided context. If the answer isn't in the context,
                say "I don't have information about that."

                Context:
                """ + context
            },
            {
                "role": "user",
                "content": question
            }
        ],
        temperature=0.2
    )
    return response.choices[0].message.content

# Usage
answer = rag_query("What's the refund policy?")
print(answer)
# → "Returns are accepted within 14 days of purchase."

answer = rag_query("Do you offer free shipping?")
print(answer)
# → "Yes, free shipping is available on orders over $50 within the US."

Document Chunking

Large documents need to be split into smaller chunks for effective retrieval:

python

def chunk_text(text, chunk_size=500, overlap=50):
    """Split text into overlapping chunks."""
    words = text.split()
    chunks = []
    start = 0

    while start < len(words):
        end = start + chunk_size
        chunk = " ".join(words[start:end])
        chunks.append(chunk)
        start = end - overlap  # Overlap for context continuity

    return chunks

Chunking Strategies

Strategy	How	Best For
Fixed size	Split every N words/tokens	Simple, predictable
Sentence-based	Split at sentence boundaries	Natural text
Paragraph-based	Split at paragraph breaks	Structured documents
Semantic	Use LLM to find topic boundaries	Complex documents
Recursive	Try large chunks, subdivide if too big	Mixed-format documents

RAG vs Fine-Tuning

Aspect	RAG	Fine-Tuning
Data freshness	Real-time, always current	Static after training
Setup effort	Moderate (build pipeline)	High (prepare dataset, train)
Cost	Per-query (retrieval + LLM)	Upfront training cost
Best for	Factual Q&A, documentation	Style, format, domain expertise
Knowledge updates	Just update the documents	Retrain the model
Transparency	Can show source documents	Black box

When to Use RAG

RAG is ideal for:

Internal documentation Q&A — Company wikis, policies, procedures
Customer support — Product docs, FAQs, troubleshooting guides
Legal/compliance — Search through contracts, regulations
Research — Query across papers, reports, data
E-commerce — Product information, inventory queries

RAG Limitations

Retrieval quality — Bad retrieval = bad answers (garbage in, garbage out)
Context window limits — Can only pass so many chunks to the LLM
Latency — Extra step (retrieval) adds response time
Chunking challenges — Poor chunking loses context
Not a replacement for fine-tuning — RAG handles knowledge, not behavior

Key Takeaways

RAG = Retrieval + Generation — Fetch relevant docs, then answer with them
Embeddings capture meaning — Similar questions find similar documents
Chunking matters — How you split documents affects retrieval quality
RAG is the #1 technique for building knowledge-grounded AI applications
Start simple — A basic RAG pipeline can be built in under 100 lines of code

In the next topic, we'll dive deeper into vector databases and embeddings.