Module 10: Generative AI, LLMs & Prompt EngineeringBy Ashok Kumar Yadav4 min read

Retrieval-Augmented Generation (RAG): Architecture and Setup

LLMs do not know your private documents and can invent answers. RAG retrieves relevant passages at question time and puts them in the prompt, so answers are grounded in your data.

RAG architecture: documents chunked, embedded, stored in a vector database and retrieved for an LLM prompt

Quick answer: RAG has two stages. Index: split documents into chunks, embed them and store vectors. Query: embed the question, retrieve the nearest chunks, add them to the prompt and let the LLM answer only from that context.

The RAG pipeline

Indexing happens once, or whenever documents change. Query time runs for every question.

Step	What happens
Load	Read PDFs, pages or databases
Chunk	Split into 200 to 500 word passages with overlap
Embed	Convert each chunk to a vector
Store	Save vectors and metadata in a vector database
Retrieve	Find the top-k chunks for a question
Generate	Answer using only the retrieved context

Chunk and index documents

Chunk size is a trade-off: small chunks are precise but lose context, large chunks add noise. Overlap prevents answers from being cut in half.

Chunk, embed and store with ChromaDB

# pip install chromadb sentence-transformers
import chromadb
from sentence_transformers import SentenceTransformer

def chunk(text, size=120, overlap=30):
    words = text.split()
    step = size - overlap
    return [" ".join(words[i:i+size]) for i in range(0, max(len(words) - overlap, 1), step)]

docs = {"policy.txt": open("policy.txt").read()}
model = SentenceTransformer("all-MiniLM-L6-v2")
db = chromadb.PersistentClient(path="rag_db")
col = db.get_or_create_collection("docs")

for name, text in docs.items():
    parts = chunk(text)
    col.add(ids=[f"{name}-{i}" for i in range(len(parts))],
            documents=parts,
            embeddings=model.encode(parts).tolist(),
            metadatas=[{"source": name}] * len(parts))
print("chunks stored:", col.count())

Retrieve and generate

Tell the model to answer only from the context and to say so when the answer is missing. That instruction reduces hallucination more than any other single change.

Retrieval and prompt assembly

def retrieve(q, k=4):
    r = col.query(query_embeddings=model.encode([q]).tolist(), n_results=k)
    return list(zip(r["documents"][0], [m["source"] for m in r["metadatas"][0]]))

def build_prompt(q):
    ctx = "\n\n".join(f"[{s}] {t}" for t, s in retrieve(q))
    return ("Answer using only the context. If it is not there, say you don't know. "
            "Cite sources in brackets.\n\nContext:\n" + ctx + "\n\nQuestion: " + q)

print(build_prompt("What is the refund window?"))   # send this to your LLM

Evaluate and improve retrieval

Most RAG failures are retrieval failures. Build a small test set of questions with known source passages and measure hit rate at k. Improve by tuning chunk size, adding metadata filters, using hybrid keyword plus vector search, or adding a reranker. Evaluate answers for faithfulness (supported by context) and relevance separately.

Hands-on practice

Run the examples in the downloadable Jupyter notebook. The extra checks below focus on mistakes that appear in real projects, not just syntax.

Measure retrieval hit rate

Practice example

tests=[('q1','docA',['docA','docC']),('q2','docB',['docD','docB'])]
hits=sum(expected in retrieved for _,expected,retrieved in tests)
print('hit_rate@2 =',hits/len(tests))

hit_rate@2 = 1.0

Evaluate retrieval separately from answer generation so you know where failures originate.

Add citations to the assembled prompt

Practice example

chunks=[('policy.pdf','Refunds are allowed within 30 days.'),('faq.md','Digital goods are final sale.')]
ctx='\n'.join(f'[{s}] {t}' for s,t in chunks)
print(ctx)

[policy.pdf] Refunds are allowed within 30 days.
[faq.md] Digital goods are final sale.

Preserving source IDs makes citations possible downstream.

Self-check

Can you explain why the first practice result is trustworthy rather than just plausible?
What would you monitor or test before using this RAG Architecture workflow in production?
Which assumption in the example would break first on a larger or messier dataset?

Frequently asked questions

RAG or fine-tuning?

Use RAG for changing or private knowledge and citations. Use fine-tuning for style and format. Many systems use both.

What is a good chunk size?

Start around 200 to 400 words with 10 to 20 percent overlap, then tune using your own retrieval test set.

Does RAG stop hallucinations completely?

No. It reduces them by grounding answers, but you still need prompt rules, citations and evaluation.

Primary references

Use these official sources when exact syntax, defaults or definitions matter beyond the examples here.

Written by Ashok Kumar Yadav
Ashok Kumar Yadav writes Xenors tutorials on applied AI, programming and data workflows. Every lesson is built around code you can run and check yourself.
GitHub · LinkedIn