Listen to this Post
Introduction:
Retrieval-Augmented Generation (RAG) has emerged as the definitive architecture for building production-ready AI applications that don’t hallucinate. Unlike standalone large language models that rely solely on pretrained parameters, RAG systems dynamically retrieve relevant external knowledge at query time, grounding responses in verifiable facts. Yet most implementations fail in production because engineers overlook the critical nuances of chunking, embedding selection, and retrieval optimization. This guide breaks down the complete RAG pipeline from first principles to production deployment, giving you the technical arsenal to build AI systems that actually work.
Learning Objectives:
- Master the end‑to‑end RAG pipeline: document ingestion, chunking, embedding generation, vector storage, retrieval, and response generation
- Implement production‑grade chunking strategies and select optimal embedding models for your use case
- Build a fully functional RAG system using LangChain, ChromaDB, and open‑source LLMs with real‑time streaming
- Document Ingestion and Chunking — The Foundation of RAG
Every RAG pipeline begins with document processing. Raw documents—PDFs, text files, HTML, or databases—must be loaded and transformed into manageable pieces. The most common approach uses LangChain’s document loaders combined with recursive text splitters.
Step‑by‑step implementation:
First, install the required dependencies:
pip install langchain langchain-community langchain-openai faiss-cpu pypdf chromadb tiktoken
Load PDF documents and split them into overlapping chunks:
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
Load documents
loader = PyPDFLoader("document.pdf")
documents = loader.load()
Configure chunking with overlap
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000, Each chunk ~1000 characters
chunk_overlap=200, Overlap preserves context across boundaries
separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(documents)
Why chunking matters: Documents lose context when arbitrarily split. A chunk containing revenue figures without the company name becomes useless for retrieval. The `RecursiveCharacterTextSplitter` intelligently splits on paragraph boundaries first, then sentences, then words—preserving semantic coherence.
Chunk size guidelines:
| Strategy | Chunk Size | Overlap | Best For |
|-|||-|
| Fixed-size | 512 tokens | 50 tokens | General purpose, fast retrieval |
| Sentence-based | Variable | 1-2 sentences | Semantic coherence |
| Recursive | Hierarchical | 200 chars | Complex, multi‑section documents |
2. Generating Embeddings — Converting Text to Vectors
Embeddings are numerical representations of text that capture semantic meaning. Each chunk is passed through an embedding model, producing a high‑dimensional vector. These vectors enable semantic search—finding chunks that are conceptually similar to a user’s question, not just keyword matches.
Step‑by‑step implementation:
Choose an embedding model based on your requirements:
Option 1: OpenAI (cloud, best performance) from langchain_openai import OpenAIEmbeddings embeddings = OpenAIEmbeddings(model="text-embedding-ada-002") Option 2: HuggingFace (local, no API key) from langchain_huggingface import HuggingFaceEmbeddings embeddings = HuggingFaceEmbeddings( model_name="all-MiniLM-L6-v2" 384-dim, fast, lightweight ) Option 3: Ollama (local, fully offline) First pull the model: ollama pull nomic-embed-text from langchain_community.embeddings import OllamaEmbeddings embeddings = OllamaEmbeddings(model="nomic-embed-text")
Generate embeddings for all chunks and store them:
Create vector store with ChromaDB from langchain_community.vectorstores import Chroma vectorstore = Chroma.from_documents( documents=chunks, embedding=embeddings, persist_directory="./chroma_db" Persist to disk )
Production recommendation (2025): For general purpose, `text-embedding-ada-002` offers the best balance of performance and cost. For offline deployments, `all-MiniLM-L6-v2` runs entirely locally with no API dependencies. For maximum accuracy, `bge-large-en-v1.5` consistently outperforms alternatives on MTEB benchmarks.
3. Vector Databases — The Retrieval Engine
Vector databases index embeddings and enable fast similarity search. At query time, the user’s question is converted to an embedding, and the database returns the most semantically similar chunks using algorithms like HNSW (Hierarchical Navigable Small World) or IVFFlat.
Step‑by‑step setup:
Install ChromaDB (local, persistent) pip install chromadb Or FAISS (lightweight, in‑memory) pip install faiss-cpu
Implementing retrieval:
Create a retriever with top‑k configuration
retriever = vectorstore.as_retriever(
search_type="similarity", or "mmr" for diversity
search_kwargs={"k": 4} Retrieve top 4 chunks
)
Query the vector store
query = "What is the company's revenue for 2024?"
results = retriever.invoke(query)
for doc in results:
print(f"Content: {doc.page_content[:200]}...")
print(f"Source: {doc.metadata.get('source')}")
Vector database selection guide:
| Requirement | Recommended |
|-|-|
| Production scalability | Pinecone, Milvus |
| Open‑source, self‑hosted | Weaviate, Qdrant |
| Local development | Chroma, FAISS |
| Hybrid search (vector + keyword) | Weaviate with BM25 |
For production systems, implement hybrid search combining dense (vector) and sparse (keyword) retrieval to capture both semantic meaning and exact lexical matches.
4. Retrieval Strategies — Getting the Right Context
Retrieval quality determines answer quality. Beyond simple top‑k similarity, advanced strategies significantly improve relevance.
Step‑by‑step retrieval optimization:
1. Maximum Marginal Relevance (MMR) - reduces redundancy
retriever = vectorstore.as_retriever(
search_type="mmr",
search_kwargs={"k": 4, "fetch_k": 10} Fetch 10, return 4 diverse
)
<ol>
<li>Metadata filtering - narrow search scope
retriever = vectorstore.as_retriever(
search_kwargs={
"k": 4,
"filter": {"source": "annual_report.pdf"} Only search this document
}
)</p></li>
<li><p>Self‑query retrieval - extract filters from natural language
from langchain.retrievers import SelfQueryRetriever
retriever = SelfQueryRetriever.from_llm(llm, vectorstore, document_content_description)
Production best practices:
- Reranking: Apply a cross‑encoder reranker to reorder retrieved chunks by relevance—essential for production quality
- Query expansion: Generate multiple variations of the user’s question to improve recall
- Contextual chunk headers: Prepend chunks with document and section titles to restore lost context
5. Generation — Grounding LLM Responses
The final stage constructs a prompt containing the retrieved context and the user’s question, then passes it to an LLM for response generation.
Step‑by‑step implementation:
from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate
from langchain.schema.runnable import RunnablePassthrough
from langchain.schema.output_parser import StrOutputParser
Define the prompt template
template = """You are an assistant that answers questions based only on the provided context.
If the context doesn't contain the answer, say "I don't have enough information to answer that."
Context:
{context}
Question: {question}
Answer:"""
prompt = ChatPromptTemplate.from_template(template)
Initialize LLM
llm = ChatOpenAI(model="gpt-4", temperature=0) Low temperature for factual answers
Build the RAG chain
def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
Execute
response = rag_chain.invoke("What was the revenue for Acme Inc in 2024?")
print(response)
Token‑level streaming for better UX:
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler llm = ChatOpenAI( model="gpt-4", streaming=True, callbacks=[StreamingStdOutCallbackHandler()] ) Streams response token by token in real‑time
6. End‑to‑End RAG Pipeline — Complete Working Example
Here’s a complete, production‑ready RAG implementation using local models (no API keys required):
Complete RAG pipeline with Ollama + ChromaDB
from langchain_community.document_loaders import TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_community.llms import Ollama
from langchain.prompts import ChatPromptTemplate
from langchain.schema.runnable import RunnablePassthrough
from langchain.schema.output_parser import StrOutputParser
<ol>
<li>Load and chunk
loader = TextLoader("knowledge_base.txt")
docs = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
chunks = text_splitter.split_documents(docs)</p></li>
<li><p>Embed and store (local, offline)
embeddings = OllamaEmbeddings(model="nomic-embed-text")
vectorstore = Chroma.from_documents(chunks, embeddings, persist_directory="./db")
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})</p></li>
<li><p>LLM (local, offline)
llm = Ollama(model="llama3", temperature=0)</p></li>
<li><p>Prompt and chain
template = "Context: {context}\nQuestion: {question}\nAnswer:"
prompt = ChatPromptTemplate.from_template(template)</p></li>
</ol>
<p>def format_docs(docs):
return "\n\n".join(doc.page_content for doc in docs)
rag_chain = (
{"context": retriever | format_docs, "question": RunnablePassthrough()}
| prompt
| llm
| StrOutputParser()
)
<ol>
<li>Query
print(rag_chain.invoke("What is the main topic of the document?"))
Setup commands for local environment:
Install Ollama curl -fsSL https://ollama.ai/install.sh | sh Linux/macOS Or visit https://ollama.ai for Windows Pull required models ollama pull llama3 ollama pull nomic-embed-text Verify ollama run llama3 "Hello, world!"
7. Production Hardening — Security, Observability, and Evaluation
API Security: Never hardcode API keys. Use environment variables:
export OPENAI_API_KEY="your-key" export LANGCHAIN_API_KEY="your-langsmith-key"
Observability with LangSmith:
import os os.environ["LANGCHAIN_TRACING_V2"] = "true" os.environ["LANGCHAIN_API_KEY"] = "your-key" All chain executions are now traced and debuggable
Evaluation metrics:
- Precision@k: Fraction of retrieved chunks that are relevant
- Recall@k: Fraction of relevant chunks that were retrieved
- Faithfulness: Whether the generated answer is grounded in the retrieved context
- Answer relevance: Whether the answer directly addresses the question
Hallucination mitigation: RAG reduces hallucinations by grounding responses in retrieved facts. For maximum reliability, implement:
– Citation tracking: Include source references in responses
– Faithfulness verification: Use a separate LLM to check if the answer is supported by the retrieved context
– Confidence scoring: Reject answers when retrieved context is insufficient
What Undercode Say:
- RAG is not optional for production AI — Standalone LLMs hallucinate; RAG grounds responses in verifiable data, making it the minimum viable architecture for any serious AI application
- Chunking is the most underestimated hyperparameter — Poor chunking destroys retrieval quality regardless of how good your embedding model or LLM is; invest time in semantic chunking strategies
- Production RAG requires hybrid search — Vector embeddings alone miss exact keyword matches; combine dense retrieval with sparse (BM25) for comprehensive coverage
- Observability is non‑negotiable — Without tracing and evaluation, you’re flying blind; LangSmith or similar tools are essential for debugging and optimization
Analysis: The RAG landscape has evolved dramatically since 2020. We’ve moved from naive retrieval to advanced modular pipelines, and now to agentic RAG where AI agents autonomously plan, retrieve, validate, and refine. The core insight remains: LLMs are powerful reasoning engines, but they need external memory to be reliable. As of 2026, hybrid search with reranking is the production standard, and GraphRAG is emerging for complex reasoning tasks. Engineers who master the full pipeline—from chunking strategy to retrieval optimization to generation—will build AI systems that enterprises actually trust.
Prediction:
- +1 RAG will become the default architecture for 90%+ of enterprise AI applications by 2027, displacing standalone fine‑tuning as the primary method for domain‑specific knowledge integration
- +1 Agentic RAG systems that autonomously plan retrieval strategies and self‑correct will replace static pipelines, boosting accuracy by 35%+ in complex reasoning tasks
- +1 Open‑source, fully local RAG stacks (Ollama + ChromaDB + HuggingFace) will mature to production grade, eliminating API dependency and data privacy concerns for regulated industries
- -1 Organizations that treat RAG as a “plug‑and‑play” solution without investing in retrieval evaluation and chunking optimization will see hallucination rates above 20%, eroding user trust
- -1 The compute cost of production RAG (embedding generation + vector search + LLM inference) will remain a barrier for small teams, driving consolidation toward specialized RAG‑as‑a‑service providers
▶️ Related Video (82% Match):
🎯Let’s Practice For Free:
🎓 Live Courses & Certifications:
Join Undercode Academy for Verified Certifications
🚀 Request a Custom Project:
Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands
IT/Security Reporter URL:
Reported By: Ayushi Shukla – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅
