Stop Building RAG Systems Like It’s 2023 — The 10‑Step Pipeline That Actually Works + Video

By /

Listen to this Post

Featured Image

Introduction:

Retrieval-Augmented Generation (RAG) has become the backbone of enterprise AI, yet most implementations fail because teams treat it as a simple “vector database + LLM” exercise. The reality is far more nuanced: a production‑grade RAG pipeline involves ten distinct stages — from intelligent data ingestion and semantic chunking to hybrid search, cross‑encoder reranking, and continuous evaluation. Getting each step right is the difference between a system that hallucinates and one that delivers reliable, grounded answers at scale.

Learning Objectives:

  • Understand the complete end‑to‑end RAG pipeline — from data ingestion to response validation
  • Implement hybrid search strategies that combine dense vector retrieval with BM25 keyword matching
  • Apply cross‑encoder reranking to boost retrieval precision from top‑100 to top‑10 candidates
  • Master chunking strategies, metadata extraction, and evaluation metrics for production RAG systems
  1. Data Ingestion and Document Preparation — The Foundation of Any RAG System

Most RAG projects fail before the first embedding is generated — because they neglect the data layer. Ingestion isn’t just about pointing to a folder of PDFs; it’s about building a repeatable, scalable pipeline that handles structured (databases, APIs) and unstructured (websites, Word files, emails) data sources. The goal is to prepare a unified knowledge base that can be refreshed incrementally as new information arrives.

Step‑by‑step guide:

  1. Identify all data sources — internal wikis, Confluence, SharePoint, Salesforce, public documentation, and REST APIs.
  2. Build extractors for each format: `PyPDF2` or `pypdf` for PDFs, `python-docx` for Word, `BeautifulSoup` for HTML, and `requests` for REST APIs.
  3. Normalise text — strip headers/footers, resolve Unicode characters, and remove boilerplate (copyright notices, page numbers).
  4. Implement deduplication — use MinHash or simple exact‑hash comparison to eliminate near‑duplicate documents before they enter the pipeline.
  5. Set up a change‑detection mechanism — monitor file timestamps or API versioning to trigger re‑ingestion only when content actually changes.

Linux / Windows commands for automation:

 Linux: watch a directory for new PDFs and trigger ingestion
inotifywait -m /data/ingest -e create -e moved_to |
while read path action file; do
python ingest.py --file "$path/$file"
done

 Windows PowerShell: monitor a folder and log changes
$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = "C:\Data\Ingest"
$watcher.Filter = ".pdf"
Register-ObjectEvent $watcher "Created" -Action { python ingest.py --file $Event.SourceEventArgs.FullPath }
  1. Chunking and Metadata Extraction — Where Precision Is Won or Lost

Chunking is the most underestimated step in the RAG pipeline. Split too small, and you lose context; split too large, and you dilute semantic signal. The industry standard starts with 512‑token chunks with 10–15% overlap (about 50 tokens), then tunes based on your specific document corpus. Overlap ensures that a key sentence isn’t severed at a chunk boundary. Beyond size, you must extract rich metadata: titles, authors, publication dates, section headings, and custom tags. This metadata becomes a filter that dramatically improves retrieval precision.

Step‑by‑step guide:

  1. Choose a chunking strategy — fixed‑size token splitting (simplest), sentence‑aware splitting (better for prose), or semantic splitting (most advanced, using embedding similarity to find natural break points).
  2. Set initial parameters — chunk_size = 512, `chunk_overlap = 50` (about 10%).
  3. Extract metadata during chunking — for each chunk, attach source_file, page_number, chapter, chunk_index, and any domain‑specific tags.
  4. Store metadata alongside embeddings in your vector database so you can apply filters like `WHERE author = “Smith” AND year > 2020` during search.
  5. Benchmark different chunk sizes — run retrieval tests with 256, 512, and 1024 tokens to see which yields the highest `recall@k` for your use case.

Python snippet using LangChain:

from langchain.text_splitter import RecursiveCharacterTextSplitter

text_splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
length_function=len,
separators=["\n\n", "\n", " ", ""]
)
chunks = text_splitter.split_documents(documents)

Attach metadata
for i, chunk in enumerate(chunks):
chunk.metadata["chunk_index"] = i
chunk.metadata["source"] = source_file_name
  1. Building the Knowledge Index — Vector Databases and Embedding Models

Once chunks are prepared, you need to generate embeddings and store them in a searchable index. The choice of embedding model matters: sentence‑transformers (all‑MiniLM‑L6‑v2), OpenAI (text‑embedding‑ada‑002), or Cohere each have different latency‑cost‑quality trade‑offs. For the vector database, options range from ChromaDB (lightweight, local) to Pinecone (managed, cloud) and pgvector (PostgreSQL extension). The index must support fast approximate nearest neighbour (ANN) search and, crucially, metadata filtering.

Step‑by‑step guide:

  1. Select your embedding model — for open‑source, use sentence-transformers/all-MiniLM-L6-v2; for cloud, use OpenAI’s text-embedding-3-small.
  2. Choose a vector database — ChromaDB for prototyping, Pinecone or Weaviate for production, pgvector if you’re already on PostgreSQL.
  3. Generate embeddings for each chunk and store them with their metadata and the original text.
  4. Create indexes — enable HNSW (Hierarchical Navigable Small World) or IVF (Inverted File) indexes for sub‑second latency.
  5. Implement an update strategy — when source documents change, recompute embeddings for affected chunks and update the index incrementally.

Linux command to monitor index size:

 For ChromaDB, check collection count
curl -s http://localhost:8000/api/v1/collections | jq '.[].metadata'
  1. Connect APIs and Design Prompts — Orchestrating the Retrieval‑Generation Loop

This stage bridges retrieval and generation. You need to integrate your vector database with an LLM via an orchestration framework like LangChain or LlamaIndex. More importantly, you must design system prompts that instruct the LLM how to use the retrieved context. Zero‑shot prompts work for simple Q&A; few‑shot prompts (with examples) improve consistency for complex tasks. The prompt should explicitly tell the model: “Only answer based on the provided context. If the context doesn’t contain the answer, say ‘I don’t know.’”

Step‑by‑step guide:

  1. Set up the orchestrator — install LangChain or LlamaIndex and connect to your vector database.
  2. Define the retrieval chain — retriever = vector_store.as_retriever(search_kwargs={"k": 5}).
  3. Craft a system prompt — include clear instructions about grounding, citation, and refusal.
  4. Add a re‑ranking step (see Section 5) between retrieval and generation to feed only the most relevant chunks into the LLM.
  5. Test with edge cases — ambiguous queries, out‑of‑domain questions, and multi‑part questions to refine your prompt.

Python snippet (LangChain):

from langchain.chains import RetrievalQA
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o", temperature=0)
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=vector_store.as_retriever(search_kwargs={"k": 5})
)
response = qa_chain.invoke("What is the refund policy?")
  1. Rewrite User Queries — The Hidden Lever for Better Retrieval

Users rarely ask perfect questions. They use ambiguous terms, omit context, or ask multi‑part questions that confuse the retriever. Query rewriting — also called query expansion — transforms the original question into one or more better‑formed queries. Techniques include: expanding acronyms, adding synonyms, breaking complex questions into sub‑questions, and even generating hypothetical answers (HyDE) to retrieve documents that would contain those answers.

Step‑by‑step guide:

  1. Detect ambiguity — use an LLM to identify unclear terms and ask for clarification, or automatically expand them.
  2. Implement multi‑query retrieval — generate 3–5 reformulations of the original question and retrieve chunks for each, then deduplicate.
  3. Use HyDE (Hypothetical Document Embeddings) — generate a hypothetical answer first, embed that, and use it for retrieval (works well for fact‑based questions).
  4. Log query‑rewrite pairs — build a dataset to fine‑tune your rewriter over time.

  5. Perform Hybrid Search — Combine Semantic and Keyword Signals

Vector search alone misses exact‑match terms (e.g., error codes, product IDs). Keyword search alone misses synonyms and conceptual relationships. Hybrid search combines both: it runs a dense vector search (semantic) and a sparse keyword search (BM25 or TF‑IDF) in parallel, then fuses the results using Reciprocal Rank Fusion (RRF). This gives you the best of both worlds — documents that are semantically relevant and contain the exact terms the user asked for.

Step‑by‑step guide:

  1. Enable BM25 indexing in your vector database (Pinecone, Weaviate, and Milvus support this natively).
  2. Set up parallel retrieval — run vector search (top‑100) and keyword search (top‑100) simultaneously.
  3. Apply RRF — combine scores using `1/(k + rank)` where `k` is a constant (usually 60).
  4. Return the fused top‑k (e.g., top‑10) to the reranker.
  5. Monitor hybrid search performance — track whether adding keyword search improves `recall@k` for your test queries.

Python snippet for manual RRF:

def reciprocal_rank_fusion(vector_results, keyword_results, k=60):
scores = {}
for rank, doc in enumerate(vector_results):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
for rank, doc in enumerate(keyword_results):
scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank + 1)
return sorted(scores.items(), key=lambda x: x[bash], reverse=True)

7. Rerank Retrieved Results — The Precision Booster

After hybrid search returns 50–100 candidates, you apply a cross‑encoder reranker — a more powerful (but slower) model that scores each (query, document) pair jointly. Unlike bi‑encoders (which produce static embeddings), cross‑encoders can model fine‑grained relevance and are much more accurate. The classic pattern is: retrieve top‑100 with a fast bi‑encoder, rerank top‑10 with a cross‑encoder. Popular cross‑encoder models include `cross-encoder/ms-marco-MiniLM-L-6-v2` and zenlm/zen-reranker.

Step‑by‑step guide:

  1. Install a cross‑encoder library — pip install sentence-transformers.
  2. Load a pre‑trained cross‑encoder — model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2").
  3. For each query, take the top‑100 candidates from hybrid search and compute cross‑encoder scores.
  4. Sort by score and keep only the top‑5 to top‑10 chunks.
  5. Pass these reranked chunks to the LLM for generation.
  6. Benchmark — compare answer quality with and without reranking; you should see a noticeable lift in factual accuracy.

Python snippet:

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")
pairs = [[query, doc.page_content] for doc in top_100_docs]
scores = reranker.predict(pairs)
top_10_docs = [doc for _, doc in sorted(zip(scores, top_100_docs), reverse=True)[:10]]
  1. Generate and Validate Responses — Reduce Hallucinations at the Source

With the top‑reranked chunks in hand, you now generate the final answer. But generation isn’t the end — you must validate the output. Techniques include: fact‑checking against the source chunks (using an LLM to verify each claim), self‑consistency (generate multiple answers and pick the most consistent), and confidence scoring (ask the LLM to rate its own confidence). The goal is to catch hallucinations before they reach the user.

Step‑by‑step guide:

  1. Generate the answer using your LLM with the reranked context.
  2. Run a fact‑checking pass — prompt the LLM: “Does the answer contain any information not supported by the provided context? If yes, rewrite.”
  3. Add citations — for each sentence, reference the source chunk ID.
  4. Implement a fallback — if the LLM’s confidence is below a threshold, return “I couldn’t find a reliable answer.”
  5. Log all generations — build a dataset for offline evaluation.

9. Evaluate and Optimize — The Never‑Ending Cycle

A RAG system is never “done.” You must continuously measure retrieval quality and answer quality using a suite of metrics. Key metrics include: Context Relevance (are the retrieved chunks relevant?), Faithfulness (is the answer grounded in the context?), and Answer Relevance (does the answer address the query?). Tools like RAGAS and AgentEval provide off‑the‑shelf evaluation frameworks.

Step‑by‑step guide:

  1. Create a test set — collect 100–500 real user queries with ground‑truth answers.
  2. Run your pipeline on the test set and compute metrics (recall@k, MRR, faithfulness).
  3. A/B test changes — compare chunk sizes, embedding models, and rerankers.
  4. Monitor production — log query‑response pairs and periodically re‑evaluate.
  5. Iterate — use the insights to tune chunking, prompts, and retrieval parameters.

Linux command to run evaluation batch:

for query in $(cat test_queries.txt); do
python evaluate.py --query "$query" >> eval_results.log
done

What Undercode Say:

  • “Chunking is not a one‑size‑fits‑all parameter — you must experiment with size, overlap, and strategy for your specific document types.” Many teams blindly use 512 tokens and wonder why retrieval fails. The reality is that legal contracts, technical manuals, and conversational data each demand different chunking approaches.
  • “Hybrid search + cross‑encoder reranking is the secret sauce of production RAG.” Vector‑only systems miss exact matches; keyword‑only systems miss synonyms. Combining them with RRF, then applying a cross‑encoder, routinely lifts retrieval precision by 20–30%.
  • “Evaluation is not optional — if you can’t measure it, you can’t improve it.” The most sophisticated pipeline is useless without continuous, automated evaluation against a representative test set.
  • “Query rewriting is underrated.” Most users ask vague or malformed questions. A simple query‑rewriting step can dramatically improve retrieval quality without changing anything else in the pipeline.
  • “Start simple, then optimise.” Begin with a basic pipeline (ingest → chunk → embed → retrieve → generate). Only after it works end‑to‑end should you add hybrid search, reranking, and query rewriting. Premature optimisation is the enemy of a working RAG system.

Prediction:

  • +1 RAG will become the default interface for enterprise knowledge management within 24 months, displacing traditional search and FAQ systems.
  • +1 Open‑source embedding models and cross‑encoders will continue to close the gap with proprietary APIs, making high‑quality RAG accessible to every organisation.
  • -1 The complexity of production RAG — chunking, hybrid search, reranking, evaluation — will create a significant skills gap, leading to many failed or underperforming deployments.
  • -1 Without robust evaluation and monitoring, RAG systems will amplify biases and hallucinations, eroding trust in AI‑powered knowledge tools.
  • +1 Agentic RAG — where the system decides when and how to retrieve — will emerge as the next evolution, moving beyond the static “retrieve‑then‑generate” pattern.

▶️ Related Video (80% Match):

🎯Let’s Practice For Free:

🎓 Live Courses & Certifications:

Join Undercode Academy for Verified Certifications

🚀 Request a Custom Project:

Secure, high-velocity infrastructure and disruptive technological engineering. Contact our engineering team for high-tier development and proprietary systems:
[email protected]
💎 Smart Architecture | 🛡️ Secure by Design | ⭐ Trusted by Thousands

IT/Security Reporter URL:

Reported By: Thescholarbaniya Most – Hackers Feeds
Extra Hub: Undercode MoN
Basic Verification: Pass ✅

🔐JOIN OUR CYBER WORLD [ CVE News • HackMonitor • UndercodeNews ]

💬 Whatsapp | 💬 Telegram

📢 Follow UndercodeTesting & Stay Tuned:

𝕏 formerly Twitter 🐦 | @ Threads | 🔗 Linkedin | 🦋BlueSky

Related Posts: