Building Production-Ready RAG Systems with LangChain & ChromaDB

The advent of large language models (LLMs) has revolutionized how we interact with information, opening up unprecedented possibilities for automation, content generation, and intelligent assistance. However, as powerful as models like GPT-4 or Llama 2 are, they come with inherent limitations: they can “hallucinate” (generate factually incorrect information), their knowledge is capped at their training data cutoff, and they often lack domain-specific expertise relevant to a particular organization or use case.

This is where Retrieval-Augmented Generation (RAG) steps in, transforming LLMs from general knowledge engines into highly specialized, factual, and up-to-date information assistants. By grounding LLM responses in external, authoritative knowledge bases, RAG significantly mitigates hallucinations and allows LLMs to interact with the latest, most relevant data.

In this comprehensive guide, we, as fellow engineers at Khadervali.com, will dive deep into building production-ready RAG systems. We’ll leverage two leading open-source tools that have become cornerstones in the LLM application development ecosystem: LangChain for orchestration and ChromaDB as our vector store. Our goal isn’t just to build a prototype, but to understand the architectural considerations, best practices, and implementation details necessary for a system that can reliably perform in a production environment.

By the end of this article, you’ll have a solid understanding of:

• The core principles and components of a RAG system.
• How LangChain simplifies complex LLM workflows.
• The role of ChromaDB as an efficient, lightweight vector database.
• A detailed architectural blueprint for a production RAG system.
• Step-by-step implementation with practical code examples.
• Critical considerations for deploying and maintaining RAG in production.

Understanding Retrieval-Augmented Generation (RAG)

What is RAG and Why Do We Need It?

At its heart, Retrieval-Augmented Generation is a technique that enhances the capabilities of LLMs by giving them access to external, up-to-date, and relevant information. Instead of relying solely on the knowledge embedded during their training, RAG systems first retrieve pertinent documents or data snippets from a specified knowledge base and then provide these retrieved items to the LLM as additional context for generating a response.

Imagine asking an LLM a question about your company’s latest internal policy update. Without RAG, the LLM would likely either hallucinate an answer based on its general training data or state that it doesn’t have the information. With RAG, the system would first search your internal policy documents, find the relevant sections, and then feed those sections to the LLM along with your original question. The LLM can then synthesize an accurate answer, directly citing or referencing the provided context.

The benefits of RAG are profound:

• Reduced Hallucinations: By grounding responses in factual, retrieved data, the LLM is less likely to invent information.
• Access to Current Information: RAG bypasses the training data cutoff, allowing LLMs to answer questions about events or data that occurred after their last training update.
• Domain Specificity: Tailor LLMs to specific domains (e.g., legal, medical, internal company knowledge) by providing domain-specific documents.
• Explainability and Trust: Users can verify responses by examining the source documents that were retrieved, fostering greater trust in the AI system.
• Cost-Effectiveness: Instead of fine-tuning an entire LLM for new knowledge (which is expensive and computationally intensive), RAG allows you to update the knowledge base independently.

The Core Components of a RAG System

A typical RAG system can be broken down into two main phases: the indexing phase (offline) and the retrieval & generation phase (online). Each phase involves several critical components:

1. Data Ingestion/Loading: This is where raw data from various sources (PDFs, websites, databases, plain text files, etc.) is loaded into the system. LangChain provides excellent `DocumentLoaders` for this purpose.

2. Chunking: Raw documents are often too large to fit into an LLM’s context window or to be efficiently searched. They are split into smaller, manageable “chunks” or “passages.” Effective chunking is crucial for retrieval quality.

3. Embedding: Each text chunk is converted into a numerical vector (an embedding) using an embedding model. These embeddings capture the semantic meaning of the text, allowing for similarity searches.

4. Vector Database: The generated embeddings, along with their original text chunks and any associated metadata, are stored in a vector database (like ChromaDB). This database is optimized for fast similarity searches.

5. Retrieval: When a user poses a query, that query is also converted into an embedding. The system then performs a similarity search in the vector database to find the most relevant chunks whose embeddings are closest to the query’s embedding.

6. LLM Integration: The retrieved chunks are combined with the original user query and a prompt template to form a comprehensive prompt for the LLM.

7. Generation: The LLM processes this augmented prompt and generates a coherent, factual response based on the provided context.

LangChain: The Orchestrator of LLM Applications

A Brief Introduction to LangChain

LangChain is a powerful framework designed to simplify the development of applications powered by language models. It provides a modular and extensible architecture that makes it easy to connect LLMs with other data sources and computational agents. Think of LangChain as the glue that binds various LLM components together, enabling complex workflows with relative ease.

Key abstractions in LangChain include:

• LLMs: Integrations with various language models (OpenAI, Hugging Face, Cohere, etc.).
• Prompts: Tools for constructing, optimizing, and managing prompts for LLMs.
• Chains: Sequences of calls to LLMs or other utilities. Chains allow you to combine multiple components into a single application.
• Retrievers: Components responsible for fetching documents from a knowledge base (often a vector store) based on a query.
• Document Loaders: Utilities for loading data from different sources into a standardized document format.
• Text Splitters: Tools for splitting large documents into smaller, manageable chunks.
• Embeddings: Interfaces for various embedding models to convert text into numerical vectors.
• Vector Stores: Integrations with vector databases to store and retrieve document embeddings.
• Agents: More complex systems that use LLMs to decide which actions to take, observe the results, and repeat until a task is complete.

Key LangChain Components for RAG

For building a RAG system, several LangChain components are particularly vital:

• Document Loaders: These are the entry point for your data. Examples include `PyPDFLoader` for PDF files, `WebBaseLoader` for web pages, `CSVLoader`, `DirectoryLoader` (to load all documents from a directory), and many more. They abstract away the specifics of reading different file formats.
• Text Splitters: Once documents are loaded, they need to be broken down. `RecursiveCharacterTextSplitter` is a popular choice, intelligently splitting text by various characters while attempting to keep related content together. The `chunk_size` and `chunk_overlap` parameters are critical here.
• Embeddings: LangChain provides a unified interface for various embedding models. `OpenAIEmbeddings` uses OpenAI’s powerful embedding models, while `HuggingFaceEmbeddings` allows you to leverage a wide range of open-source models hosted on Hugging Face, often suitable for local or cost-sensitive deployments.
• Vector Stores: This is where LangChain integrates with vector databases. The `Chroma` class in LangChain acts as an adapter, allowing you to store and retrieve documents from a ChromaDB instance seamlessly.
• Retrievers: A `VectorStoreRetriever` is the most common retriever for RAG. It takes a query, embeds it, queries the underlying vector store for similar documents, and returns them. LangChain also supports more advanced retrievers like `MultiQueryRetriever` or `ContextualCompressionRetriever`.
• LLMs/ChatModels: LangChain provides interfaces for various LLMs. `ChatOpenAI` is commonly used for OpenAI’s chat models, while `HuggingFaceHub` allows integration with models from Hugging Face.
• Chains: For RAG, the `create_retrieval_chain` function (or older `RetrievalQA` chain) is fundamental. It orchestrates the retrieval of documents, combines them with the user’s question, and sends them to the LLM for a final answer. The `StuffDocumentsChain` is often used internally to fit all retrieved documents into a single prompt.

ChromaDB: Your Local-First Vector Store

Why ChromaDB for RAG?

ChromaDB is an open-source vector database that has gained significant traction due to its ease of use, lightweight nature, and excellent integration with tools like LangChain. It’s an ideal choice for many RAG applications, particularly during development and for production systems that value simplicity and embeddability.

Here’s why ChromaDB stands out:

• Open Source and Free: No licensing costs, active community development.
• Easy to Get Started: It can run entirely in-memory or persist to disk with minimal setup, making it perfect for rapid prototyping and local development.
• Embedded or Client-Server: ChromaDB can run as an embedded database within your Python application (like SQLite) or as a standalone client-server instance, offering flexibility for deployment.
• LangChain Integration: As mentioned, LangChain has first-class support for ChromaDB, streamlining its usage in RAG pipelines.
• Metadata Filtering: Beyond just vector similarity search, ChromaDB allows you to store metadata alongside your embeddings and filter results based on this metadata, significantly improving retrieval precision.
• Scalability Considerations: While often lauded for its ease of use, ChromaDB also offers options for scaling. Its persistent client is great for single-machine deployments, and for distributed or high-throughput needs, it can be run in client-server mode, allowing for more robust deployments. However, for extremely large-scale, distributed production environments, you might eventually look into options like Pinecone, Weaviate, or Qdrant. For many, ChromaDB in its persistent or client-server form is more than sufficient.

ChromaDB Fundamentals

At its core, ChromaDB organizes data into “collections.” A collection is analogous to a table in a relational database and holds your documents, their embeddings, and associated metadata.

• Collections: A named group of embeddings, documents, and metadata. You create a collection to store a specific set of knowledge.
• Documents: The raw text snippets that were chunked from your original data.
• Embeddings: The numerical vector representations of these documents.
• Metadata: Key-value pairs associated with each document and its embedding. This is incredibly powerful for filtering, allowing you to restrict searches to specific document types, authors, dates, or any other relevant attributes.
• Basic Operations:
  • add(): Adds documents, embeddings, and metadata to a collection.
  • get(): Retrieves items by their IDs.
  • query(): The primary method for similarity search. You provide a query embedding or text, and it returns the most similar documents.
  • delete(): Removes items from a collection.

Architectural Blueprint of a Production-Ready RAG System

High-Level Overview

Let’s visualize the architecture of a production-ready RAG system using a simplified, high-level diagram in words:

+-----------------+                      +-----------------------+
|                 |                      |                       |
|   USER QUERY    | --->  WEB/API Layer  |   LANGCHAIN RAG APP   |
|                 |                      | (Orchestration Logic) |
+-----------------+                      +-----------------------+
        ^                                           |
        |                                           | Query Embedding
        |                                           v
        |                                   +-------------------+
        |                                   |                   |
        |      LLM Response                 |     RETRIEVER     |
        | <----------------------------     | (ChromaDB Search) |
        |                                   |                   |
        |                                   +-------------------+
        |                                           |
        |                                           | Retrieved Context
        |                                           v
        |                                   +-------------------+
        |                                   |                   |
        |                                   |        LLM        |
        |                                   | (e.g., GPT-4, Llama2)|
        |                                   |                   |
        |                                   +-------------------+

This flow describes the online inference path. There’s also a critical offline indexing pipeline:

+-----------------+        +---------------------+        +---------------------+
|                 |        |                     |        |                     |
|  RAW DATA SOURCES <----> |  DATA INGESTION     | --->   |  TEXT CHUNKING      |
| (PDFs, docs, web)|        | (LangChain Loaders) |        | (LangChain Splitter)|
+-----------------+        +---------------------+        +---------------------+
                                       |                             |
                                       |                             |
                                       v                             v
                               +-----------------------+        +-----------------------+
                               |                       |        |                       |
                               |  EMBEDDING GENERATION | --->   |  VECTOR STORAGE       |
                               | (LangChain Embeddings)|        | (ChromaDB Collections)|
                               +-----------------------+        +-----------------------+

Detailed Workflow Stages

Phase 1: Data Ingestion & Indexing (Offline Process)

This phase is typically run periodically or whenever your knowledge base updates. It’s about transforming raw data into an optimized format for fast retrieval.

1. Data Sources: Identify and gather all relevant documents. This could be a directory of PDF files, a database of support tickets, a collection of web pages, or internal confluence pages.
2. Document Loading (LangChain Loaders): Use LangChain’s `DocumentLoaders` to read data from these sources. For example, `DirectoryLoader` can load all `.pdf` files from a specified folder using `PyPDFLoader`. Each document is loaded into a LangChain `Document` object, which contains `page_content` (the text) and `metadata` (source, page number, etc.).
3. Text Splitting (LangChain TextSplitters): Large `Document` objects are split into smaller, semantically meaningful chunks. The `RecursiveCharacterTextSplitter` is highly recommended. It tries to split based on paragraphs, then sentences, then words, ensuring context is maintained within chunks as much as possible. Parameters like `chunk_size` (e.g., 500 characters) and `chunk_overlap` (e.g., 50 characters) are crucial for balancing context and LLM token limits.
4. Embedding Generation (LangChain Embeddings): Each text chunk is passed to an embedding model (e.g., OpenAI’s `text-embedding-ada-002`, or a local `sentence-transformers` model). This model transforms the text into a dense numerical vector, capturing its semantic meaning.
5. Vector Storage (ChromaDB): The generated embeddings, along with their original text chunks and any associated metadata (source file, section, etc.), are stored in a ChromaDB collection. ChromaDB indexes these vectors, making them searchable by similarity.

Code Example: Indexing a directory of documents

import os
from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.embeddings import OpenAIEmbeddings, HuggingFaceEmbeddings
from langchain_community.vectorstores import Chroma

# Configuration
DATA_PATH = "data/" # Directory containing your documents
CHROMA_PATH = "chroma_db/" # Path to store ChromaDB persistent client
EMBEDDING_MODEL_NAME = "text-embedding-ada-002" # Or "sentence-transformers/all-MiniLM-L6-v2"

# 1. Load documents
def load_documents(data_path):
    print(f"Loading documents from {data_path}...")
    loader = DirectoryLoader(data_path, glob="*.pdf", loader_cls=PyPDFLoader)
    documents = loader.load()
    print(f"Loaded {len(documents)} documents.")
    return documents

# 2. Split documents into chunks
def split_documents(documents):
    print("Splitting documents into chunks...")
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=500,
        chunk_overlap=50,
        length_function=len,
        add_start_index=True,
    )
    chunks = text_splitter.split_documents(documents)
    print(f"Split into {len(chunks)} chunks.")
    return chunks

# 3. Create embeddings
def create_embeddings(model_name):
    print(f"Creating embeddings with model: {model_name}...")
    if "openai" in model_name.lower():
        # Ensure OPENAI_API_KEY is set in environment variables
        return OpenAIEmbeddings(model=model_name)
    else:
        # For local models, ensure 'sentence-transformers' is installed
        return HuggingFaceEmbeddings(model_name=model_name)

# 4. Store embeddings in ChromaDB
def store_in_chroma(chunks, embeddings_model, chroma_path):
    print(f"Storing {len(chunks)} chunks in ChromaDB at {chroma_path}...")
    # Clear out the database first, if it exists
    if os.path.exists(chroma_path):
        import shutil
        shutil.rmtree(chroma_path)
        print(f"Removed existing ChromaDB at {chroma_path}")

    db = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings_model,
        persist_directory=chroma_path
    )
    db.persist()
    print(f"Finished storing data in ChromaDB. {db._collection.count()} items.")
    return db

def main_indexing():
    # Make sure 'data/' directory exists and contains some PDFs
    # For testing, create a dummy file:
    # with open("data/sample.pdf", "w") as f: f.write("This is a sample PDF content.")
    # (Note: PyPDFLoader needs actual PDF content, this is just for directory creation)

    if not os.path.exists(DATA_PATH):
        os.makedirs(DATA_PATH)
        print(f"Created {DATA_PATH} directory. Please add PDF documents here.")
        return

    documents = load_documents(DATA_PATH)
    if not documents:
        print("No documents found to index. Please add PDFs to the 'data/' directory.")
        return

    chunks = split_documents(documents)
    embeddings = create_embeddings(EMBEDDING_MODEL_NAME)
    vector_db = store_in_chroma(chunks, embeddings, CHROMA_PATH)
    print("Indexing complete.")

if __name__ == "__main__":
    main_indexing()

Phase 2: Retrieval & Generation (Online Process)

This is the real-time interaction phase, where user queries are processed to generate responses.

1. User Query Input: A user submits a question or prompt through a web interface, chatbot, or API.
2. Query Embedding: The user’s query is also converted into a numerical vector using the *same embedding model* that was used during the indexing phase. This consistency is crucial for accurate similarity search.
3. Similarity Search (ChromaDB): The query embedding is used to perform a similarity search (e.g., cosine similarity) against all the document embeddings stored in ChromaDB. The top N most similar chunks are retrieved.
4. Context Assembly: The retrieved text chunks are combined with the original user query and a predefined system prompt (e.g., “Answer the question based only on the provided context.”) into a single, cohesive prompt. LangChain’s chains handle this concatenation.
5. Prompt Engineering (LangChain): The system prompt is carefully crafted to guide the LLM’s behavior, instructing it on how to use the context, what to do if information is missing, and the desired output format.
6. LLM Call (LangChain): The assembled prompt is then sent to the chosen LLM (e.g., OpenAI’s GPT-4, Llama 2 via Hugging Face).
7. Response Generation: The LLM generates a response based on the provided context and the prompt instructions. This response is then returned to the user.

Building a Production-Ready RAG System: Step-by-Step Implementation

Now, let’s put it all together. We’ll build a simple RAG system that can answer questions based on a local directory of PDF documents.

Setup and Dependencies

First, ensure you have the necessary libraries installed. We’ll use LangChain for orchestration, ChromaDB for vector storage, OpenAI for embeddings and the LLM (you can swap this for HuggingFace if preferred), and PyPDF to read PDFs.

pip install langchain langchain-community langchain-openai chromadb pypdf

You’ll also need an OpenAI API key, which should be set as an environment variable:

export OPENAI_API_KEY="your_openai_api_key_here"

Or, for HuggingFace models:

pip install sentence-transformers # For local embeddings
export HUGGINGFACEHUB_API_TOKEN="your_huggingface_api_token_here" # For HuggingFace LLMs

Data Preparation & Indexing

We’ll reuse the indexing logic from the previous section. Make sure you have a `data/` directory with some PDF files. For example, you could download a few public research papers or policy documents. Let’s assume you have a file like `data/company_policy.pdf` or `data/product_manual.pdf`.

Run the `main_indexing()` function from the previous code block once to prepare your vector database. This will create a `chroma_db/` directory containing your indexed data.

# (Assuming the main_indexing() function from the previous section is available)
# Run this once to populate your ChromaDB
# python your_indexing_script.py

Querying and Generation

Now, let’s write the code to query our indexed data and get responses from an LLM.

import os

from langchain_community.vectorstores import Chroma

from langchain_community.embeddings import OpenAIEmbeddings, HuggingFaceEmbeddings

from langchain.chat_models import ChatOpenAI # For OpenAI LLMs

from langchain_core.prompts import ChatPromptTemplate

from langchain.chains.combine_documents import create_stuff_documents_chain

from langchain.chains import create_retrieval_chain

# Configuration (must match indexing script)

CHROMA_PATH = "chroma_db/"

EMBEDDING_MODEL_NAME = "text-embedding-ada-002"

LLM_MODEL_NAME = "gpt-4-turbo-preview" # Or "gpt-3.5-turbo"
# 1. Initialize Embeddings (must match indexing setup)

def get_embeddings(model_name):

    if "openai" in model_name.lower():

        return OpenAIEmbeddings(model=model_name)

    else:

        return HuggingFaceEmbeddings(model_name=model_name)
# 2. Load the Vector Store

def load_vector_store(chroma_path, embeddings_model):

    print(f"Loading ChromaDB from {chroma_path}...")

    db = Chroma(persist_directory=chroma_path, embedding_function=embeddings_model)

    print(f"ChromaDB loaded with {db._collection.count()} items.")

    return db
# 3. Setup the LLM

def setup_llm(model_name):

    print(f"Setting up LLM: {model_name}...")

    if "gpt" in model_name.lower():

        # Ensure OPENAI_API_KEY is set

        return ChatOpenAI(model=model_name, temperature=0.1) # Lower temp for more factual

    else:

        # For HuggingFace, ensure HUGGINGFACEHUB_API_TOKEN is set

        # And choose a