Building Production-Ready RAG Systems with LangChain & ChromaDB

As senior software engineers, we’re constantly seeking to leverage cutting-edge technologies to solve complex problems and deliver tangible value. In the rapidly evolving landscape of Large Language Models (LLMs), one of the most impactful architectures to emerge is Retrieval-Augmented Generation (RAG). RAG systems empower LLMs with up-to-date, domain-specific, and factual information, drastically reducing hallucinations and extending their knowledge beyond their training cut-off dates.

At Khadervali.com, we believe in sharing practical, production-oriented insights. This comprehensive guide will walk you through the journey of building a robust, scalable, and production-ready RAG system using two powerful open-source tools: LangChain for orchestrating the entire pipeline and ChromaDB as your efficient, persistent vector store. We’ll dive deep into architecture, code examples, advanced techniques, and crucial real-world considerations for deploying these systems.

By the end of this article, you’ll have a solid understanding of how to:

• Architect a RAG system from data ingestion to response generation.
• Implement data loading, chunking, embedding, and storage with LangChain and ChromaDB.
• Build intelligent retrieval mechanisms, including advanced filtering.
• Integrate LLMs effectively for context-aware generation.
• Address critical production concerns like scalability, performance, and monitoring.

Understanding Retrieval-Augmented Generation (RAG)

The allure of LLMs is their ability to understand and generate human-like text across a vast array of topics. However, they come with inherent limitations:

• Knowledge Cut-off: LLMs are trained on datasets up to a certain point in time, meaning they lack knowledge of recent events or proprietary information.
• Hallucinations: Without access to specific facts, LLMs can confidently generate plausible but incorrect information.
• Lack of Specificity: General-purpose LLMs often struggle with highly specialized domain questions.

RAG directly addresses these challenges by augmenting the LLM’s inherent knowledge with external, relevant information retrieved from a trusted source. Instead of trying to “teach” the LLM new facts through fine-tuning (which is expensive and difficult to update), RAG provides the LLM with the necessary context at the time of query.

How RAG Works: The Core Components

A typical RAG system operates in two main phases:

1. Indexing/Data Ingestion Phase (Offline/Batch): This is where your external knowledge base is prepared.

   • Data Loading: Documents (PDFs, web pages, databases, plain text) are loaded from various sources.
   • Text Splitting/Chunking: Large documents are broken down into smaller, manageable chunks. This is crucial because LLMs have token limits, and smaller chunks lead to more precise retrieval.
   • Embedding: Each text chunk is converted into a numerical vector (an embedding) by an embedding model. These embeddings capture the semantic meaning of the text.
   • Vector Store: The embeddings, along with their original text chunks and any associated metadata, are stored in a vector database (like ChromaDB) for efficient similarity search.

2. Retrieval & Generation Phase (Online/Real-time): This occurs when a user submits a query.

   • Query Embedding: The user’s query is also converted into an embedding using the same embedding model used during indexing.
   • Retrieval: The query embedding is used to perform a similarity search in the vector store. The vector store returns the top-N most semantically similar text chunks (context) from the indexed knowledge base.
   • Prompt Construction: The retrieved context is then combined with the user’s original query and a system prompt, forming a comprehensive prompt for the LLM.
   • Generation: The LLM receives this augmented prompt and generates a response based on its general knowledge and the provided context.

Conceptual Diagram (in words):

[Data Sources] --> [LangChain Document Loaders] --> [LangChain Text Splitters] --> [Embedding Model] --> [ChromaDB (Vector Store)]
        ^                                                                                             |
        |                                                                                             |
    [User Query] --> [Embedding Model] --> [ChromaDB (Retrieval)] --> [Relevant Context] --> [LLM] --> [Generated Response]

This architecture is incredibly powerful because it keeps the LLM’s core model static while dynamically updating its knowledge through the external vector store. This makes RAG systems more agile, cost-effective, and accurate for domain-specific applications.

<

>

LangChain: The Orchestration Layer

LangChain is an open-source framework designed to simplify the development of applications powered by LLMs. It provides a structured way to connect LLMs with external data sources, computation, and agents. For RAG systems, LangChain is an invaluable tool, acting as the glue that binds all the components together.

Key LangChain Modules for RAG

• Document Loaders: Abstractions for loading data from various sources (e.g., PDFs, CSVs, web pages, databases, Notion, etc.) into a standardized Document object format.

• Text Splitters: Tools for breaking down large Document objects into smaller, semantically meaningful chunks. This is critical for effective retrieval.

• Embeddings: Interfaces to different embedding models (e.g., OpenAI, HuggingFace, Cohere) that convert text into numerical vectors.

• Vector Stores: Integrations with various vector databases (like ChromaDB, Pinecone, Weaviate) to store and retrieve document embeddings.

• Retrievers: Components that abstract the logic of fetching relevant documents from a vector store based on a query. They can be simple similarity searchers or more complex with metadata filtering, re-ranking, etc.

• Chains: Sequential or complex logic flows that combine LLMs with other components (like retrievers) to perform specific tasks, such as question-answering over documents.

• LLMs: Interfaces to various LLM providers (OpenAI, Anthropic, Google, HuggingFace Hub) for text generation.

LangChain’s modular design allows you to swap out components easily, experiment with different models or vector stores, and build complex workflows without reinventing the wheel.

ChromaDB: Your Persistent Vector Store

ChromaDB is an open-source embedding database designed to make it easy to build LLM applications. It’s lightweight, developer-friendly, and offers powerful features for storing and querying embeddings. For many RAG applications, especially those starting out or requiring a local-first approach, ChromaDB is an excellent choice.

Why Choose ChromaDB for RAG?

• Simplicity and Ease of Use: ChromaDB is incredibly straightforward to set up and use. You can run it entirely in-memory, locally persistent, or in a client-server mode.

• Persistence: Unlike purely in-memory solutions, ChromaDB can persist your embeddings and documents to disk, meaning your knowledge base isn’t lost when your application restarts.

• Metadata Filtering: Beyond similarity search, ChromaDB allows you to store rich metadata alongside your embeddings and use this metadata to filter search results. This is crucial for precise retrieval in production systems (e.g., “Find documents about ‘LangChain’ published after 2023 from ‘blog’ source”).

• Open Source: Being open source, it offers transparency and flexibility, with a growing community.

• LangChain Integration: ChromaDB has first-class integration with LangChain, making it seamless to use within your RAG pipelines.

• Scalability (for its tier): While not designed for petabyte-scale data like some enterprise vector databases, ChromaDB handles millions of embeddings efficiently, making it suitable for a wide range of production RAG applications.

ChromaDB Deployment Options

• Local/Persistent Client: This is the simplest mode. ChromaDB runs embedded within your Python application, persisting data to a local directory. Great for development, testing, and smaller production applications.

• Client-Server (HTTP API): For more robust production deployments, you can run ChromaDB as a separate server accessible via an HTTP API. This allows multiple applications to connect to the same vector store, offers better resource management, and simplifies scaling the vector store independently.

Architectural Blueprint of a Production RAG System

Let’s map out the detailed architecture, focusing on how LangChain and ChromaDB fit into the two main phases of a production-ready RAG system.

Phase 1: Data Ingestion Pipeline (Offline/Batch)

This pipeline is responsible for preparing your knowledge base. It’s typically run periodically (e.g., daily, weekly) or triggered by new data arrival.

Components:

1. Data Sources: Your raw information. This could be anything: documentation in PDF, Markdown files, web scraped content, internal databases, Confluence wikis, ticketing systems, etc.

2. LangChain Document Loaders: LangChain provides a vast array of loaders. For instance, PyPDFLoader for PDFs, WebBaseLoader for URLs, DirectoryLoader for local files, and many others for specific platforms like Notion, SharePoint, etc.

3. LangChain Text Splitters: After loading, large documents need to be split. The choice of splitter is crucial. RecursiveCharacterTextSplitter is a good general-purpose choice, attempting to split by paragraphs, then sentences, then words, to keep chunks semantically coherent. More advanced techniques like semantic chunking (using an LLM to determine chunk boundaries) or fixed-size chunks with overlap are also possible.

4. Embedding Model: This model converts each text chunk into a high-dimensional vector. Popular choices include:

   • OpenAI’s text-embedding-ada-002 (or newer models) for strong performance.
   • HuggingFace embeddings (e.g., Sentence-BERT models like all-MiniLM-L6-v2) for open-source, often self-hosted solutions.

   The choice impacts accuracy and cost/latency.

5. ChromaDB Client: This component interacts with the ChromaDB instance. It creates collections (analogous to tables in a relational database), adds documents (text chunks, embeddings, and metadata), and ensures persistence.

Ingestion Process Flow:

[Raw Data] --> [Document Loader] --> [Raw Documents] --> [Text Splitter] --> [Text Chunks] --> [Embedding Model] --> [Embeddings] --> [ChromaDB (Collection)]
                                                                               (with associated metadata)

Code Example: Data Ingestion Script

First, ensure you have the necessary libraries installed:

pip install langchain chromadb openai pypdf python-dotenv

Then, set up your environment variables, e.g., in a .env file:

OPENAI_API_KEY="your_openai_api_key_here"

Now, the Python script:

import os
from dotenv import load_dotenv
from langchain_community.document_loaders import PyPDFLoader, TextLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma

# Load environment variables
load_dotenv()

# Configuration
DATA_PATH = "./data/" # Directory to store your documents
CHROMA_PERSIST_DIRECTORY = "./chroma_db/"
COLLECTION_NAME = "my_rag_knowledge_base"
EMBEDDING_MODEL_NAME = "text-embedding-ada-002" # Or a local HF model

def load_and_chunk_documents(data_dir: str):
    """Loads documents from a directory and chunks them."""
    documents = []
    for root, _, files in os.walk(data_dir):
        for file in files:
            file_path = os.path.join(root, file)
            print(f"Loading {file_path}...")
            if file.endswith(".pdf"):
                loader = PyPDFLoader(file_path)
            elif file.endswith(".txt"):
                loader = TextLoader(file_path)
            # Add more loaders as needed (e.g., CSVLoader, WebBaseLoader)
            else:
                print(f"Skipping unsupported file type: {file}")
                continue
            documents.extend(loader.load())

    # Split documents into chunks
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=1000,
        chunk_overlap=200,
        length_function=len,
        add_start_index=True,
    )
    chunks = text_splitter.split_documents(documents)
    print(f"Split {len(documents)} documents into {len(chunks)} chunks.")
    return chunks

def embed_and_store_in_chroma(chunks, persist_directory: str, collection_name: str):
    """Embeds document chunks and stores them in ChromaDB."""
    # Initialize embedding model
    embeddings = OpenAIEmbeddings(model=EMBEDDING_MODEL_NAME)

    # Create ChromaDB client and add documents
    print(f"Creating/Loading ChromaDB collection '{collection_name}' at {persist_directory}...")
    db = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_directory,
        collection_name=collection_name
    )
    db.persist()
    print("ChromaDB ingestion complete.")
    return db

if __name__ == "__main__":
    # 1. Prepare data directory (create some dummy files for testing)
    os.makedirs(DATA_PATH, exist_ok=True)
    with open(os.path.join(DATA_PATH, "langchain_overview.txt"), "w") as f:
        f.write("LangChain is a framework for developing applications powered by large language models. "
                "It enables applications to be context-aware and to reason. "
                "LangChain provides abstractions for working with LLMs, chains, agents, and more. "
                "It supports various document loaders, text splitters, embedding models, and vector stores. "
                "Key components include document loaders, text splitters, vectorstores, retrievers, and chains. "
                "LangChain's modular design makes it easy to build complex LLM applications.")
    with open(os.path.join(DATA_PATH, "chromadb_features.txt"), "w") as f:
        f.write("ChromaDB is an open-source embedding database. It's lightweight and easy to use. "
                "Features include persistence, metadata filtering, and strong integration with LangChain. "
                "You can run ChromaDB in local mode or client-server mode. "
                "It's great for storing embeddings and their associated documents for RAG. "
                "ChromaDB allows you to query for similar documents based on their semantic meaning. "
                "It supports adding and deleting documents dynamically.")
    
    # Ensure a PDF is present if you want to test PyPDFLoader
    # You'd typically place a PDF file in the 'data' directory.
    # For example, download a sample PDF and name it 'sample.pdf' in the data directory.
    print("Please ensure you have .txt files or .pdf files in the './data/' directory for a full test.")

    # 1. Load and Chunk Documents
    document_chunks = load_and_chunk_documents(DATA_PATH)

    # 2. Embed and Store in ChromaDB
    vector_db = embed_and_store_in_chroma(document_chunks, CHROMA_PERSIST_DIRECTORY, COLLECTION_NAME)

    print(f"Knowledge base with {vector_db._collection.count()} chunks successfully built and persisted.")

Phase 2: Retrieval & Generation Pipeline (Online/Real-time)

This pipeline is triggered by a user query and runs in real-time to generate a context-aware response.

Components:

1. User Query: The natural language question or prompt from the user.

2. Embedding Model: The same embedding model used during ingestion is crucial here to ensure semantic consistency when comparing the query to document chunks.

3. ChromaDB Retrieval: The query embedding is used to search the ChromaDB collection. ChromaDB returns the top-K most similar document chunks. This is where advanced filtering using metadata can also be applied.

4. LangChain Retriever Abstraction: LangChain wraps the ChromaDB search into a generic Retriever interface, making it easy to swap out vector stores or add pre/post-processing steps to the retrieval.

5. LLM Integration: The core large language model (e.g., GPT-4, Claude, Llama 2). LangChain provides a unified interface for interacting with various LLMs.

6. Prompt Engineering for RAG: A well-crafted prompt is essential. It instructs the LLM on its role, provides the retrieved context, and asks the user’s question, guiding the LLM to use the provided context effectively.

7. LangChain RAG Chain: LangChain offers specific chains like create_retrieval_chain or the older RetrievalQA that automate the process of taking a query, retrieving documents, and passing them to an LLM for generation.

Retrieval & Generation Process Flow:

[User Query] --> [Embedding Model] --> [Query Embedding] --> [ChromaDB (Retrieval)] --> [Top K Relevant Chunks (Context)] --> [Prompt Engineering] --> [LLM] --> [Generated Response]

Code Example: Querying Script

import os
from dotenv import load_dotenv
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

# Load environment variables
load_dotenv()

# Configuration (must match ingestion script)
CHROMA_PERSIST_DIRECTORY = "./chroma_db/"
COLLECTION_NAME = "my_rag_knowledge_base"
EMBEDDING_MODEL_NAME = "text-embedding-ada-002"
LLM_MODEL_NAME = "gpt-3.5-turbo" # or "gpt-4", etc.

def setup_rag_chain():
    """Sets up the RAG retrieval and generation chain."""
    # Initialize embedding model
    embeddings = OpenAIEmbeddings(model=EMBEDDING_MODEL_NAME)

    # Load existing ChromaDB instance
    print(f"Loading ChromaDB collection '{collection_name}' from {CHROMA_PERSIST_DIRECTORY}...")
    db = Chroma(
        persist_directory=CHROMA_PERSIST_DIRECTORY,
        embedding_function=embeddings,
        collection_name=COLLECTION_NAME
    )

    # Create a retriever
    retriever = db.as_retriever(search_kwargs={"k": 3}) # Retrieve top 3 most relevant documents

    # Initialize LLM
    llm = ChatOpenAI(model=LLM_MODEL_NAME, temperature=0.1)

    # Define prompt template for combining documents with query
    prompt = ChatPromptTemplate.from_messages([
        ("system", "You are a helpful AI assistant. Answer the user's questions based ONLY on the provided context. "
                   "If the answer is not in the context, politely state that you don't have enough information."),
        ("user", "Context: {context}\n\nQuestion: {input}")
    ])

    # Create a chain to combine documents and pass to LLM
    document_combiner_chain = create_stuff_documents_chain(llm, prompt)

    # Create the full retrieval chain
    retrieval_rag_chain = create_retrieval_chain(retriever, document_combiner_chain)

    print("RAG chain setup complete.")
    return retrieval_rag_chain

if __name__ == "__main__":
    rag_chain = setup_rag_chain()

    while True:
        query = input("\nEnter your question (or 'exit' to quit): ")
        if query.lower() == 'exit':
            break

        print("\nSearching knowledge base...")
        try:
            response = rag_chain.invoke({"input": query})
            print("\n--- AI Response ---")
            print(response["answer"])
            print("\n--- Sources ---")
            for doc in response["context"]:
                print(f"- Source: {doc.metadata.get('source', 'Unknown')}, Page: {doc.metadata.get('page', 'N/A')}")
            print("-------------------\n")
        except Exception as e:
            print(f"An error occurred: {e}")

<

>

Advanced Retrieval Techniques for Production

For a truly production-ready RAG system, basic similarity search might not always be enough. Here are some advanced techniques:

Metadata Filtering

Storing metadata (e.g., source, author, date, department) with your chunks allows for precise filtering during retrieval. This is critical for controlling the scope of your RAG system, ensuring privacy, or targeting specific document types.

Example during ingestion (adding metadata):

from langchain_community.document_loaders import TextLoader

from langchain.text_splitter import RecursiveCharacterTextSplitter

from langchain_openai import OpenAIEmbeddings

from langchain_community.vectorstores import Chroma

import os

# Assuming your setup (load_dotenv, etc.)

# ...
# Load documents with specific metadata

doc1_path = os.path.join(DATA_PATH, "internal_policy_2023.txt")

doc2_path = os.path.join(DATA_PATH, "public_faq.txt")
with open(doc1_path, "w") as f:

    f.write("This is an internal company policy document for 2023. It covers vacation days and sick leave.")

with open(doc2_path, "w") as f:

    f.write("Frequently Asked Questions for customers. Includes product warranty and return policies.")
loader1 = TextLoader(doc1_path)

docs1 = loader1.load()

for doc in docs1:

    doc.metadata = {"source": "internal_policy", "year": 2023}
loader2 = TextLoader(doc2_path)

docs2 = loader2.load()

for doc in docs2:

    doc.metadata = {"source": "public_faq", "department": "customer_support"}
all_docs = docs1 + docs2
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)

chunks = text_splitter.split_documents(all_docs)
embeddings = OpenAIEmbeddings(model=EMBEDDING_MODEL_NAME)

db = Chroma.from_documents(

    documents=chunks,

    embedding=embeddings,

    persist_directory=CHROMA_PERSIST_DIRECTORY,

    collection_name=COLLECTION_NAME + "_with_metadata"

)

db.persist()

print("ChromaDB with metadata ingested.")
# Example retrieval with metadata filtering:

print("\nQuery