Khader Vali 
Building Production-Ready RAG with LangChain & ChromaDB
As senior software engineers and technical writers at Khadervali.com, we’re constantly exploring the cutting edge of AI to deliver robust, scalable solutions. In the rapidly evolving landscape of Large Language Models (LLMs), one technique has stood out for its ability to enhance factual accuracy, reduce hallucinations, and incorporate domain-specific knowledge: Retrieval Augmented Generation (RAG).
RAG systems combine the generative power of LLMs with a retrieval mechanism that fetches relevant information from a knowledge base. This allows LLMs to ground their responses in specific, up-to-date data, making them far more reliable for real-world applications. However, moving from a proof-of-concept RAG system to a production-ready solution involves navigating a myriad of challenges, from scalability and performance to observability and cost optimization.
This comprehensive guide will walk you through building a production-ready RAG system using two powerful tools: LangChain for orchestrating the LLM workflow and ChromaDB as our robust vector database. We’ll cover architecture, implementation details with code examples, crucial production considerations, and real-world scenarios to ensure your RAG system is not just functional, but truly ready for prime time.
Understanding Retrieval Augmented Generation (RAG)
Before diving into the implementation, let’s briefly recap what RAG is and why it’s so transformative. Traditional LLMs are trained on vast datasets, but their knowledge is static and limited to their training cutoff. They can also “hallucinate” information, generating plausible but incorrect facts.
RAG addresses these limitations by augmenting the LLM’s generation process with an external knowledge retrieval step. Here’s the core flow:
- User Query: A user asks a question or provides a prompt.
- Retrieval: The system identifies relevant documents or data snippets from a curated knowledge base that might contain the answer. This is typically done by converting the query into a numerical representation (an embedding) and finding similar embeddings in a vector database.
- Augmentation: The retrieved relevant information is then added to the original user query, forming an enriched prompt.
- Generation: The LLM receives this augmented prompt and generates a response based on both its internal knowledge and the provided context.
This process ensures that the LLM has access to the most current and domain-specific information, significantly improving accuracy and trustworthiness.
Why LangChain and ChromaDB for Production?
For building production-grade RAG systems, selecting the right tools is paramount. LangChain and ChromaDB offer a compelling combination:
- LangChain: The Orchestration Layer
- Modularity: Provides a standardized interface for various LLMs, embedding models, document loaders, text splitters, and vector stores.
- Chains and Agents: Simplifies complex multi-step interactions, allowing you to build sophisticated workflows beyond simple Q&A.
- Rapid Prototyping to Production: Its abstraction layers help quickly build prototypes that can be refined for production use.
- Ecosystem: A rich set of integrations and active community support.
- ChromaDB: The Vector Store Foundation
- Open-Source & Embeddable: Can run locally, client-server, or within your application, offering flexibility for deployment.
- Ease of Use: Simple API for adding, querying, and managing embeddings.
- Scalability: Designed to handle millions of vectors efficiently, crucial for growing knowledge bases.
- Persistence: Stores data persistently, so your indexed documents are safe across sessions.
- Developer Experience: Focuses on a smooth experience for developers integrating vector search.
Together, they form a powerful stack for building RAG applications that are not only intelligent but also robust, maintainable, and scalable.
The Architecture of a Production RAG System
A production-ready RAG system is more than just a simple script. It typically involves several interconnected components and workflows. Let’s outline a common architecture:
+-------------------+
| User Request |
+---------+---------+
|
v
+-----------------------+ +-------------------+
| Data Ingestion Microservice | | API Gateway/Frontend |
| (Batch/Streaming) | +---------+---------+
+-----------+-----------+ |
| v
v +-------------------+
+-----------------------+ | RAG Orchestrator |
| Document Loader | | (LangChain App) |
+-----------+-----------+ +---------+---------+
| |
v |
+-----------------------+ +---------+---------+
| Text Splitter | | Query Embedder |
+-----------+-----------+ +---------+---------+
| |
v |
+-----------------------+ +---------+---------+
| Embedding Model |<---------------->| Vector Store (ChromaDB) |
+-----------+-----------+ +---------+---------+
| |
v |
+-----------------------+ +---------+---------+
| Vector Store Indexer | | Retrieval Module |
| (ChromaDB Client) |<---------------->| (LangChain Retriever) |
+-----------+-----------+ +---------+---------+
| |
v |
+-----------------------+ +---------+---------+
| Storage (e.g., S3, DB)| | LLM Prompt Builder |
+-----------------------+ +---------+---------+
|
v
+-------------------+
| LLM (e.g., OpenAI, |
| Anthropic, Local) |
+---------+---------+
|
v
+-------------------+
| Response & |
| Post-processing |
+---------+---------+
|
v
+-------------------+
| User Response |
+-------------------+
Let’s break down these components:
- Data Ingestion Pipeline:
- Document Loaders: Ingests data from various sources (PDFs, websites, databases, APIs, etc.).
- Text Splitters: Breaks down large documents into smaller, manageable chunks (e.g., paragraphs, sentences, fixed token size) suitable for embedding and retrieval.
- Embedding Model: Converts text chunks into high-dimensional numerical vectors (embeddings).
- Vector Store Indexer: Takes these embeddings and stores them in a vector database (ChromaDB) for efficient similarity search. This process often runs asynchronously or in batches.
- RAG Orchestrator (LangChain Application):
- API Gateway/Frontend: Receives user queries.
- Query Embedder: Converts the incoming user query into an embedding using the same embedding model used for documents.
- Retrieval Module: Queries the vector store (ChromaDB) with the query embedding to find the most semantically similar document chunks.
- LLM Prompt Builder: Constructs a comprehensive prompt for the LLM, incorporating the original user query and the retrieved context. This is where prompt engineering plays a crucial role.
- LLM: Generates a response based on the augmented prompt. This can be a cloud-based service (OpenAI, Anthropic) or a self-hosted open-source model.
- Response Post-processing: Cleans up, formats, or filters the LLM’s raw output before presenting it to the user.
- Vector Store (ChromaDB): The central repository for all document embeddings, enabling fast similarity searches.
- Monitoring & Observability: Critical for production, includes logging, tracing (e.g., LangSmith), and metrics to understand system health and performance.
- Deployment Infrastructure: Typically involves Docker, Kubernetes, or serverless functions for scalability and reliability.
Deep Dive into LangChain Components
LangChain is a framework designed to simplify the development of applications powered by LLMs. It offers a structured way to combine different LLM-related components. Let’s look at the key components relevant to RAG:
1. LLMs and Chat Models
LangChain provides uniform interfaces for interacting with various LLM providers.
from langchain_openai import OpenAI, ChatOpenAI
from langchain_community.llms import HuggingFaceHub # For open-source models
# Initialize a standard LLM (text completion)
llm = OpenAI(api_key="YOUR_OPENAI_API_KEY", temperature=0.7)
# Initialize a chat model (for conversational interfaces)
chat_model = ChatOpenAI(api_key="YOUR_OPENAI_API_KEY", temperature=0.7)
# Example with a Hugging Face model
# from langchain_community.llms import HuggingFaceHub
# llm_hf = HuggingFaceHub(
# repo_id="google/flan-t5-large",
# model_kwargs={"temperature":0.5, "max_length":64}
# )
2. Document Loaders
These components are responsible for fetching data from different sources and loading them into a standard `Document` format (text content and metadata).
from langchain_community.document_loaders import PyPDFLoader, WebBaseLoader, CSVLoader
from langchain_core.documents import Document
# Load from a PDF file
pdf_loader = PyPDFLoader("example.pdf")
pdf_docs = pdf_loader.load()
# Load from a URL
web_loader = WebBaseLoader("https://www.khadervali.com/blog/production-ready-rag")
web_docs = web_loader.load()
# Load from a CSV file
csv_loader = CSVLoader("data.csv")
csv_docs = csv_loader.load()
# You can also create documents manually
manual_doc = Document(page_content="This is a manually created document chunk.", metadata={"source": "manual"})
3. Text Splitters
Long documents need to be split into smaller, meaningful chunks for effective embedding and retrieval. LangChain offers various strategies.
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Initialize a recursive character text splitter
# It tries to split by different characters (e.g., "\n\n", "\n", " ", "")
# `chunk_size` is the maximum length of a chunk in characters (or tokens, depending on model)
# `chunk_overlap` ensures continuity between chunks
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=1000,
chunk_overlap=200,
length_function=len, # Use len for character count, or a tokenizer for token count
is_separator_regex=False,
)
# Split the loaded documents
all_docs = pdf_docs + web_docs + csv_docs
chunks = text_splitter.split_documents(all_docs)
print(f"Number of original documents: {len(all_docs)}")
print(f"Number of chunks after splitting: {len(chunks)}")
print(f"First chunk content: {chunks[0].page_content[:200]}...")
4. Embedding Models
These models convert text into numerical vectors (embeddings) that capture semantic meaning. LangChain integrates with many providers.
from langchain_openai import OpenAIEmbeddings
from langchain_community.embeddings import HuggingFaceEmbeddings
# Initialize OpenAI Embeddings
openai_embeddings = OpenAIEmbeddings(api_key="YOUR_OPENAI_API_KEY")
# Initialize Hugging Face local embeddings (e.g., Sentence Transformers)
# Ensure you have the 'sentence-transformers' package installed
hf_embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
# Example: Get embedding for a text
text_embedding = openai_embeddings.embed_query("What is RAG?")
print(f"Embedding dimension: {len(text_embedding)}")
5. Retrievers
Retrievers are responsible for fetching relevant documents from a knowledge base. In RAG, this typically involves querying a vector store.
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings
from langchain.chains import RetrievalQA
# We'll use ChromaDB for this (see next section for full setup)
# For now, assume a ChromaDB instance `vectordb` is already populated.
# vectordb = Chroma(...)
# Create a retriever from the vector store
# `search_kwargs` can include 'k' (number of results) and 'score_threshold' (for similarity)
# retriever = vectordb.as_retriever(search_kwargs={"k": 5})
# Example usage (conceptual, assuming `retriever` exists)
# query = "What are the benefits of using RAG?"
# retrieved_docs = retriever.invoke(query)
# print(f"Retrieved {len(retrieved_docs)} documents.")
# print(f"Content of first retrieved doc: {retrieved_docs[0].page_content[:150]}...")
6. Chains
Chains are sequences of calls to LLMs or other utilities. For RAG, the `RetrievalQA` chain is a common starting point, but custom chains offer more flexibility.
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
# 1. Define your prompt
prompt = ChatPromptTemplate.from_template("""Answer the user's question based on the provided context.
If you don't know the answer, just say that you don't know, don't try to make up an answer.
Context: {context}
Question: {input}""")
# 2. Initialize LLM and Embeddings
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0, api_key="YOUR_OPENAI_API_KEY")
embeddings = OpenAIEmbeddings(api_key="YOUR_OPENAI_API_KEY")
# 3. Create a dummy ChromaDB for demonstration
# In a real scenario, this would be populated with your actual documents
vectorstore = Chroma.from_texts(
["The capital of France is Paris.", "The Eiffel Tower is in Paris.", "France is in Europe."],
embedding=embeddings
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 2})
# 4. Create a chain that combines the retrieved documents into a single string
document_combiner_chain = create_stuff_documents_chain(llm, prompt)
# 5. Create the full RAG retrieval chain
rag_chain = create_retrieval_chain(retriever, document_combiner_chain)
# Now, you can invoke the chain
# response = rag_chain.invoke({"input": "What is the capital of France?"})
# print(response["answer"])
# print(response["context"]) # Shows the documents retrieved
Deep Dive into ChromaDB
ChromaDB is an open-source embedding database that makes it easy to build LLM applications. It’s designed to be simple to use, yet powerful enough for production workloads.
ChromaDB Installation and Basic Usage
Installation is straightforward:
pip install chromadb
ChromaDB can run in several modes:
- In-memory: Easiest for development, data is lost when the script ends.
- Persistent client: Stores data on disk, suitable for local development and smaller production setups.
- Client/Server: For distributed and scalable production environments, running ChromaDB as a separate service.
Initializing ChromaDB
import chromadb
from chromadb.utils import embedding_functions
# 1. In-memory client (data lost on exit)
# client_in_memory = chromadb.Client()
# 2. Persistent client (data stored locally on disk)
# This is often suitable for a single application instance in production
persistent_client = chromadb.PersistentClient(path="./chroma_data")
# 3. Client for a remote ChromaDB server (for distributed production)
# client_http = chromadb.HttpClient(host="localhost", port=8000)
# (You'd need to run `chroma run --path ./chroma_data` in another terminal for this)
# Define an embedding function. It's crucial that this matches the one used for your documents.
# For simplicity, we'll use a local HuggingFace embedding function provided by ChromaDB
# For production, you'd likely use OpenAIEmbeddings, CohereEmbeddings, or a robust self-hosted model.
hf_ef = embedding_functions.SentenceTransformerEmbeddingFunction(model_name="all-MiniLM-L6-v2")
# Create or get a collection
# A collection is where your documents and their embeddings are stored.
collection_name = "my_rag_documents"
collection = persistent_client.get_or_create_collection(
name=collection_name,
embedding_function=hf_ef # Associate an embedding function with the collection
)
Adding Documents to ChromaDB
Documents in ChromaDB consist of IDs, embeddings, metadatas, and actual text content.
# Example documents
documents = [
"The quick brown fox jumps over the lazy dog.",
"Artificial intelligence is transforming industries globally.",
"Machine learning is a subset of AI.",
"Python is a popular programming language for AI.",
"Data science involves statistical analysis and machine learning.",
]
metadatas = [
{"source": "article_1", "page": "1"},
{"source": "report_ai", "date": "2023-01-15"},
{"source": "report_ai", "date": "2023-01-15"},
{"source": "book_python", "chapter": "1"},
{"source": "course_ds", "module": "2"},
]
# Generate unique IDs for each document
ids = [f"doc_{i}" for i in range(len(documents))]
# Add documents to the collection. ChromaDB will use the defined embedding function
# to convert the texts into embeddings before storing.
collection.add(
documents=documents,
metadatas=metadatas,
ids=ids
)
print(f"Number of documents in collection: {collection.count()}")
Querying ChromaDB
Retrieving documents is as simple as calling the `query` method.
# Query the collection
query_text = "What is AI and machine learning?"
results = collection.query(
query_texts=[query_text],
n_results=2, # Number of most relevant results to retrieve
# You can also filter by metadata here: where={"source": "report_ai"}
)
print("\nQuery Results:")
for i in range(len(results['documents'][0])):
print(f" Document: {results['documents'][0][i]}")
print(f" Metadata: {results['metadatas'][0][i]}")
print(f" Distance (lower is better): {results['distances'][0][i]:.4f}")
print("-" * 20)
Integrating ChromaDB with LangChain
LangChain has a direct integration with ChromaDB, making it seamless to use as a vector store.
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings # Or HuggingFaceEmbeddings
import os
# Ensure your OpenAI API key is set as an environment variable
# os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"
# Initialize your embedding model
embeddings = OpenAIEmbeddings()
# Directory to persist ChromaDB data
persist_directory = "./chroma_db_langchain"
# Load documents (from earlier example or new ones)
documents_to_index = [
Document(page_content="Khadervali.com offers expert insights into AI development.", metadata={"source": "website"}),
Document(page_content="Building production-ready systems requires careful planning.", metadata={"source": "blog_post"}),
Document(page_content="LangChain simplifies LLM application orchestration.", metadata={"source": "tool_guide"}),
Document(page_content="ChromaDB provides an easy-to-use vector database for embeddings.", metadata={"source": "tool_guide"}),
]
# Create a Chroma vector store from documents
# This will embed the documents and store them in ChromaDB
vectordb = Chroma.from_documents(
documents=documents_to_index,
embedding=embeddings,
persist_directory=persist_directory
)
# You can also load an existing persistent ChromaDB
# vectordb_loaded = Chroma(persist_directory=persist_directory, embedding_function=embeddings)
# Turn the vector store into a retriever
retriever = vectordb.as_retriever(search_kwargs={"k": 2})
# Now, this retriever can be used directly in LangChain chains.
# query_results = retriever.invoke("Tell me about Khadervali.com")
# print("Retrieved documents via LangChain Chroma retriever:")
# for doc in query_results:
# print(f"- {doc.page_content[:100]}...")
Building a Production-Ready RAG Workflow with LangChain and ChromaDB
Let’s tie everything together into a more concrete, production-oriented example. We’ll simulate a knowledge base Q&A system for an internal company wiki.
Scenario: Internal Wiki Q&A Chatbot
Imagine your company has a vast internal wiki with documentation, policies, HR FAQs, and technical guides. Employees often struggle to find specific information quickly. We want to build a chatbot that can answer questions accurately by retrieving relevant information from this wiki.
Step 1: Setup and Dependencies
pip install langchain langchain-openai chromadb pypdf beautifulsoup4
import os
from dotenv import load_dotenv
# Load environment variables from .env file
load_dotenv()
# Set your API keys
# For production, use secure environment variable management
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")
# os.environ["HUGGINGFACEHUB_API_TOKEN"] = os.getenv("HUGGINGFACEHUB_API_TOKEN") # If using HF models
Step 2: Data Ingestion Pipeline
This pipeline will run periodically (e.g., daily, weekly) or whenever new wiki content is published. It involves loading, splitting, embedding, and indexing documents.
from langchain_community.document_loaders import PyPDFLoader, WebBaseLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_community.vectorstores import Chroma
from langchain_core.documents import Document
import shutil # For clearing the vector store for fresh start
# Configuration
VECTOR_DB_PATH = "./chroma_wiki_db"
CHUNK_SIZE = 1000
CHUNK_OVERLAP = 200
EMBEDDING_MODEL_NAME = "text-embedding-ada-002" # OpenAI's default
def clean_vector_store(path):
"""Deletes the vector store directory."""
if os.path.exists(path):
shutil.rmtree(path)
print(f"Cleaned existing vector store at {path}")
def load_and_split_documents(sources):
"""Loads documents from various sources and splits them into chunks."""
all_documents = []
for source_type, path_or_url in sources:
print(f"Loading from {source_type}: {path_or_url}")
if source_type == "pdf":
loader = PyPDFLoader(path_or_url)
elif source_type == "web":
loader = WebBaseLoader(path_or_url)
# Add more loaders as needed (CSVLoader, DirectoryLoader, etc.)
else:
raise ValueError(f"Unsupported source type: {source_type}")
all_documents.extend(loader.load())
# Add a synthetic document for demonstration
all_documents.append(Document(
page_content="Our company policy on remote work states that employees can work remotely up to 3 days a week, with prior manager approval.",
metadata={"source": "HR Policy", "date": "2023-10-26"}
))
text_splitter = RecursiveCharacterTextSplitter(
chunk_size=CHUNK_SIZE,
chunk_overlap=CHUNK_OVERLAP,
length_function=len,
)
chunks = text_splitter.split_documents(all_documents)
print(f"Loaded {len(all_documents)} raw documents and split into {len(chunks)} chunks.")
return chunks
def index_documents_to_chromadb(chunks, db_path, embedding_model):
"""Embeds document chunks and stores them in ChromaDB."""
# Initialize OpenAI Embeddings
embeddings = OpenAIEmbeddings(model=embedding_model)
# Clean existing DB for a fresh start (for demo purposes)
clean_vector_store(db_path)
# Create ChromaDB from documents, persisting to disk
vectordb = Chroma.from_documents(
documents=chunks,
embedding=embeddings,
persist_directory=db_path
)
vectordb.persist()
Khader Vali
Senior Software Engineer specializing in cloud architecture, real-time systems, and enterprise-scale applications.
