How to Build a Local RAG Pipeline Using Ollama, LangChain, and PostgreSQL

Introduction

If you've spent any time building AI applications over the past couple of years, you've probably noticed a quiet but important shift happening in 2026: teams are moving away from "just call the OpenAI API" as the default architecture. Data privacy regulations are tightening, API costs add up fast at scale, and a lot of organizations simply don't want their proprietary documents leaving their own infrastructure.

That's where local Retrieval-Augmented Generation (RAG) comes in.

RAG is the technique of grounding a large language model's responses in your own data by retrieving relevant chunks of information and feeding them into the model's context window before it generates an answer. Instead of relying purely on what the model "remembers" from training, you're handing it fresh, relevant facts at query time.

In this guide, we're going to build a completely local, offline-capable RAG pipeline using three tools that pair together surprisingly well:

Ollama — to run open-source LLMs and embedding models directly on your machine (or your own server)
LangChain — to orchestrate the retrieval and generation logic
PostgreSQL with the pgvector extension — to store and search vector embeddings alongside your regular relational data

By the end of this article, you'll have a working RAG system that never sends a single byte of your data to a third-party API. This is especially valuable if you're working in healthcare, legal, finance, government, or any domain where data residency and confidentiality actually matter — not just as a nice-to-have, but as a hard requirement.

Let's get into it.

Why Go Local? Understanding the Motivation

Before we touch any code, it's worth understanding why a local RAG stack has become such a popular architecture choice.

Data Privacy and Compliance

When you send documents to a hosted LLM API, that data typically passes through a third party's servers. For many organizations — especially those bound by HIPAA, GDPR, or internal data governance policies — this is a non-starter. Running everything locally means your sensitive contracts, patient records, or internal knowledge base never leave your network boundary.

Cost Predictability

Hosted LLM APIs charge per token. At scale, especially for RAG systems that stuff large context windows with retrieved documents on every single query, costs can spiral quickly. Local inference has a fixed cost: your hardware. Once you've invested in decent compute, your marginal cost per query drops to nearly zero.

Latency and Reliability

A local pipeline isn't subject to third-party API rate limits, outages, or network latency. If your GPU or CPU can handle the load, your system stays responsive regardless of what's happening with an external provider's infrastructure.

Full Control Over the Stack

With a local setup, you control the model version, the embedding model, the chunking strategy, and the database — nothing changes underneath you without your consent. This matters a lot for regulated industries that need reproducible, auditable AI behavior.

Core Concepts You Need to Know

Let's quickly unpack the three pillars of this architecture before we start building.

What Is RAG, Really?

At its core, RAG is a two-stage process:

Retrieval — Given a user's question, search a knowledge base for the most semantically relevant pieces of text.
Generation — Pass those retrieved chunks, along with the original question, to an LLM, which synthesizes a natural-language answer grounded in that context.

The "semantic" part of retrieval is powered by embeddings — numerical vector representations of text where similar meanings end up close together in vector space. Instead of keyword matching, RAG systems use vector similarity search (usually cosine similarity or dot product) to find relevant content even when the wording doesn't match exactly.

What Is Ollama?

Ollama is a lightweight runtime for downloading, managing, and serving open-source LLMs locally. It abstracts away the messy parts of running models (quantization formats, GPU memory management, tokenizer setup) behind a simple CLI and REST API. You can pull a model with one command:

ollama pull llama3.1

And it exposes an OpenAI-compatible-ish local API on http://localhost:11434 that LangChain can talk to directly. Ollama also supports dedicated embedding models, which is exactly what we need for the retrieval half of RAG.

What Is LangChain?

LangChain is a framework that provides standardized abstractions for building LLM applications — document loaders, text splitters, vector store integrations, retrievers, and chains that tie everything together. It's not doing anything magical under the hood, but it saves you from writing a lot of boilerplate glue code, and it has first-class support for both Ollama and pgvector.

What Is pgvector?

pgvector is a PostgreSQL extension that adds a native vector data type and similarity search operators (<-> for L2 distance, <=> for cosine distance, <#> for inner product) directly into Postgres. This means you don't need a separate, dedicated vector database like Pinecone or Weaviate — you can store your embeddings right next to your existing relational data, run joins between them, and rely on Postgres's mature tooling for backups, replication, and indexing.

For most teams that already run PostgreSQL in production, this is a huge operational win: one less system to deploy, monitor, and secure.

Architecture Overview

Here's the high-level flow of the pipeline we're building:

Ingest documents (PDFs, markdown, text files) and split them into chunks.
Embed each chunk using a local embedding model served by Ollama.
Store the chunks and their embeddings in a PostgreSQL table using pgvector.
Query: when a user asks a question, embed the question, run a similarity search against pgvector, and retrieve the top-k relevant chunks.
Generate: pass the retrieved chunks plus the question to a local LLM (via Ollama) using a prompt template, and return the grounded answer.

Everything in this pipeline — embedding, storage, retrieval, and generation — runs on infrastructure you control.

Setting Up the Environment

Step 1: Install and Configure Ollama

Download Ollama from ollama.com or install it via your package manager, then pull the models we'll use — a generation model and a dedicated embedding model:

# Install Ollama (macOS/Linux)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a chat/generation model
ollama pull llama3.1

# Pull an embedding model
ollama pull nomic-embed-text

Verify the server is running:

ollama serve

By default, Ollama listens on http://localhost:11434.

Step 2: Set Up PostgreSQL with pgvector

The easiest way to get a Postgres instance with pgvector pre-installed is via Docker:

docker run --name pgvector-db \
  -e POSTGRES_USER=raguser \
  -e POSTGRES_PASSWORD=ragpassword \
  -e POSTGRES_DB=ragdb \
  -p 5432:5432 \
  -d pgvector/pgvector:pg16

Once it's running, connect and enable the extension:

-- Connect to your database, then run:
CREATE EXTENSION IF NOT EXISTS vector;

That single command is all it takes to turn a regular Postgres database into a vector-capable one.

Step 3: Install Python Dependencies

pip install langchain langchain-community langchain-ollama \
    langchain-postgres psycopg2-binary pypdf tiktoken

A quick note on package naming: LangChain's ecosystem has split into modular packages (langchain-ollama, langchain-postgres, etc.) to keep dependencies lean. Always check you're importing from the correct sub-package, since import paths have shifted a few times as the library matured.

Building the Pipeline Step by Step

Step 1: Load and Chunk Your Documents

Document chunking is one of the most underrated levers in RAG quality. Chunks that are too large dilute relevance; chunks that are too small lose context. A good starting point is 500–1000 tokens per chunk with some overlap.

from langchain_community.document_loaders import PyPDFLoader, DirectoryLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

# Load all PDFs from a folder
loader = DirectoryLoader("./knowledge_base", glob="**/*.pdf", loader_cls=PyPDFLoader)
documents = loader.load()

# Split into overlapping chunks
splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,
    chunk_overlap=120,
    separators=["\n\n", "\n", ". ", " ", ""],
)
chunks = splitter.split_documents(documents)

print(f"Loaded {len(documents)} documents, split into {len(chunks)} chunks")

RecursiveCharacterTextSplitter is a good default because it tries to split on natural boundaries (paragraphs, then sentences, then words) before falling back to hard character cuts — this preserves semantic coherence within each chunk.

Step 2: Generate Embeddings with Ollama

from langchain_ollama import OllamaEmbeddings

embeddings = OllamaEmbeddings(
    model="nomic-embed-text",
    base_url="http://localhost:11434",
)

# Quick sanity check
vector = embeddings.embed_query("What is retrieval-augmented generation?")
print(f"Embedding dimension: {len(vector)}")

nomic-embed-text produces 768-dimensional vectors and performs well for general-purpose document retrieval while remaining lightweight enough to run on CPU if needed.

Step 3: Store Embeddings in PostgreSQL with pgvector

LangChain's PGVector integration handles table creation, insertion, and similarity search for you.

from langchain_postgres import PGVector

CONNECTION_STRING = "postgresql+psycopg2://raguser:ragpassword@localhost:5432/ragdb"
COLLECTION_NAME = "company_knowledge_base"

vector_store = PGVector(
    embeddings=embeddings,
    collection_name=COLLECTION_NAME,
    connection=CONNECTION_STRING,
    use_jsonb=True,
)

# Add our chunks to the vector store
vector_store.add_documents(chunks)

Under the hood, this creates a table with a vector column, stores each chunk's text, metadata (source file, page number, etc.), and its embedding. You can inspect it directly with SQL if you're curious:

SELECT id, document, cmetadata
FROM langchain_pg_embedding
LIMIT 5;

Step 4: Build the Retriever

retriever = vector_store.as_retriever(
    search_type="similarity",
    search_kwargs={"k": 4},
)

results = retriever.invoke("What is our refund policy?")
for doc in results:
    print(doc.page_content[:200], "...\n")

k=4 means we retrieve the four most relevant chunks. This is a tunable parameter — more on that in the best practices section.

Step 5: Connect the Local LLM

from langchain_ollama import ChatOllama

llm = ChatOllama(
    model="llama3.1",
    base_url="http://localhost:11434",
    temperature=0.1,
)

We keep temperature low because RAG answers should be grounded and factual, not creative.

Step 6: Assemble the Full RAG Chain

Now we wire retrieval and generation together using LangChain's composable chain syntax (LCEL).

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

RAG_PROMPT = ChatPromptTemplate.from_template(
    """You are a helpful assistant answering questions based only on the
provided context. If the answer isn't in the context, say you don't know.
Do not make up information.

Context:
{context}

Question:
{question}

Answer:"""
)


def format_docs(docs):
    return "\n\n".join(doc.page_content for doc in docs)


rag_chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | RAG_PROMPT
    | llm
    | StrOutputParser()
)

# Ask a question
answer = rag_chain.invoke("What is our refund policy?")
print(answer)

That's it — a fully functional, fully local RAG pipeline. No API keys, no external calls, no data leaving your machine.

Real-World Example: Internal Policy Assistant

Let's ground this in a practical scenario. Imagine an HR team that wants employees to be able to ask natural-language questions about internal policies — leave entitlements, expense rules, remote work guidelines — without digging through a 40-page PDF handbook.

With the pipeline above, the workflow looks like this:

Drop all policy PDFs into ./knowledge_base.
Run the ingestion script once (or on a schedule, whenever documents are updated) to populate pgvector.
Expose the rag_chain behind a simple internal API (FastAPI works great here) so employees can query it via a Slack bot or internal web app.

from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Query(BaseModel):
    question: str

@app.post("/ask")
def ask(query: Query):
    answer = rag_chain.invoke(query.question)
    return {"answer": answer}

Because everything runs on-premises, HR data and internal policy documents never touch an external API — which is often a hard compliance requirement for anything involving employee records.

This same pattern extends naturally to legal contract review, clinical documentation search, internal engineering wikis, or customer support knowledge bases — anywhere you have a body of text that people need to query conversationally.

Best Practices for Production-Grade Local RAG

1. Tune Your Chunking Strategy to Your Content

Don't treat chunk size as a fixed constant. Legal contracts with dense clauses benefit from smaller chunks with more overlap; narrative documentation can tolerate larger chunks. Experiment and evaluate retrieval quality on a held-out set of representative questions.

2. Use Metadata Filtering

Store metadata like source, department, date, or access_level alongside each chunk, and filter at query time:

retriever = vector_store.as_retriever(
    search_kwargs={"k": 4, "filter": {"department": "finance"}}
)

This lets you scope retrieval per user role or document category — critical for multi-tenant or access-controlled knowledge bases.

3. Index Your Vector Column

For anything beyond a small proof of concept, add an index so similarity search stays fast as your table grows:

CREATE INDEX ON langchain_pg_embedding
USING hnsw (embedding vector_cosine_ops);

HNSW (Hierarchical Navigable Small World) indexes offer a strong balance of query speed and recall for most RAG workloads at scale.

4. Evaluate Retrieval Separately from Generation

A common failure mode is blaming the LLM for a bad answer when the real problem is that retrieval pulled the wrong chunks. Build a small evaluation set of question/expected-source pairs and check retrieval precision independently before troubleshooting the generation step.

5. Match Model Size to Hardware

Larger local models (70B+) produce noticeably better reasoning but need serious GPU memory. If you're running on consumer hardware, quantized 7B–8B models like llama3.1:8b or mistral offer a solid quality-to-resource ratio. Benchmark on your actual queries, not generic leaderboards.

6. Version Your Embedding Model

If you ever switch embedding models, you must re-embed your entire corpus — mixing vectors from different embedding models in the same similarity search produces meaningless results. Track which embedding model/version generated each vector.

🚀 Pro Tips

Cache embeddings for unchanged documents. Re-embedding your entire corpus on every ingestion run wastes compute — hash document content and skip chunks that haven't changed.
Use pgvector's IVFFlat index instead of HNSW if you have a very large, mostly static dataset and can tolerate a build step — it can be faster to build, though HNSW generally wins on query-time recall.
Run a reranker after initial retrieval. A lightweight cross-encoder reranker (even a local one) applied to your top-20 retrieved chunks before selecting the final top-4 can meaningfully improve answer relevance.
Stream responses. ChatOllama supports streaming — use it in production UIs so users see tokens appear in real time instead of waiting for the full generation.
Set a strict system prompt guardrail telling the model to explicitly say "I don't know" when context is insufficient — this single change dramatically reduces hallucination in RAG systems.
Monitor GPU/CPU load with ollama ps to understand which models are loaded in memory and avoid unexpected memory pressure when running multiple models concurrently.

Common Mistakes to Avoid

Skipping chunk overlap entirely. Zero overlap between chunks can split a sentence or idea right at the boundary, causing the retriever to miss context that spans two chunks.
Using a generation model as your embedding model. Chat-tuned LLMs aren't optimized for producing similarity-friendly vectors — always use a model specifically trained for embeddings, like nomic-embed-text or mxbai-embed-large.
Forgetting to enable the pgvector extension per database. CREATE EXTENSION vector is scoped to a single database — if you create a new database, you need to run it again.
Not indexing the vector column. Sequential scans over tens of thousands of embeddings will get painfully slow; this is easy to miss in early development when your dataset is small.
Overstuffing the context window. Retrieving too many chunks (k=20 for example) can dilute relevance and push the model toward "lost in the middle" behavior, where it ignores information buried in a long context.
Ignoring prompt injection risks from retrieved content. If your knowledge base includes user-submitted content, malicious text embedded in a document could attempt to manipulate the LLM's instructions — sanitize and validate ingested content.
Assuming "local" means "no ops burden." You're still responsible for backups, index maintenance, model updates, and monitoring — local doesn't mean maintenance-free.

📌 Key Takeaways

A local RAG stack built on Ollama + LangChain + pgvector gives you full data privacy, predictable costs, and no dependency on third-party APIs.
pgvector lets you avoid standing up a separate vector database by extending PostgreSQL you likely already run in production.
Retrieval quality — driven by chunking strategy, embedding model choice, and indexing — matters more to final answer quality than which LLM you pick for generation.
Production-readiness requires attention to indexing (HNSW), metadata filtering, evaluation, and prompt guardrails against hallucination — not just wiring the components together.
This architecture scales from a small internal policy assistant to enterprise-grade knowledge retrieval systems, all while keeping sensitive data fully on-premises.

Conclusion

Building a RAG pipeline used to mean juggling a hosted LLM API, a dedicated vector database service, and an orchestration layer — three separate vendors, three separate bills, and three separate places your data had to travel through. The stack we just built collapses all of that into infrastructure you fully own: Ollama for local inference, LangChain for orchestration, and PostgreSQL with pgvector as a single, unified store for both your relational and vector data.

This isn't just an academic exercise. As privacy regulations tighten and organizations become more deliberate about where their data lives, local-first RAG architectures are quickly becoming the default choice for internal tools — not just a fallback for teams that can't use hosted APIs.

Start small: ingest a handful of documents, get a basic retrieval loop working, and iterate on chunking and retrieval quality before you worry about scale. The fundamentals you've learned here — embedding, storing, retrieving, and generating — apply whether you're indexing ten documents or ten million.