How to Build a Local GraphRAG Pipeline with PostgreSQL, LangChain, and Ollama for Multi-Hop Reasoning

Introduction

If you've shipped a Retrieval-Augmented Generation (RAG) system in production, you've probably hit the same wall I did: your vector database confidently retrieves five "semantically similar" chunks, and your LLM still gives an answer that misses the actual connection between two facts buried in different documents.

Ask a standard RAG pipeline something like "Which vendor supplied the faulty component that caused the Q3 outage, and who approved that vendor's contract?" and you'll usually get a shrug, a hallucination, or an answer that only addresses half the question. That's because the vendor name lives in one document, the outage report lives in another, and the contract approval lives in a third — and cosine similarity has no concept of "these three facts are connected."

This is exactly the problem GraphRAG (Graph Retrieval-Augmented Generation) was designed to solve. Instead of treating your knowledge base as a flat pile of embeddings, GraphRAG models it as a knowledge graph — entities connected by relationships — so the retrieval step can traverse connections instead of just measuring distance.

In this tutorial, we're going to build a complete, production-realistic GraphRAG pipeline that runs entirely on your local machine. No OpenAI API key, no cloud vector database, no Neo4j license. Just:

• Ollama for running open-weight LLMs locally (extraction + generation)
• LangChain for orchestrating extraction, embedding, and retrieval
• PostgreSQL (with pgvector) as both our vector store and our graph store
• Next.js as the application layer that serves answers to end users

By the end, you'll have a working pipeline that can answer multi-hop questions your current RAG stack simply can't.

Why Vector Search Alone Fails at Multi-Hop Reasoning

Before we build anything, it's worth being precise about the failure mode we're fixing.

A standard RAG pipeline does roughly this:

1. Chunk your documents.
2. Embed each chunk.
3. At query time, embed the question and retrieve the top-k nearest chunks.
4. Stuff those chunks into a prompt and ask the LLM to answer.

This works beautifully for single-hop questions — "What is our refund policy?" — where the answer lives in one contiguous chunk of text. It breaks down for multi-hop questions, where the answer requires chaining facts across multiple documents:

• "What is the current job title of the person who founded the company that acquired our biggest competitor?"
• "Which incidents were caused by the same root cause as the March outage?"
• "List all employees who report, directly or indirectly, to the VP of Engineering."

None of these questions can be answered by finding "similar" text. They require traversal: find entity A, follow a relationship to entity B, follow another relationship to entity C. Vector search has no native concept of a relationship — it only knows about proximity in embedding space.

GraphRAG closes this gap by extracting a structured layer — entities and relationships — that sits alongside your embeddings, giving the retrieval step an actual path to walk.

What Is GraphRAG, Really?

At its core, GraphRAG is a two-layer retrieval architecture:

• Layer 1 — Vector Layer: Standard chunk embeddings, used to find entry points into the graph (i.e., "which entities are relevant to this question?").
• Layer 2 — Graph Layer: A structured knowledge graph of (subject, predicate, object) triplets extracted from your source documents, used to expand context by following relationships outward from those entry points.

The retrieval flow looks like this:

1. Embed the user's query.
2. Use vector search to find the most relevant chunks/entities (entry points).
3. Traverse the graph N hops outward from those entities.
4. Assemble the traversed subgraph + original chunk text into context.
5. Pass everything to the LLM for final answer synthesis.

This hybrid approach is why GraphRAG consistently outperforms plain RAG on multi-hop benchmarks — it doesn't replace vector search, it augments it with structural awareness.

Architecture Overview

Here's the full local stack we're building:

┌──────────────────┐      ┌──────────────────┐      ┌───────────────────────┐
│  Raw Documents   │ ──▶ │   LangChain +     │ ──▶ │      PostgreSQL       │
│ (PDFs, Markdown) │      │ Ollama (extract) │      │  - entities table     │
└──────────────────┘      └──────────────────┘      │  - relationships table│
                                                    │  - chunks + pgvector  │
                                                    └───────────────────────┘
                                                                │
                                                                ▼
                                                    ┌───────────────────────┐
                                                    │   Retrieval Engine    │
                                                    │  (vector + recursive  │
                                                    │   CTE graph walk)     │
                                                    └───────────────────────┘
                                                                │
                                                                ▼
                                                    ┌───────────────────────┐
                                                    │  Ollama (generation)  │
                                                    └───────────────────────┘
                                                                │
                                                                ▼
                                                    ┌───────────────────────┐
                                                    │   Next.js API Route   │
                                                    └───────────────────────┘

The key architectural decision here is using PostgreSQL as the graph store instead of Neo4j or a dedicated graph database. In 2026, with pgvector maturity and recursive CTE performance improvements in Postgres 17+, this is a completely viable choice for small-to-medium knowledge graphs (hundreds of thousands of nodes), and it means one less piece of infrastructure to run, back up, and secure.

Prerequisites

Before you start, make sure you have:

• Ollama installed, with a capable local model pulled — we'll use llama3.1:8b-instruct for extraction and generation, and nomic-embed-text for embeddings.
• PostgreSQL 16+ with the pgvector extension installed.
• Python 3.11+ with langchain, langchain-ollama, psycopg[binary], and pydantic installed.
• Node.js 20+ for the Next.js frontend.

Pull the models you need:

ollama pull llama3.1:8b-instruct
ollama pull nomic-embed-text

Install the Python dependencies:

pip install langchain langchain-ollama langchain-community psycopg[binary] pydantic pgvector

Step 1: Designing the PostgreSQL Schema for a Knowledge Graph

The heart of this architecture is a relational schema that can represent a graph without needing a graph-native database. We need three tables: chunks (for vector search), entities (graph nodes), and relationships (graph edges).

-- Enable the vector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Raw document chunks, used for vector-based entry-point retrieval
CREATE TABLE chunks (
    id SERIAL PRIMARY KEY,
    document_id TEXT NOT NULL,
    content TEXT NOT NULL,
    embedding VECTOR(768), -- dimension matches nomic-embed-text
    created_at TIMESTAMPTZ DEFAULT now()
);

CREATE INDEX chunks_embedding_idx
    ON chunks USING hnsw (embedding vector_cosine_ops);

-- Graph nodes: unique entities extracted from the text
CREATE TABLE entities (
    id SERIAL PRIMARY KEY,
    name TEXT NOT NULL,
    entity_type TEXT, -- e.g. 'Person', 'Organization', 'Product', 'Event'
    source_chunk_id INTEGER REFERENCES chunks(id),
    UNIQUE (name, entity_type)
);

-- Graph edges: relationships (triplets) between entities
CREATE TABLE relationships (
    id SERIAL PRIMARY KEY,
    source_entity_id INTEGER REFERENCES entities(id) ON DELETE CASCADE,
    target_entity_id INTEGER REFERENCES entities(id) ON DELETE CASCADE,
    relation_type TEXT NOT NULL, -- e.g. 'ACQUIRED', 'FOUNDED', 'REPORTS_TO'
    source_chunk_id INTEGER REFERENCES chunks(id),
    confidence FLOAT DEFAULT 1.0
);

CREATE INDEX relationships_source_idx ON relationships(source_entity_id);
CREATE INDEX relationships_target_idx ON relationships(target_entity_id);

Notice that relationships stores source_entity_id, target_entity_id, and relation_type — that's our (subject, predicate, object) triplet. The source_chunk_id foreign key on both tables is critical: it lets us trace every graph fact back to the exact text it came from, which is what makes this pipeline hallucination-resistant rather than just "graph-flavored."

Step 2: Extracting Entity-Relationship Triplets with LangChain + Ollama

This is where the magic happens. We use a local LLM, constrained with structured output, to read each chunk and extract triplets.

# extract.py
from langchain_ollama import ChatOllama
from pydantic import BaseModel, Field
from typing import List

class Triplet(BaseModel):
    subject: str = Field(description="The entity performing or owning the relation")
    subject_type: str = Field(description="Category of the subject, e.g. Person, Organization")
    predicate: str = Field(description="The relationship, in UPPER_SNAKE_CASE, e.g. ACQUIRED")
    object: str = Field(description="The entity being related to")
    object_type: str = Field(description="Category of the object")

class ExtractionResult(BaseModel):
    triplets: List[Triplet]

llm = ChatOllama(model="llama3.1:8b-instruct", temperature=0)
structured_llm = llm.with_structured_output(ExtractionResult)

EXTRACTION_PROMPT = """You are a precise knowledge graph extraction engine.
Read the text below and extract every factual relationship as a triplet.

Rules:
- Only extract facts explicitly stated in the text. Do not infer or assume.
- Use consistent entity names (avoid pronouns; resolve "he"/"the company" to real names).
- Predicates must be short, UPPER_SNAKE_CASE verbs (e.g. FOUNDED, ACQUIRED, WORKS_AT).
- If no relationships exist, return an empty list.

Text:
{chunk_text}
"""

def extract_triplets(chunk_text: str) -> ExtractionResult:
    prompt = EXTRACTION_PROMPT.format(chunk_text=chunk_text)
    return structured_llm.invoke(prompt)

A few design choices worth calling out:

• temperature=0 — extraction should be deterministic and grounded, not creative. Any variance here directly translates into graph noise.
• with_structured_output — this forces the model to conform to our Pydantic schema, which drastically reduces malformed JSON parsing errors compared to asking for free-form JSON in the prompt.
• The explicit "only extract facts explicitly stated" instruction — this is your first and most important line of defense against hallucinated graph edges. If the LLM starts inventing relationships, your whole graph becomes unreliable.

Chunking and Running Extraction at Scale

# ingest.py
import psycopg
from langchain_ollama import OllamaEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter
from extract import extract_triplets

embeddings = OllamaEmbeddings(model="nomic-embed-text")
splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)

conn = psycopg.connect("dbname=graphrag user=postgres")

def ingest_document(document_id: str, raw_text: str):
    chunks = splitter.split_text(raw_text)

    with conn.cursor() as cur:
        for chunk_text in chunks:
            vector = embeddings.embed_query(chunk_text)

            cur.execute(
                """INSERT INTO chunks (document_id, content, embedding)
                   VALUES (%s, %s, %s) RETURNING id""",
                (document_id, chunk_text, vector),
            )
            chunk_id = cur.fetchone()[0]

            result = extract_triplets(chunk_text)

            for triplet in result.triplets:
                subj_id = upsert_entity(cur, triplet.subject, triplet.subject_type, chunk_id)
                obj_id = upsert_entity(cur, triplet.object, triplet.object_type, chunk_id)

                cur.execute(
                    """INSERT INTO relationships
                       (source_entity_id, target_entity_id, relation_type, source_chunk_id)
                       VALUES (%s, %s, %s, %s)""",
                    (subj_id, obj_id, triplet.predicate, chunk_id),
                )
        conn.commit()

def upsert_entity(cur, name: str, entity_type: str, chunk_id: int) -> int:
    cur.execute(
        """INSERT INTO entities (name, entity_type, source_chunk_id)
           VALUES (%s, %s, %s)
           ON CONFLICT (name, entity_type) DO UPDATE SET name = EXCLUDED.name
           RETURNING id""",
        (name, entity_type, chunk_id),
    )
    return cur.fetchone()[0]

The ON CONFLICT ... DO UPDATE clause is a neat trick for an "upsert that returns an id" — Postgres won't let you RETURNING on a no-op DO NOTHING, so we do a harmless no-op update instead to guarantee we always get the entity's id back, whether it's newly created or already existed.

Step 3: Multi-Hop Retrieval with Recursive CTEs

This is the part that replaces a graph database. PostgreSQL's recursive Common Table Expressions can walk a graph to an arbitrary depth directly in SQL.

-- Multi-hop traversal: find everything within N hops of a starting entity
WITH RECURSIVE graph_walk AS (
    -- Base case: start from our entry-point entity
    SELECT
        e1.id AS source_id,
        e1.name AS source_name,
        r.relation_type,
        e2.id AS target_id,
        e2.name AS target_name,
        1 AS hop_count,
        ARRAY[e1.id] AS visited
    FROM entities e1
    JOIN relationships r ON r.source_entity_id = e1.id
    JOIN entities e2 ON e2.id = r.target_entity_id
    WHERE e1.name ILIKE %(start_entity)s

    UNION ALL

    -- Recursive case: keep walking outward, avoiding cycles
    SELECT
        gw.target_id,
        gw.target_name,
        r.relation_type,
        e2.id,
        e2.name,
        gw.hop_count + 1,
        gw.visited || e2.id
    FROM graph_walk gw
    JOIN relationships r ON r.source_entity_id = gw.target_id
    JOIN entities e2 ON e2.id = r.target_entity_id
    WHERE gw.hop_count < %(max_hops)s
      AND e2.id != ALL(gw.visited)  -- cycle prevention
)
SELECT DISTINCT source_name, relation_type, target_name, hop_count
FROM graph_walk
ORDER BY hop_count;

The visited array and the e2.id != ALL(gw.visited) guard are non-negotiable in any recursive graph query — without cycle detection, a graph with a loop (e.g., A REPORTS_TO B, B REPORTS_TO A due to a bad extraction) will cause an infinite recursion and crash your query.

Wiring It Into the Retrieval Pipeline

# retrieve.py
import psycopg
from langchain_ollama import OllamaEmbeddings

embeddings = OllamaEmbeddings(model="nomic-embed-text")
conn = psycopg.connect("dbname=graphrag user=postgres")

def find_entry_entities(query: str, top_k: int = 3):
    """Step 1: Use vector search to find entry-point chunks, then their entities."""
    query_vector = embeddings.embed_query(query)
    with conn.cursor() as cur:
        cur.execute(
            """
            SELECT DISTINCT e.name
            FROM chunks c
            JOIN entities e ON e.source_chunk_id = c.id
            ORDER BY c.embedding <=> %s::vector
            LIMIT %s
            """,
            (query_vector, top_k),
        )
        return [row[0] for row in cur.fetchall()]

def multi_hop_context(entity_name: str, max_hops: int = 2):
    """Step 2: Traverse the graph outward from an entry entity."""
    with conn.cursor() as cur:
        cur.execute(RECURSIVE_QUERY, {"start_entity": entity_name, "max_hops": max_hops})
        return cur.fetchall()

def build_context(query: str) -> str:
    entry_entities = find_entry_entities(query)
    facts = []
    for entity in entry_entities:
        for row in multi_hop_context(entity):
            source, relation, target, hop = row
            facts.append(f"{source} --[{relation}]--> {target}")
    return "\n".join(sorted(set(facts)))

Finally, generation:

# generate.py
from langchain_ollama import ChatOllama
from retrieve import build_context

llm = ChatOllama(model="llama3.1:8b-instruct", temperature=0.2)

ANSWER_PROMPT = """Answer the question using ONLY the facts below.
If the facts don't contain enough information, say so explicitly — do not guess.

Facts (graph relationships):
{context}

Question: {question}
Answer:"""

def answer_question(question: str) -> str:
    context = build_context(question)
    prompt = ANSWER_PROMPT.format(context=context, question=question)
    return llm.invoke(prompt).content

That final instruction — "If the facts don't contain enough information, say so explicitly" — is the single highest-leverage line in this entire pipeline for reducing hallucinations. It gives the model explicit permission to admit uncertainty instead of filling gaps with plausible-sounding fiction.

Real-World Example: Corporate Knowledge Base

Let's say you've ingested three internal documents:

• Doc A: "Redwood Systems was founded by Maria Chen in 2019."
• Doc B: "Redwood Systems acquired Bluefin Analytics in 2024."
• Doc C: "Maria Chen now serves as CTO at Horizon Robotics."

A plain vector-RAG system, asked "Who founded the company that acquired Bluefin Analytics, and where do they work now?", would likely retrieve only Doc B and Doc C (highest similarity to "Bluefin Analytics" and "where do they work") — missing Doc A entirely, since it doesn't mention "Bluefin Analytics" at all.

Our GraphRAG pipeline instead extracts:

Maria Chen --[FOUNDED]--> Redwood Systems
Redwood Systems --[ACQUIRED]--> Bluefin Analytics
Maria Chen --[WORKS_AT]--> Horizon Robotics

Starting from the entity "Bluefin Analytics" (found via vector search), a 2-hop traversal correctly surfaces all three facts, because the graph connects them even though they never appear together in a single chunk. The LLM can now correctly answer: "Redwood Systems, founded by Maria Chen, acquired Bluefin Analytics. Maria Chen now works at Horizon Robotics."

Step 4: Exposing It in a Next.js Application

Wrap the Python retrieval/generation logic behind a small FastAPI service, then call it from a Next.js API route:

// app/api/ask/route.ts
import { NextRequest, NextResponse } from "next/server";

export async function POST(req: NextRequest) {
  const { question } = await req.json();

  if (!question || typeof question !== "string") {
    return NextResponse.json({ error: "Missing question" }, { status: 400 });
  }

  const res = await fetch("http://localhost:8000/answer", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ question }),
  });

  if (!res.ok) {
    return NextResponse.json({ error: "Retrieval service failed" }, { status: 502 });
  }

  const data = await res.json();
  return NextResponse.json({ answer: data.answer, sources: data.sources });
}

On the frontend, a simple component consumes this:

// app/components/AskGraph.tsx
"use client";
import { useState } from "react";

export default function AskGraph() {
  const [question, setQuestion] = useState("");
  const [answer, setAnswer] = useState("");
  const [loading, setLoading] = useState(false);

  async function handleAsk() {
    setLoading(true);
    const res = await fetch("/api/ask", {
      method: "POST",
      body: JSON.stringify({ question }),
    });
    const data = await res.json();
    setAnswer(data.answer);
    setLoading(false);
  }

  return (
    <div className="max-w-xl mx-auto p-4">
      <input
        className="border w-full p-2 rounded"
        value={question}
        onChange={(e) => setQuestion(e.target.value)}
        placeholder="Ask a multi-hop question..."
      />
      <button onClick={handleAsk} className="mt-2 px-4 py-2 bg-black text-white rounded">
        {loading ? "Thinking..." : "Ask"}
      </button>
      {answer && <p className="mt-4 whitespace-pre-wrap">{answer}</p>}
    </div>
  );
}

Returning sources (the source_chunk_id values we tracked back in Step 1) alongside the answer lets your UI show citations — a small addition that massively increases user trust in the system.

🚀 Pro Tips

• Deduplicate entities aggressively. "Maria Chen," "M. Chen," and "Chen" will fragment your graph into disconnected islands unless you run an entity-resolution pass (even a simple fuzzy-match + LLM-confirmation step helps enormously).
• Cap your traversal depth at 2–3 hops. Beyond that, the number of paths grows combinatorially and you'll flood your context window with low-relevance facts. Rank by relation relevance if you need deeper walks.
• Store confidence scores on relationships and filter low-confidence edges out of retrieval by default — this gives you a dial to trade off graph completeness against precision.
• Use a smaller, faster model for extraction and a larger model for generation. Extraction runs once per chunk at ingestion time (batch-friendly); generation runs per user query (latency-sensitive). An 8B model for extraction and generation is a good starting default, but consider a 70B-class model for generation if your hardware allows it.
• Version your graph schema. As you refine your entity_type and relation_type taxonomies, you'll want to re-run extraction on old documents — keep the source chunk reference so this is always possible.
• Add a nightly consolidation job that merges near-duplicate entities and relation types using embedding similarity on entity names — this keeps graph quality from degrading as you ingest more documents over time.

Common Mistakes to Avoid

• Skipping cycle detection in recursive queries. A single bidirectional relationship pair (A KNOWS B, B KNOWS A) will cause infinite recursion without a visited array guard — always test with intentionally cyclic data before shipping.
• Letting the extraction model run at high temperature. Creative extraction produces creative — i.e., fabricated — relationships. Keep temperature=0 for extraction, full stop.
• Treating the graph as the only source of truth. The graph is a lossy compression of your documents. Always keep the source_chunk_id link so the LLM (and your UI) can fall back to raw text for nuance the triplet extraction missed.
• Ignoring entity type collisions. Without a UNIQUE (name, entity_type) constraint, "Apple" the fruit and "Apple" the company will silently merge into one node, poisoning traversal results.
• Over-chunking documents. Chunks that are too small (e.g., single sentences) lose the surrounding context an extraction model needs to correctly identify what a pronoun refers to. 500–1000 characters with overlap is a reasonable starting point.
• Forgetting to index your foreign keys. relationships.source_entity_id and target_entity_id are on the hot path for every single traversal query — an unindexed graph query will fall over as soon as you exceed a few thousand relationships.

Best Practices

• Hybrid retrieval by default. Always combine vector search (for finding entry points) with graph traversal (for expanding context) — neither alone is sufficient for the full range of user questions.
• Instrument your pipeline. Log which entities were used as entry points, how many hops were walked, and how many facts were assembled into context for every query — this is invaluable for debugging bad answers.
• Validate extraction output against your schema strictly. Use Pydantic's structured output support in LangChain rather than parsing free-form JSON from the model — it eliminates an entire category of ingestion failures.
• Run evaluation on multi-hop QA pairs specifically, not just single-hop questions, since that's the exact capability GraphRAG is meant to unlock. Build a small internal benchmark of 2–3 hop questions and track answer accuracy over time as you tune chunk size, hop depth, and confidence thresholds.
• Keep the graph schema domain-specific but not overly rigid. Start with a small controlled vocabulary of relation_type values relevant to your domain (e.g., ACQUIRED, FOUNDED, REPORTS_TO, CAUSED) and expand it deliberately rather than letting the LLM invent new predicates unchecked.

📌 Key Takeaways

• Vector search alone can't answer questions that require chaining facts across multiple documents — it retrieves by similarity, not by relationship.
• GraphRAG adds a structured entity-relationship layer on top of your existing embeddings, giving retrieval an actual path to traverse.
• You don't need a dedicated graph database to implement GraphRAG — PostgreSQL with pgvector and recursive CTEs handles both vector and graph retrieval in a single, operationally simple system.
• A fully local stack (Ollama + LangChain + PostgreSQL) makes this architecture practical for privacy-sensitive or air-gapped environments without sacrificing capability.
• Grounding every extracted triplet in a source_chunk_id, and instructing your generation model to admit uncertainty, are the two most effective levers for reducing hallucinations in this pipeline.

Conclusion

GraphRAG isn't a replacement for vector-based RAG — it's the missing second half of the retrieval story. Vector search excels at finding what's relevant; graph traversal excels at finding what's connected. By running both on top of a single PostgreSQL instance, you get a genuinely local, auditable, and cost-free way to answer the multi-hop questions that trip up conventional RAG systems every day.

The pipeline we built in this tutorial — chunk, extract, store as a graph, retrieve hybridly, generate with explicit grounding — is a pattern you can extend far beyond the corporate knowledge base example: customer support ticket correlation, research paper citation graphs, legal contract clause tracing, or incident root-cause analysis are all naturally multi-hop problems waiting for exactly this architecture.

Start small: ingest a handful of documents, inspect the extracted triplets by hand, and tune your extraction prompt before scaling up. A clean, well-typed graph of a hundred documents will outperform a noisy graph of ten thousand every time.

References

• LangChain Documentation — python.langchain.com
• Ollama Model Library — ollama.com/library
• PostgreSQL Recursive Queries (WITH RECURSIVE) — postgresql.org/docs
• pgvector Extension — github.com/pgvector/pgvector
• Microsoft Research, "From Local to Global: A Graph RAG Approach to Query-Focused Summarization"
• Next.js Documentation — nextjs.org/docs