Building a Production-Ready RAG System with LangChain and Next.js

Introduction

Large Language Models are remarkably good at reasoning and language generation, but they have a fundamental limitation: their knowledge is frozen at training time, and they know nothing about your private documents, internal wikis, or proprietary data. Ask a vanilla LLM about your company's refund policy or last week's product release notes, and it will either refuse to answer or — worse — confidently make something up.

Retrieval Augmented Generation (RAG) solves this problem by combining a retrieval system with a language model. Instead of relying purely on what the model memorized during training, RAG fetches relevant context from an external knowledge base at query time and feeds it into the model's prompt. The result is an assistant that can answer questions grounded in real, current, and private data.

In this guide, we'll build a production-ready RAG system from the ground up using:

LangChain for orchestrating document ingestion, retrieval, and generation
Next.js (App Router) for a modern, streaming-capable frontend and API layer
A vector database (we'll use Pinecone as our primary example, with notes on alternatives) for semantic search
OpenAI or Anthropic models for embeddings and generation

By the end, you'll understand not just how to get RAG "working" in a demo, but how to make it resilient, observable, and scalable enough to trust in production.

What Is RAG, Really?

At its core, RAG is a two-stage pipeline:

Retrieval — Given a user's query, search a knowledge base (typically a vector store) for the most semantically relevant chunks of text.
Generation — Pass those retrieved chunks, along with the original query, into an LLM prompt so it can generate a grounded, contextual answer.

flowchart LR
    A[User Query] --> B[Embed Query]
    B --> C[Vector Search]
    C --> D[Retrieved Chunks]
    D --> E[Augmented Prompt]
    A --> E
    E --> F[LLM Generation]
    F --> G[Grounded Answer]

This sounds simple, but each stage hides a surprising amount of complexity. How do you split documents into chunks? How do you choose an embedding model? How do you handle documents that update frequently? How do you prevent the system from retrieving irrelevant or contradictory context? These are the questions that separate a weekend prototype from a system you can actually ship.

Why Not Just Fine-Tune the Model Instead?

A common question is: "Why not just fine-tune an LLM on our data instead of doing retrieval?" Fine-tuning teaches a model new patterns and style, but it's a poor mechanism for injecting facts — especially facts that change often. Fine-tuning is expensive to keep updated, doesn't scale well to large or frequently changing corpora, and doesn't give you citations or traceability. RAG, by contrast, lets you swap out the underlying documents at any time without retraining anything, and it naturally supports showing users where an answer came from.

Why LangChain and Next.js?

LangChain has matured into a solid orchestration layer for LLM applications. Its main value in a RAG context is:

A large ecosystem of document loaders (PDF, Notion, Confluence, S3, web pages, etc.)
Built-in text splitters tuned for different content types
A unified interface across embedding providers and vector stores
LCEL (LangChain Expression Language), which lets you compose retrievers, prompts, and models into a single pipeable chain that supports streaming, batching, and async execution out of the box

Next.js, especially with the App Router, is a natural fit for the frontend because:

Route Handlers give you a clean, serverless-friendly API layer colocated with your frontend code
Native support for streaming responses, which is essential for a responsive chat experience
Server Components let you fetch and render initial data (like conversation history) without extra client-side round trips
Easy deployment on edge/serverless platforms with good cold-start characteristics

Together, they let a single team own both the AI pipeline and the user-facing product without juggling two separate stacks.

System Architecture

Before writing code, it helps to sketch the full architecture. A production RAG system typically has two distinct pipelines:

1. Ingestion Pipeline (Offline / Batch)

This runs whenever new documents are added or updated:

Documents → Loaders → Splitters → Embeddings → Vector Store

2. Query Pipeline (Online / Real-Time)

This runs on every user request:

User Query → Query Embedding → Vector Search → Context Assembly
→ Prompt Construction → LLM → Streamed Response → Client

Keeping these pipelines separate is important. Ingestion is I/O-heavy and can run as a background job or scheduled task; the query pipeline needs to be fast and low-latency since a real user is waiting on the other end.

Setting Up the Project

Let's scaffold the project. We'll use a standard Next.js App Router project with LangChain's JavaScript SDK.

npx create-next-app@latest rag-app --typescript --app --tailwind
cd rag-app
npm install langchain @langchain/openai @langchain/pinecone @langchain/community
npm install @pinecone-database/pinecone

Set up your environment variables in .env.local:

OPENAI_API_KEY=sk-...
PINECONE_API_KEY=...
PINECONE_INDEX=rag-production-index

Building the Ingestion Pipeline

The quality of your RAG system is determined far more by ingestion quality than by which LLM you choose. Let's build this carefully.

Step 1: Loading Documents

LangChain provides loaders for nearly any source. Here's an example loading PDFs from a directory:

// scripts/ingest.ts
import { DirectoryLoader } from "langchain/document_loaders/fs/directory";
import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";

const loader = new DirectoryLoader("./data", {
  ".pdf": (path) => new PDFLoader(path),
});

const rawDocs = await loader.load();
console.log(`Loaded ${rawDocs.length} documents`);

Step 2: Chunking Strategy

This is the single most underrated decision in a RAG system. Chunk too large, and you dilute relevance and blow your context budget. Chunk too small, and you lose coherence and context.

import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";

const splitter = new RecursiveCharacterTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 150,
  separators: ["\n\n", "\n", ". ", " ", ""],
});

const splitDocs = await splitter.splitDocuments(rawDocs);

A few practical guidelines:

1,000 characters (~200-250 tokens) with 10-15% overlap is a solid default for prose-heavy content.
For structured content (FAQs, code docs), consider splitting by semantic boundaries (headings, functions) rather than raw character count.
Always attach metadata (source file, page number, section title) to each chunk — this is what enables citations later.

splitDocs.forEach((doc, i) => {
  doc.metadata = {
    ...doc.metadata,
    chunkId: i,
    ingestedAt: new Date().toISOString(),
  };
});

Step 3: Embedding and Storing

import { OpenAIEmbeddings } from "@langchain/openai";
import { PineconeStore } from "@langchain/pinecone";
import { Pinecone } from "@pinecone-database/pinecone";

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const index = pinecone.Index(process.env.PINECONE_INDEX!);

const embeddings = new OpenAIEmbeddings({
  model: "text-embedding-3-large",
});

await PineconeStore.fromDocuments(splitDocs, embeddings, {
  pineconeIndex: index,
  maxConcurrency: 5,
});

console.log("Ingestion complete.");

Run this as a standalone script (npx tsx scripts/ingest.ts) or wire it into a CI job / scheduled cron so your knowledge base stays fresh.

Building the Retrieval and Generation Chain

Now for the query-time pipeline, using LCEL to compose everything cleanly.

// lib/rag-chain.ts
import { ChatOpenAI } from "@langchain/openai";
import { PineconeStore } from "@langchain/pinecone";
import { OpenAIEmbeddings } from "@langchain/openai";
import { Pinecone } from "@pinecone-database/pinecone";
import { ChatPromptTemplate } from "@langchain/core/prompts";
import { StringOutputParser } from "@langchain/core/output_parsers";
import {
  RunnablePassthrough,
  RunnableSequence,
} from "@langchain/core/runnables";
import { formatDocumentsAsString } from "langchain/util/document";

const pinecone = new Pinecone({ apiKey: process.env.PINECONE_API_KEY! });
const index = pinecone.Index(process.env.PINECONE_INDEX!);

const embeddings = new OpenAIEmbeddings({ model: "text-embedding-3-large" });

export async function getRetriever() {
  const vectorStore = await PineconeStore.fromExistingIndex(embeddings, {
    pineconeIndex: index,
  });
  return vectorStore.asRetriever({ k: 5 });
}

const SYSTEM_PROMPT = `You are a helpful assistant that answers questions
strictly based on the provided context. If the answer isn't in the context,
say you don't know — do not make up information. Cite the source file for
any claim you make.

Context:
{context}`;

const prompt = ChatPromptTemplate.fromMessages([
  ["system", SYSTEM_PROMPT],
  ["human", "{question}"],
]);

const model = new ChatOpenAI({
  model: "gpt-4o",
  temperature: 0.1,
  streaming: true,
});

export async function buildRagChain() {
  const retriever = await getRetriever();

  return RunnableSequence.from([
    {
      context: retriever.pipe(formatDocumentsAsString),
      question: new RunnablePassthrough(),
    },
    prompt,
    model,
    new StringOutputParser(),
  ]);
}

Note the low temperature — for factual, grounded answers you want the model to stay close to the retrieved context rather than being creative. Also note the explicit instruction to say "I don't know" when the context is insufficient. This single line prevents a huge share of hallucination issues in practice.

Exposing the Chain via a Next.js Route Handler

Now let's wire this into a Next.js API route that streams tokens back to the client as they're generated.

// app/api/chat/route.ts
import { buildRagChain } from "@/lib/rag-chain";
import { StreamingTextResponse } from "ai";

export const runtime = "nodejs";

export async function POST(req: Request) {
  const { question } = await req.json();

  if (!question || typeof question !== "string") {
    return new Response("Missing 'question' field", { status: 400 });
  }

  const chain = await buildRagChain();
  const stream = await chain.stream(question);

  const encoder = new TextEncoder();
  const readable = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        controller.enqueue(encoder.encode(chunk));
      }
      controller.close();
    },
  });

  return new StreamingTextResponse(readable);
}

Streaming matters more than most developers assume for RAG applications specifically, because generation latency is already higher than a plain chat completion (you're paying the retrieval cost plus generation cost). Streaming tokens as they arrive keeps perceived latency low even when total response time is a few seconds.

Building the Frontend Chat Interface

On the client, we consume the stream and render it incrementally.

// app/components/ChatBox.tsx
"use client";

import { useState } from "react";

export default function ChatBox() {
  const [question, setQuestion] = useState("");
  const [answer, setAnswer] = useState("");
  const [loading, setLoading] = useState(false);

  async function handleSubmit(e: React.FormEvent) {
    e.preventDefault();
    setLoading(true);
    setAnswer("");

    const res = await fetch("/api/chat", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ question }),
    });

    const reader = res.body?.getReader();
    const decoder = new TextDecoder();

    if (!reader) return;

    while (true) {
      const { done, value } = await reader.read();
      if (done) break;
      setAnswer((prev) => prev + decoder.decode(value));
    }

    setLoading(false);
  }

  return (
    <div className="max-w-2xl mx-auto p-4">
      <form onSubmit={handleSubmit} className="flex gap-2">
        <input
          value={question}
          onChange={(e) => setQuestion(e.target.value)}
          placeholder="Ask something about your documents..."
          className="flex-1 border rounded px-3 py-2"
        />
        <button
          type="submit"
          disabled={loading}
          className="bg-black text-white px-4 py-2 rounded disabled:opacity-50"
        >
          {loading ? "Thinking..." : "Ask"}
        </button>
      </form>
      {answer && (
        <div className="mt-4 whitespace-pre-wrap border rounded p-4 bg-gray-50">
          {answer}
        </div>
      )}
    </div>
  );
}

This is intentionally minimal — in a real product you'd add conversation history, source citations rendered as clickable links, and error states, but this gives you the core streaming loop working end to end.

Making It Production-Ready

Getting a RAG demo working is a weekend project. Making it production-ready is where the real engineering happens. Here's what changes.

1. Hybrid Search, Not Just Vector Search

Pure semantic search struggles with exact-match queries (product codes, names, acronyms). Combine vector search with keyword-based (BM25) search and merge results — most managed vector databases (Pinecone, Weaviate, Qdrant) now support hybrid search natively.

2. Re-ranking

Retrieve more candidates than you need (say, top 20) and then re-rank them with a dedicated cross-encoder re-ranker (e.g., Cohere Rerank) before passing the top 5 to the LLM. This consistently improves answer quality more than almost any other single change.

import { CohereRerank } from "@langchain/cohere";

const reranker = new CohereRerank({ model: "rerank-english-v3.0" });
const rerankedDocs = await reranker.compressDocuments(candidateDocs, question);

3. Caching

Cache embeddings for repeated queries and cache full responses for identical questions where appropriate (e.g., FAQ-style queries). Semantic caching — matching new queries against previously answered ones by embedding similarity — can meaningfully cut both cost and latency.

4. Guardrails and Grounding Checks

Add a lightweight post-generation check that verifies the answer is actually supported by the retrieved context, and falls back to "I don't have enough information" when it isn't. This can be a smaller, cheaper LLM call or a rule-based check on citation presence.

5. Observability

Instrument every stage: retrieval latency, number of chunks retrieved, token usage, generation latency, and user feedback (thumbs up/down). LangChain integrates with LangSmith for exactly this kind of tracing, which becomes invaluable when debugging why a specific answer went wrong.

6. Rate Limiting and Cost Control

LLM and embedding calls cost real money per request. Add rate limiting at the API route level (e.g., using Upstash Redis) and set hard caps on max_tokens and retrieved chunk count to avoid runaway costs from adversarial or accidental usage spikes.

7. Handling Document Updates

Vector stores don't automatically know when a source document is deleted or updated. Track a content hash per document and re-embed only changed chunks; delete stale vectors by their source metadata when a document is removed.

Best Practices

Version your prompts. Treat prompt templates like code — store them in version control and log which version generated each response.
Set a strict "grounded-only" system prompt. Explicitly instruct the model to decline when context is insufficient.
Keep chunk metadata rich. Source, page number, and timestamps make citations and debugging dramatically easier.
Evaluate continuously. Build a small golden dataset of question/answer pairs and run automated evaluation (e.g., with RAGAS or LangSmith evaluators) on every pipeline change.
Separate ingestion from serving infrastructure. Ingestion jobs should never share compute or rate limits with your live query path.
Use streaming everywhere on the frontend. It's the single highest-leverage UX improvement for LLM-backed apps.

🚀 Pro Tips

Tune chunkOverlap per content type. Legal or technical documents benefit from higher overlap (20%+) to preserve cross-reference context; conversational content needs less.
Use k + reranking, not just a bigger k. Retrieving 5 well-ranked chunks beats retrieving 15 unranked ones — bigger context windows are not a substitute for retrieval quality.
Log retrieved chunks alongside answers. When users report a bad answer, the first thing to check is what was retrieved, not what the model generated.
Set temperature near zero for factual RAG. Save higher temperatures for summarization or creative use cases layered on top of retrieval.
Pre-warm your vector store connection in serverless environments to avoid cold-start latency spikes on the first request.

Common Mistakes to Avoid

Treating chunking as an afterthought. Poor chunking is the number one cause of bad RAG answers — teams often spend all their effort tuning prompts when the real problem is upstream.
Skipping metadata. Without source metadata, you can't build citations, can't debug retrieval quality, and can't safely delete outdated content.
Not testing retrieval in isolation. Always evaluate the retriever's precision/recall separately from the full generation pipeline — a great LLM can't fix bad retrieval.
Ignoring token budgets. Stuffing too many chunks into the prompt increases cost and can actually reduce answer quality due to the "lost in the middle" effect in long contexts.
No fallback behavior. Systems that always attempt an answer, even with irrelevant context, erode user trust. Explicit "I don't know" responses are a feature, not a failure.
Forgetting to re-embed on document updates. Stale vectors silently degrade answer accuracy over time.
Blocking the UI on full generation. Not streaming responses makes even a fast RAG pipeline feel sluggish.

📌 Key Takeaways

RAG grounds LLM responses in your own data by retrieving relevant context at query time, which reduces hallucinations and keeps answers current without retraining the model.
Ingestion quality — chunking strategy, metadata, and embedding choice — has a bigger impact on answer quality than which LLM you use for generation.
LangChain's LCEL makes it easy to compose retrievers, prompts, and models into a single streamable, observable chain.
Next.js Route Handlers paired with streaming responses deliver a responsive chat experience while keeping your API layer clean and type-safe.
Production readiness requires hybrid search, re-ranking, caching, observability, and explicit grounding checks — not just a working retrieval loop.

Conclusion

A working RAG demo is easy to build in an afternoon; a RAG system you can trust in production is a different challenge entirely. The core loop — retrieve, augment, generate — stays the same, but production quality comes from the details: how you chunk documents, how you rank retrieved context, how you handle stale data, and how you observe the system when things go wrong.

LangChain gives you the orchestration primitives to compose this pipeline cleanly, while Next.js gives you a fast, streaming-capable frontend and API layer that keeps the whole stack in one codebase. Start with the architecture in this guide, instrument everything from day one, and iterate on retrieval quality before you touch the prompt — that ordering alone will save you weeks of debugging.