Architecting an Autonomous Market Intelligence SaaS: Orchestrating n8n, LangGraph, and Ollama with Next.js

Introduction: Why Scrapers Break and Agents Don't (As Often)

If you have ever maintained a lead enrichment scraper, you know the routine. It works perfectly for three weeks. Then a target site renames one CSS class, wraps its pricing table in a new component, or adds a cookie wall, and your pipeline starts quietly writing null into half your database. Nobody notices until a customer emails asking why their weekly report says a 400-person company has "0 employees."

That fragility is exactly why Data-as-a-Service (DaaS) has become such an interesting space in 2026. Buyers no longer want raw CSV dumps. They want structured, trustworthy market intelligence: who a company is hiring, what they charge, which tools they use, and what changed since last week. And they will pay a recurring subscription for it, as long as the data stays fresh and accurate.

In this guide, I'll walk through how I would architect an autonomous market intelligence SaaS from scratch. Not a toy demo, but a system you could actually put in front of paying customers. Here's the stack:

• n8n for scheduling, triggers, and workflow orchestration
• Node.js + TypeScript REST API as the job gateway
• LangGraph for stateful, self-healing AI agents
• Ollama for privacy-first local LLM inference
• PostgreSQL for structured intelligence, raw snapshots, and entitlements
• Stripe and Razorpay for multi-tier subscriptions across global and Indian markets
• Next.js, TypeScript, and Shadcn UI for the customer dashboard
• Docker on AWS for deployment

By the end, you'll have a clear mental model of how the pieces talk to each other, plus working code for the parts that matter most.

The Big Picture: How the System Fits Together

Before writing any code, let's agree on the flow. One clean way to think about it is as five layers, each with a single job:

1. Orchestration layer (n8n): Decides when work happens. Cron schedules, customer-triggered refreshes, and webhook events all live here.
2. Gateway layer (Node.js REST API): Decides whether work is allowed. It authenticates requests, checks entitlements, de-duplicates jobs, and pushes them into a queue.
3. Intelligence layer (LangGraph + Ollama): Does the actual thinking. It fetches pages, extracts structured data, validates it, and repairs its own mistakes.
4. Storage layer (PostgreSQL): Remembers everything, from raw HTML snapshots and structured company profiles to change history and billing state.
5. Experience layer (Next.js dashboard): Shows customers what they paid for and lets them export reports.

The key design decision here is separation of concerns between n8n and LangGraph. I've seen teams try to cram complex agent logic into n8n's AI nodes, and it works until you need loops, branching retries, unit tests, and version control. On the flip side, writing your own scheduler and integration layer in pure code is a waste of time when n8n already does it well.

So the rule is simple: n8n orchestrates, LangGraph reasons. They communicate over plain HTTP.

A Real-World Example

Imagine a customer, a B2B sales team selling HR software. They upload a list of 2,000 target company domains and subscribe to the "Growth" plan. Every Monday they want to know:

• Which companies posted new engineering or HR roles
• Whether pricing pages changed
• Which tech stack signals appeared (for example, a new careers page powered by a specific ATS)
• A short, human-readable "why this account is warm" summary

The system needs to process those 2,000 domains every week, gracefully handle sites that go down or change layout, and deliver a clean report without anyone babysitting it. Let's build that.

Layer 1: Orchestrating Workflows with n8n

n8n shines as a "nervous system" for your SaaS. In this architecture, I use it for three kinds of workflows:

• Scheduled sweeps: A cron trigger fires every Sunday night, pulls all active watchlists from PostgreSQL, and batches domains into jobs.
• On-demand refreshes: When a customer clicks "Refresh now" in the dashboard, Next.js calls an n8n webhook.
• Post-processing: When the agent service finishes a batch, it calls back into n8n, which generates the report, emails it, and pings Slack.

Here's a trimmed version of the scheduled sweep workflow, exported as JSON. In real life you'd build this visually, but seeing the structure helps:

{
  "name": "Weekly Intelligence Sweep",
  "nodes": [
    {
      "name": "Every Sunday 22:00",
      "type": "n8n-nodes-base.scheduleTrigger",
      "parameters": {
        "rule": { "interval": [{ "field": "cronExpression", "expression": "0 22 * * 0" }] }
      }
    },
    {
      "name": "Fetch Active Watchlists",
      "type": "n8n-nodes-base.postgres",
      "parameters": {
        "operation": "executeQuery",
        "query": "SELECT w.id AS watchlist_id, w.tenant_id, d.domain FROM watchlists w JOIN watchlist_domains d ON d.watchlist_id = w.id JOIN entitlements e ON e.tenant_id = w.tenant_id WHERE e.status = 'active';"
      }
    },
    {
      "name": "Split Into Batches",
      "type": "n8n-nodes-base.splitInBatches",
      "parameters": { "batchSize": 50 }
    },
    {
      "name": "Enqueue Batch",
      "type": "n8n-nodes-base.httpRequest",
      "parameters": {
        "method": "POST",
        "url": "={{ $env.AGENT_API_URL }}/v1/jobs/batch",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{ JSON.stringify({ items: $input.all().map(i => i.json) }) }}"
      }
    }
  ]
}

A couple of things worth noticing. First, n8n does not do any scraping here. It only decides what needs doing and hands it off. Second, the batch size is deliberately small. If one HTTP call fails, n8n's built-in retry only re-sends 50 domains, not 2,000.

Layer 2: The Node.js REST Gateway and Job Queue

The gateway is boring on purpose. Its job is to be the strict bouncer at the door. Every job request goes through it, whether it comes from n8n, the dashboard, or a customer's API key.

I like Fastify for this in 2026 because its schema-first approach pairs nicely with Zod and TypeScript, and it's fast without much tuning. For the queue, BullMQ on Redis is still the pragmatic choice for Node.js.

// apps/gateway/src/routes/jobs.ts
import { FastifyInstance } from "fastify";
import { z } from "zod";
import { Queue } from "bullmq";
import { createHash } from "node:crypto";
import { db } from "../db";
import { assertEntitlement } from "../billing/entitlements";

const enrichQueue = new Queue("enrichment", {
  connection: { url: process.env.REDIS_URL },
  defaultJobOptions: {
    attempts: 4,
    backoff: { type: "exponential", delay: 15_000 },
    removeOnComplete: 1000,
    removeOnFail: 5000,
  },
});

const BatchSchema = z.object({
  items: z
    .array(
      z.object({
        tenant_id: z.string().uuid(),
        watchlist_id: z.string().uuid(),
        domain: z.string().min(3).max(253),
      })
    )
    .max(200),
});

export async function jobRoutes(app: FastifyInstance) {
  app.post("/v1/jobs/batch", async (req, reply) => {
    const { items } = BatchSchema.parse(req.body);

    const accepted: string[] = [];

    for (const item of items) {
      await assertEntitlement(item.tenant_id, "enrichment.run");

      // Idempotency: same tenant + domain + ISO week = same job
      const week = getIsoWeek(new Date());
      const idempotencyKey = createHash("sha256")
        .update(`${item.tenant_id}:${item.domain}:${week}`)
        .digest("hex");

      const inserted = await db.query(
        `INSERT INTO enrichment_jobs (id, tenant_id, watchlist_id, domain, status)
         VALUES ($1, $2, $3, $4, 'queued')
         ON CONFLICT (id) DO NOTHING
         RETURNING id`,
        [idempotencyKey, item.tenant_id, item.watchlist_id, item.domain]
      );

      if (inserted.rowCount) {
        await enrichQueue.add("enrich-domain", item, { jobId: idempotencyKey });
        accepted.push(idempotencyKey);
      }
    }

    return reply.code(202).send({ accepted: accepted.length, jobIds: accepted });
  });
}

function getIsoWeek(d: Date): string {
  const date = new Date(Date.UTC(d.getFullYear(), d.getMonth(), d.getDate()));
  const day = date.getUTCDay() || 7;
  date.setUTCDate(date.getUTCDate() + 4 - day);
  const yearStart = new Date(Date.UTC(date.getUTCFullYear(), 0, 1));
  const week = Math.ceil(((date.getTime() - yearStart.getTime()) / 86400000 + 1) / 7);
  return `${date.getUTCFullYear()}-W${week}`;
}

Three habits here have saved me a lot of pain:

• Return 202 Accepted, never 200 OK, for asynchronous work. It sets the right expectation for callers that the job is queued, not done.
• Derive the job ID deterministically. If n8n retries the same request, Postgres and BullMQ both quietly ignore the duplicate. That one line prevents double-billing and double-scraping.
• Check entitlements at the gateway, not inside the agent. The agent should never have to care whether a customer's card expired.

Layer 3: Building Self-Healing Agents with LangGraph and Ollama

This is the heart of the platform, and where most of the interesting engineering lives.

Why LangGraph?

A traditional scraper is a straight line: fetch, parse, save. When parsing fails, the whole thing fails. An agentic pipeline is a graph with loops. It can notice its own failure, think about why, and try a different strategy.

LangGraph models this explicitly. You define a shared state, a set of nodes (functions that read and update that state), and edges (including conditional edges that decide where to go next). It also supports checkpointing, so a long-running job can resume after a crash instead of starting over.

Why Ollama?

Lead enrichment involves processing a lot of third-party web content, and customers increasingly ask, "Where does our data go?" Running models locally through Ollama means:

• No per-token API bills that scale linearly with your customer count
• Prospect lists and scraped content never leave your infrastructure
• Predictable latency you can tune with your own GPU capacity

Ollama also supports structured outputs by passing a JSON schema in the format field, which is a huge win for extraction tasks. The model is constrained to produce valid JSON matching your shape, instead of chatty prose you have to regex apart.

The Self-Healing Extraction Graph

Here's the agent's flow in plain English:

1. Fetch the page (with a headless browser fallback for JavaScript-heavy sites).
2. Clean the HTML into compact, readable text.
3. Extract structured fields using a local model with a JSON schema.
4. Validate the result with Zod plus business rules (for example, employee count must be a positive number).
5. If validation fails, Reflect: ask the model to explain what went wrong and adjust the strategy, such as trying the /about or /careers page, or switching to a bigger model.
6. Retry up to a fixed budget, then either Persist the result or mark it for human review.

Let's look at the code.

// apps/agents/src/graph/enrichment-graph.ts
import { Annotation, StateGraph, START, END } from "@langchain/langgraph";
import { ChatOllama } from "@langchain/ollama";
import { z } from "zod";
import { fetchPage, htmlToText } from "../tools/fetch";
import { persistProfile, flagForReview } from "../tools/persist";

export const CompanyProfileSchema = z.object({
  companyName: z.string().min(1),
  tagline: z.string().nullable(),
  industry: z.string().nullable(),
  employeeRange: z.enum(["1-10", "11-50", "51-200", "201-500", "501-1000", "1000+"]).nullable(),
  hqCountry: z.string().nullable(),
  openRoles: z.array(z.object({ title: z.string(), department: z.string().nullable() })),
  pricingModel: z.enum(["free", "freemium", "subscription", "usage", "enterprise", "unknown"]),
  techSignals: z.array(z.string()).max(25),
});
type CompanyProfile = z.infer<typeof CompanyProfileSchema>;

const EnrichmentState = Annotation.Root({
  tenantId: Annotation<string>(),
  domain: Annotation<string>(),
  targetPath: Annotation<string>({ reducer: (_, b) => b, default: () => "/" }),
  pageText: Annotation<string>({ reducer: (_, b) => b, default: () => "" }),
  profile: Annotation<CompanyProfile | null>({ reducer: (_, b) => b, default: () => null }),
  errors: Annotation<string[]>({ reducer: (a, b) => a.concat(b), default: () => [] }),
  attempts: Annotation<number>({ reducer: (_, b) => b, default: () => 0 }),
  modelName: Annotation<string>({ reducer: (_, b) => b, default: () => "qwen2.5:14b" }),
});
type State = typeof EnrichmentState.State;

const MAX_ATTEMPTS = 3;

function llm(model: string) {
  return new ChatOllama({
    baseUrl: process.env.OLLAMA_URL ?? "http://ollama:11434",
    model,
    temperature: 0,
    numCtx: 16384,
  });
}

async function fetchNode(state: State): Promise<Partial<State>> {
  const url = `https://${state.domain}${state.targetPath}`;
  const html = await fetchPage(url, { timeoutMs: 20_000, allowHeadless: true });
  return { pageText: htmlToText(html).slice(0, 40_000) };
}

async function extractNode(state: State): Promise<Partial<State>> {
  const extractor = llm(state.modelName).withStructuredOutput(CompanyProfileSchema, {
    name: "company_profile",
  });

  const hints = state.errors.length
    ? `Previous attempts failed for these reasons:\n- ${state.errors.join("\n- ")}\nAvoid repeating them.`
    : "";

  try {
    const profile = await extractor.invoke([
      {
        role: "system",
        content:
          "You extract factual company data from website text. Only use information present in the text. " +
          "Use null when a field is not stated. Never guess employee counts.",
      },
      { role: "user", content: `${hints}\n\nWEBSITE TEXT (${state.domain}${state.targetPath}):\n${state.pageText}` },
    ]);
    return { profile, attempts: state.attempts + 1 };
  } catch (err) {
    return { profile: null, attempts: state.attempts + 1, errors: [`extraction_error: ${String(err)}`] };
  }
}

async function validateNode(state: State): Promise<Partial<State>> {
  if (!state.profile) return {};
  const problems: string[] = [];

  if (!state.pageText.toLowerCase().includes(state.profile.companyName.toLowerCase().split(" ")[0])) {
    problems.push("companyName does not appear in the page text, likely hallucinated");
  }
  if (state.profile.openRoles.length === 0 && state.targetPath === "/") {
    problems.push("no roles found on homepage; careers data probably lives on another page");
  }

  return problems.length ? { errors: problems, profile: null } : {};
}

async function reflectNode(state: State): Promise<Partial<State>> {
  const lastError = state.errors.at(-1) ?? "";
  // Simple, deterministic strategy shifts before asking a model for help
  if (lastError.includes("careers")) return { targetPath: "/careers" };
  if (state.attempts >= 2) return { modelName: "qwen2.5:32b" };
  return { targetPath: "/about" };
}

function routeAfterValidate(state: State) {
  if (state.profile) return "persist";
  if (state.attempts >= MAX_ATTEMPTS) return "review";
  return "reflect";
}

export const enrichmentGraph = new StateGraph(EnrichmentState)
  .addNode("fetch", fetchNode)
  .addNode("extract", extractNode)
  .addNode("validate", validateNode)
  .addNode("reflect", reflectNode)
  .addNode("persist", async (s: State) => {
    await persistProfile(s.tenantId, s.domain, s.profile!);
    return {};
  })
  .addNode("review", async (s: State) => {
    await flagForReview(s.tenantId, s.domain, s.errors);
    return {};
  })
  .addEdge(START, "fetch")
  .addEdge("fetch", "extract")
  .addEdge("extract", "validate")
  .addConditionalEdges("validate", routeAfterValidate, ["persist", "reflect", "review"])
  .addEdge("reflect", "fetch")
  .addEdge("persist", END)
  .addEdge("review", END)
  .compile();

Notice what makes this "self-healing" without being magical:

• The validator is deterministic. It doesn't ask the model, "Are you sure?" It checks facts against the source text. That's the most reliable defense against hallucinated company names.
• Errors accumulate in state. Each retry feeds previous failure reasons back into the prompt, so the model doesn't repeat the same mistake.
• Reflection starts cheap. Before spending tokens on a model-driven plan, it tries obvious strategy shifts: a different page, then a bigger model.
• There's a hard budget. After three attempts, the job goes to a human review queue instead of looping forever and burning GPU time.

Wiring the Graph to the Queue

The BullMQ worker is the bridge between the gateway and the graph:

// apps/agents/src/worker.ts
import { Worker } from "bullmq";
import { enrichmentGraph } from "./graph/enrichment-graph";
import { db } from "./db";

const worker = new Worker(
  "enrichment",
  async (job) => {
    const { tenant_id, domain } = job.data;
    await db.query(`UPDATE enrichment_jobs SET status = 'running', started_at = now() WHERE id = $1`, [job.id]);

    const result = await enrichmentGraph.invoke(
      { tenantId: tenant_id, domain },
      { recursionLimit: 25, configurable: { thread_id: job.id } }
    );

    const status = result.profile ? "succeeded" : "needs_review";
    await db.query(
      `UPDATE enrichment_jobs SET status = $2, attempts = $3, finished_at = now() WHERE id = $1`,
      [job.id, status, result.attempts]
    );
    return { status };
  },
  {
    connection: { url: process.env.REDIS_URL },
    concurrency: Number(process.env.WORKER_CONCURRENCY ?? 4),
    limiter: { max: 30, duration: 60_000 }, // stay polite to target sites
  }
);

worker.on("failed", (job, err) => {
  console.error(JSON.stringify({ level: "error", jobId: job?.id, msg: err.message }));
});

Set concurrency based on your GPU, not your CPU. A single mid-range GPU serving a 14B model will not happily handle 50 parallel requests. Measure tokens per second, then size concurrency so the Ollama queue stays short.

Layer 4: Storing Structured Intelligence in PostgreSQL

PostgreSQL is the boring, brilliant backbone of this whole system. With JSONB, generated columns, row-level security (RLS), and optional pgvector for semantic search, you rarely need a second database.

Here's a schema that has served me well:

-- Multi-tenant core
CREATE TABLE tenants (
  id          uuid PRIMARY KEY DEFAULT gen_random_uuid(),
  name        text NOT NULL,
  created_at  timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE enrichment_jobs (
  id            text PRIMARY KEY,          -- deterministic idempotency key
  tenant_id     uuid NOT NULL REFERENCES tenants(id),
  watchlist_id  uuid NOT NULL,
  domain        text NOT NULL,
  status        text NOT NULL CHECK (status IN ('queued','running','succeeded','needs_review','failed')),
  attempts      int  NOT NULL DEFAULT 0,
  started_at    timestamptz,
  finished_at   timestamptz,
  created_at    timestamptz NOT NULL DEFAULT now()
);

-- Every successful extraction is an immutable snapshot
CREATE TABLE company_snapshots (
  id           bigserial PRIMARY KEY,
  tenant_id    uuid NOT NULL REFERENCES tenants(id),
  domain       text NOT NULL,
  profile      jsonb NOT NULL,
  content_hash text NOT NULL,
  captured_at  timestamptz NOT NULL DEFAULT now(),
  employee_range text GENERATED ALWAYS AS (profile->>'employeeRange') STORED,
  open_role_count int GENERATED ALWAYS AS (jsonb_array_length(profile->'openRoles')) STORED
);

CREATE INDEX ON company_snapshots (tenant_id, domain, captured_at DESC);
CREATE INDEX ON company_snapshots USING gin (profile jsonb_path_ops);

-- Row-level security: tenants only ever see their own rows
ALTER TABLE company_snapshots ENABLE ROW LEVEL SECURITY;
CREATE POLICY tenant_isolation ON company_snapshots
  USING (tenant_id = current_setting('app.tenant_id')::uuid);

The snapshot model is what turns "data" into "intelligence." Because every run creates a new row instead of overwriting the old one, you can answer the questions customers actually pay for:

-- Which companies increased open roles by 3+ since last week?
WITH ranked AS (
  SELECT domain, open_role_count, captured_at,
         LAG(open_role_count) OVER (PARTITION BY domain ORDER BY captured_at) AS prev_count
  FROM company_snapshots
  WHERE captured_at > now() - interval '14 days'
)
SELECT domain, prev_count, open_role_count, open_role_count - prev_count AS delta
FROM ranked
WHERE prev_count IS NOT NULL AND open_role_count - prev_count >= 3
ORDER BY delta DESC;

The content_hash column also lets you skip writing a new snapshot when nothing changed, which keeps storage costs sensible as you grow.

Layer 5: Monetizing with Stripe and Razorpay

Here's where many technical founders over-engineer. You do not need two separate billing systems. You need two webhook adapters feeding one entitlement table.

The internal model is deliberately simple:

CREATE TABLE entitlements (
  tenant_id        uuid PRIMARY KEY REFERENCES tenants(id),
  plan             text NOT NULL CHECK (plan IN ('starter','growth','scale')),
  status           text NOT NULL CHECK (status IN ('active','past_due','canceled')),
  domain_limit     int  NOT NULL,
  refreshes_per_month int NOT NULL,
  provider         text NOT NULL CHECK (provider IN ('stripe','razorpay')),
  provider_ref     text NOT NULL,
  current_period_end timestamptz NOT NULL,
  updated_at       timestamptz NOT NULL DEFAULT now()
);

A sensible tier structure for this kind of product might look like:

• Starter: up to 250 domains, weekly refresh, CSV export
• Growth: up to 2,500 domains, twice-weekly refresh, change alerts, API access
• Scale: custom domain limits, daily refresh, dedicated model capacity, SSO

Now the webhook handlers. Both providers sign their payloads, and you must verify signatures against the raw request body before trusting anything.

// apps/web/app/api/webhooks/stripe/route.ts
import Stripe from "stripe";
import { upsertEntitlement } from "@/lib/billing";

const stripe = new Stripe(process.env.STRIPE_SECRET_KEY!);

export async function POST(req: Request) {
  const raw = await req.text();
  const sig = req.headers.get("stripe-signature")!;

  let event: Stripe.Event;
  try {
    event = stripe.webhooks.constructEvent(raw, sig, process.env.STRIPE_WEBHOOK_SECRET!);
  } catch {
    return new Response("Invalid signature", { status: 400 });
  }

  if (event.type.startsWith("customer.subscription.")) {
    const sub = event.data.object as Stripe.Subscription;
    await upsertEntitlement({
      eventId: event.id,
      tenantId: sub.metadata.tenant_id,
      provider: "stripe",
      providerRef: sub.id,
      plan: sub.metadata.plan as "starter" | "growth" | "scale",
      status: sub.status === "active" || sub.status === "trialing" ? "active"
            : sub.status === "past_due" ? "past_due" : "canceled",
      periodEnd: new Date(sub.items.data[0].current_period_end * 1000),
    });
  }
  return new Response("ok");
}

// apps/web/app/api/webhooks/razorpay/route.ts
import crypto from "node:crypto";
import { upsertEntitlement } from "@/lib/billing";

export async function POST(req: Request) {
  const raw = await req.text();
  const signature = req.headers.get("x-razorpay-signature") ?? "";
  const expected = crypto
    .createHmac("sha256", process.env.RAZORPAY_WEBHOOK_SECRET!)
    .update(raw)
    .digest("hex");

  if (
    signature.length !== expected.length ||
    !crypto.timingSafeEqual(Buffer.from(signature), Buffer.from(expected))
  ) {
    return new Response("Invalid signature", { status: 400 });
  }

  const payload = JSON.parse(raw);
  const sub = payload.payload?.subscription?.entity;
  if (sub) {
    const statusMap: Record<string, "active" | "past_due" | "canceled"> = {
      active: "active", authenticated: "active", pending: "past_due",
      halted: "past_due", cancelled: "canceled", completed: "canceled",
    };
    await upsertEntitlement({
      eventId: req.headers.get("x-razorpay-event-id") ?? `${sub.id}:${payload.event}`,
      tenantId: sub.notes?.tenant_id,
      provider: "razorpay",
      providerRef: sub.id,
      plan: sub.notes?.plan,
      status: statusMap[sub.status] ?? "past_due",
      periodEnd: new Date(sub.current_end * 1000),
    });
  }
  return new Response("ok");
}

The upsertEntitlement function should record the eventId in a processed-events table and skip duplicates, because both providers can and will deliver the same event more than once. Field names in provider payloads also shift between API versions, so pin your API version and keep a few real webhook payloads as test fixtures.

The Next.js + Shadcn UI Dashboard

The dashboard is where customers decide whether your product is worth renewing. It doesn't need to be flashy, but it needs to be fast, clear, and honest about data freshness.

With the Next.js App Router, React Server Components let you query Postgres directly on the server, apply the tenant context, and stream the result to the browser with no client-side data fetching boilerplate.

// apps/web/app/(dashboard)/signals/page.tsx
import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
import { Badge } from "@/components/ui/badge";
import {
  Table, TableBody, TableCell, TableHead, TableHeader, TableRow,
} from "@/components/ui/table";
import { requireTenant } from "@/lib/auth";
import { withTenant } from "@/lib/db";

export default async function SignalsPage() {
  const { tenantId } = await requireTenant();

  const rows = await withTenant(tenantId, (sql) => sql`
    SELECT DISTINCT ON (domain)
      domain, profile->>'companyName' AS name, employee_range,
      open_role_count, captured_at
    FROM company_snapshots
    ORDER BY domain, captured_at DESC
    LIMIT 100
  `);

  return (
    <Card>
      <CardHeader>
        <CardTitle>Latest Account Signals</CardTitle>
      </CardHeader>
      <CardContent>
        <Table>
          <TableHeader>
            <TableRow>
              <TableHead>Company</TableHead>
              <TableHead>Size</TableHead>
              <TableHead>Open roles</TableHead>
              <TableHead>Last checked</TableHead>
            </TableRow>
          </TableHeader>
          <TableBody>
            {rows.map((r) => (
              <TableRow key={r.domain}>
                <TableCell className="font-medium">{r.name ?? r.domain}</TableCell>
                <TableCell>{r.employee_range ?? "Unknown"}</TableCell>
                <TableCell>
                  <Badge variant={r.open_role_count > 5 ? "default" : "secondary"}>
                    {r.open_role_count}
                  </Badge>
                </TableCell>
                <TableCell>{new Date(r.captured_at).toLocaleDateString()}</TableCell>
              </TableRow>
            ))}
          </TableBody>
        </Table>
      </CardContent>
    </Card>
  );
}

The withTenant helper opens a transaction, runs SET LOCAL app.tenant_id = ..., and then executes the query. Because RLS is enforced in the database, even a buggy query in the UI layer cannot leak another tenant's data. That's defense in depth you'll be very glad to have during your first security questionnaire from an enterprise buyer.

Containerizing with Docker and Deploying on AWS

Every service gets its own container. Locally, Docker Compose ties them together:

# docker-compose.yml
services:
  postgres:
    image: postgres:17
    environment:
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes: [pgdata:/var/lib/postgresql/data]

  redis:
    image: redis:7-alpine

  ollama:
    image: ollama/ollama:latest
    volumes: [ollama:/root/.ollama]
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

  n8n:
    image: n8nio/n8n:latest
    environment:
      DB_TYPE: postgresdb
      DB_POSTGRESDB_HOST: postgres
      EXECUTIONS_MODE: queue
      QUEUE_BULL_REDIS_HOST: redis
      AGENT_API_URL: http://gateway:4000
    ports: ["5678:5678"]
    depends_on: [postgres, redis]

  gateway:
    build: ./apps/gateway
    environment:
      DATABASE_URL: postgres://postgres:${POSTGRES_PASSWORD}@postgres:5432/postgres
      REDIS_URL: redis://redis:6379
    depends_on: [postgres, redis]

  agents:
    build: ./apps/agents
    environment:
      DATABASE_URL: postgres://postgres:${POSTGRES_PASSWORD}@postgres:5432/postgres
      REDIS_URL: redis://redis:6379
      OLLAMA_URL: http://ollama:11434
      WORKER_CONCURRENCY: "4"
    depends_on: [ollama, redis]

  web:
    build: ./apps/web
    ports: ["3000:3000"]
    depends_on: [postgres]

volumes:
  pgdata:
  ollama:

For production on AWS, a pragmatic layout is:

• Amazon RDS for PostgreSQL (Multi-AZ) instead of a self-managed container
• Amazon ElastiCache for Redis
• ECS on Fargate for the gateway, web app, and n8n containers, since they're CPU-bound and stateless
• ECS on EC2 GPU instances (or a small EKS node group) for Ollama and the agent workers
• Application Load Balancer with AWS WAF in front of the web app and webhook endpoints
• AWS Secrets Manager for Stripe, Razorpay, and database credentials
• Private subnets for Ollama, Postgres, and Redis, so they're never exposed to the internet

Scale the agent workers based on queue depth, not CPU. A CloudWatch metric published from BullMQ's waiting-job count is a far better autoscaling signal for this workload.

Best Practices for Production-Grade Agentic Pipelines

After building a few systems like this, these are the habits I'd treat as non-negotiable:

• Keep agents narrow. One graph per job type (company profile, pricing change, job postings) is easier to test and tune than one mega-agent that does everything.
• Validate deterministically, generate probabilistically. Let the LLM extract, but let code decide whether the extraction is trustworthy.
• Store raw inputs alongside outputs. Keep a compressed copy of the cleaned page text in object storage (S3) keyed by content hash. When a customer disputes a data point, you can show exactly what the model saw.
• Version your prompts and schemas. Store a schema_version with every snapshot so you can re-run old data through improved extractors without confusion.
• Be a polite crawler. Respect robots.txt, identify your bot with a clear user agent, cache aggressively, and rate-limit per domain.
• Design for idempotency everywhere. Jobs, webhooks, and report emails should all be safe to process twice.
• Instrument everything. Track extraction success rate, average attempts per job, tokens per second, and review-queue size. These four numbers tell you more about system health than any dashboard of CPU charts.

Common Mistakes to Avoid

These are the traps I see teams fall into most often:

• Letting the agent loop forever. Without a recursionLimit and an attempt budget, one weird page can tie up a GPU for minutes. Always cap retries.
• Trusting the model's self-assessment. Asking "Is this answer correct?" is not validation. Check extracted values against the source text and business rules.
• Putting business logic in n8n code nodes. It's tempting, but logic buried in workflow JSON is hard to test, review, and version. Move anything non-trivial into the TypeScript services.
• Parsing webhooks before verifying signatures. If you call req.json() first, you lose the raw body you need for signature checks. Read the raw text, verify, then parse.
• Sharing one database role across tenants with no RLS. One missing WHERE tenant_id = ... clause becomes a data breach. Enforce isolation in the database.
• Oversizing models. Jumping straight to the largest model makes every job slow and expensive. Start with a 7B to 14B model and escalate only on failure.
• Ignoring data-protection law. Enriching data about businesses is very different from profiling individuals. Know where GDPR, CCPA, and India's DPDP Act apply to what you collect and sell.

🚀 Pro Tips

• Use a model ladder. Route every job to a small, fast model first, and escalate to a larger model only when validation fails. In many pipelines, most pages succeed on the first, cheaper pass.
• Pre-warm Ollama. Set OLLAMA_KEEP_ALIVE so your main model stays loaded in GPU memory between jobs. Cold-loading a model on every batch quietly destroys throughput.
• Diff before you summarize. Compute structural diffs between snapshots in SQL first, then ask the LLM to explain only the changes. It's cheaper, faster, and produces far more useful "what changed" summaries.
• Expose a "confidence" field to customers. Show how many attempts a record took and whether a human reviewed it. Transparency builds trust and justifies premium tiers.
• Use LangGraph checkpointing with Postgres. The Postgres checkpointer lets interrupted jobs resume mid-graph after a deploy or a spot-instance interruption.
• Ship a sample report on your landing page. In DaaS, the report is the product demo. A real, anonymized weekly report sells better than any feature list.
• Offer INR pricing via Razorpay from day one if you're targeting Indian startups. UPI-friendly checkout can make a real difference to conversion in that market.

📌 Key Takeaways

• Split orchestration from reasoning. n8n handles schedules, triggers, and integrations; LangGraph handles stateful agent logic. They talk over clean REST APIs.
• Self-healing means extract, validate, reflect, retry, with a hard budget. Deterministic validation against source text is what makes local models reliable enough for paid data products.
• Local LLMs via Ollama are a business advantage, not just a technical one. They give you predictable costs and a strong privacy story for customers who care where their data goes.
• PostgreSQL can do almost everything. Immutable snapshots, JSONB, generated columns, and row-level security give you history, analytics, and tenant isolation in one place.
• Normalize billing providers into one entitlement model. Stripe and Razorpay become thin adapters, and the rest of your app stays provider-agnostic.
• Deploy with scaling signals that match the workload. Scale agent workers by queue depth and GPU throughput, not CPU usage.

Conclusion

The shift from fragile scrapers to agentic data pipelines isn't about adding AI for the sake of it. It's about building systems that expect the web to be messy and recover on their own when it is. A selector-based scraper breaks the moment a page changes. A well-designed LangGraph agent notices the failure, tries the careers page instead, escalates to a stronger model, and only bothers a human when it genuinely can't figure things out.

Wrap that intelligence in a disciplined architecture, with n8n for orchestration, a strict Node.js gateway, PostgreSQL as the source of truth, unified Stripe and Razorpay billing, and a clean Next.js dashboard, and you have something far more valuable than a script. You have a product that can run quietly every week, deliver fresh insight to paying customers, and keep improving as open models get better.

My suggestion: start small. Pick one narrow signal, such as hiring activity, build the full loop end to end for 100 domains, and get it in front of five real customers. Once the pipeline proves itself, adding new agents for pricing changes or tech-stack detection becomes a matter of adding new graphs, not rebuilding the platform.

Happy building.

References

• LangGraph.js Documentation
• LangChain.js Ollama Integration
• Ollama GitHub Repository and API Docs
• Ollama Structured Outputs
• n8n Documentation
• n8n Queue Mode
• BullMQ Documentation
• Fastify Documentation
• PostgreSQL Row Security Policies
• PostgreSQL JSON Types
• Stripe Webhooks Guide
• Razorpay Webhooks Documentation
• Next.js App Router Documentation
• Shadcn UI
• Docker Compose Documentation
• Amazon ECS Developer Guide