How to Fine-Tune and Export a Local LLM Using Unsloth, QLoRA, and Ollama

Introduction

If you've spent any time building with large language models, you've probably hit the same wall: general-purpose models like Llama, Mistral, or Qwen are impressively capable, but they're generalists. They don't know your company's support playbook, your internal API conventions, your legal team's preferred phrasing, or the specific tone your product needs. You can paper over this with prompt engineering and retrieval-augmented generation, and honestly, you should try that first. But at some point, prompting stops being enough — you need a model that has actually learned your domain.

That's where fine-tuning comes in, and until recently, it came with a brutal cost: full fine-tuning of even a 7B parameter model could require 80GB+ of VRAM, multiple high-end GPUs, and hours of setup fighting CUDA versions. For most developers and small teams, that put custom models firmly out of reach.

This is no longer true in 2026. Between QLoRA (quantized low-rank adaptation), Unsloth (a highly optimized training library), and Ollama (a dead-simple local model runtime), you can now fine-tune a specialized 7B–8B model on a single consumer GPU with 8–16GB of VRAM — and have it running as a local API endpoint by the end of the afternoon.

In this tutorial, we'll walk through the entire pipeline end-to-end:

1. Preparing a domain-specific instruction dataset
2. Loading a quantized base model and configuring QLoRA with Unsloth
3. Training the adapter weights
4. Merging and exporting the result to quantized GGUF format
5. Wrapping it in a custom Ollama Modelfile and serving it locally

By the end, you'll have a repeatable workflow you can point at any dataset — support tickets, internal documentation, code style guides, whatever — to produce a genuinely specialized model that runs entirely on your own hardware.

Why Fine-Tune Locally? Understanding the Stack

Before diving into code, it's worth understanding why these three tools work so well together, because each one solves a distinct bottleneck in the fine-tuning pipeline.

The VRAM Problem, and How QLoRA Solves It

Full fine-tuning updates every parameter in a model. For a 7B parameter model, that means storing weights, gradients, and optimizer states for all 7 billion parameters — which adds up to well over 60GB of memory even before you account for activations.

LoRA (Low-Rank Adaptation) sidesteps this by freezing the original model weights entirely and injecting small, trainable "adapter" matrices into specific layers (typically the attention projections). Instead of updating a 4096×4096 weight matrix, you train two much smaller matrices whose product approximates the update. This cuts trainable parameters from billions down to millions.

QLoRA takes this further by quantizing the frozen base model to 4-bit precision (using a format called NF4) before attaching the LoRA adapters. The base model's memory footprint shrinks by roughly 4x, while the adapters themselves are still trained in higher precision, preserving training quality. The result: you can fine-tune a 7B model on a GPU with as little as 8–10GB of VRAM, and larger models on 16–24GB cards.

Where Unsloth Fits In

QLoRA alone gets you into feasible territory, but the standard Hugging Face + PEFT + bitsandbytes stack still leaves performance on the table — slow attention kernels, unnecessary memory copies, and inefficient gradient checkpointing.

Unsloth rewrites the critical paths (attention computation, RoPE embeddings, cross-entropy loss) using custom Triton kernels and smarter memory management. In practice, this typically delivers:

• ~2x faster training compared to standard Hugging Face fine-tuning
• Up to 70% less VRAM usage for the same batch size and sequence length
• Zero accuracy degradation — it's a systems-level optimization, not an approximation

Unsloth also ships native, single-line export functions for GGUF, which is the piece that makes the "train locally, deploy locally" story actually seamless.

Why Ollama for Serving

Once you have a fine-tuned model, you need somewhere to run it. Ollama wraps llama.cpp in a friendly CLI and REST API, automatically manages model loading/unloading, and lets you define custom models — including your own fine-tunes — via a simple Modelfile syntax. It's the fastest path from "I have a GGUF file" to "I have a chat endpoint on localhost:11434."

Prerequisites

Before we start, make sure you have:

• A Linux machine (or WSL2 on Windows) with an NVIDIA GPU — 8GB VRAM minimum, 16GB+ recommended for comfortable batch sizes
• CUDA 12.1+ and up-to-date NVIDIA drivers
• Python 3.10 or 3.11
• Ollama installed locally
• Basic familiarity with Python and Hugging Face's datasets library

💡 No local GPU? A free-tier Google Colab or a rented cloud GPU (RunPod, Lambda, Vast.ai) works identically for the training step — you'll just need to download the resulting GGUF file to run it locally with Ollama afterward.

Step 1: Setting Up Your Environment

Start with a clean virtual environment to avoid dependency conflicts — Unsloth pins fairly specific versions of torch, xformers, and bitsandbytes.

python -m venv unsloth-env
source unsloth-env/bin/activate

pip install --upgrade pip

# Install Unsloth with CUDA-matched dependencies
pip install "unsloth[cu121-torch230] @ git+https://github.com/unslothai/unsloth.git"

# Supporting libraries
pip install trl peft accelerate bitsandbytes datasets

Note: Unsloth's install extras change frequently as new CUDA/PyTorch combinations are supported. Always check the official Unsloth GitHub repo for the extras tag matching your CUDA version before installing.

Verify the GPU is visible:

python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"

If that prints True and your GPU name, you're ready to go.

Step 2: Preparing Your Domain-Specific Dataset

This is the step people underinvest in, and it's the one that determines whether your fine-tune is actually useful. Garbage in, garbage out applies more strongly to fine-tuning than almost anywhere else in ML.

For this tutorial, let's say we're building a specialized assistant for a fictional company's internal DevOps runbook — it should answer questions about deployment procedures, incident response, and infrastructure conventions in the company's specific voice.

Choosing a Data Format

Most instruction-tuned base models expect a conversational format. We'll use the widely-supported ShareGPT-style / ChatML structure, since Unsloth has built-in helpers for it.

[
  {
    "conversations": [
      { "from": "human", "value": "How do we roll back a failed deployment on the payments service?" },
      { "from": "gpt", "value": "Run `deployctl rollback payments-service --to-last-stable`. This reverts to the last tagged stable release and triggers our automated smoke tests. If the rollback itself fails, page the on-call SRE via PagerDuty — do not attempt a manual kubectl rollback on payments-service, it bypasses our canary checks." }
    ]
  },
  {
    "conversations": [
      { "from": "human", "value": "What's our convention for naming feature flags?" },
      { "from": "gpt", "value": "Feature flags follow `team_service_feature` in snake_case, e.g. `payments_checkout_new_flow`. Always register new flags in the flags-registry repo before merging, and set a default `expires_at` of 90 days to force cleanup review." }
    ]
  }
]

Save this as train_data.jsonl (one JSON object per line) or as a .json array — both work with the datasets library.

Dataset Best Practices

• Aim for at least 200–500 high-quality examples for a noticeable behavioral shift; 1,000+ is better if you can manage it. Quality beats quantity here.
• Keep answers consistent in tone and structure. The model learns your patterns, including inconsistencies, so a messy dataset produces a wishy-washy model.
• Include edge cases and refusals. If there are things the assistant shouldn't answer (e.g., "I don't have access to production credentials"), include examples of that behavior explicitly.
• De-duplicate aggressively. Near-duplicate examples waste training steps and can cause overfitting to specific phrasings.
• Hold out a validation slice (10–15%) so you can actually measure whether the model is improving or just memorizing.

from datasets import load_dataset

dataset = load_dataset("json", data_files="train_data.jsonl", split="train")
dataset = dataset.train_test_split(test_size=0.1, seed=42)

train_ds = dataset["train"]
eval_ds = dataset["test"]

print(f"Training examples: {len(train_ds)}")
print(f"Eval examples: {len(eval_ds)}")

Step 3: Loading the Base Model with QLoRA via Unsloth

Now for the core of the pipeline. We'll load a quantized base model and attach LoRA adapters in a single, Unsloth-optimized flow. For this example we'll use an 8B-class instruction model as our base — swap in whichever base model fits your VRAM budget and licensing needs.

from unsloth import FastLanguageModel
import torch

max_seq_length = 2048
dtype = None  # Auto-detected: bfloat16 on Ampere+ GPUs, float16 otherwise
load_in_4bit = True  # This is what enables QLoRA

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/llama-3.1-8b-instruct-bnb-4bit",
    max_seq_length=max_seq_length,
    dtype=dtype,
    load_in_4bit=load_in_4bit,
)

Unsloth ships pre-quantized versions of popular base models (prefixed unsloth/) that download faster and avoid a separate quantization step. Using one of these is the path of least resistance.

Next, attach the LoRA adapters:

model = FastLanguageModel.get_peft_model(
    model,
    r=16,                      # LoRA rank — higher = more capacity, more VRAM
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_alpha=16,
    lora_dropout=0,             # 0 is optimized in Unsloth's fused kernels
    bias="none",
    use_gradient_checkpointing="unsloth",  # Unsloth's memory-efficient variant
    random_state=3407,
)

Understanding the Key Hyperparameters

A rank of 16 is a solid starting point for most domain-adaptation tasks. Go higher (32–64) if you have a large, diverse dataset and are seeing underfitting; keep it lower if you're working with a small dataset and want to avoid overfitting.

Step 4: Formatting Prompts and Training

Unsloth includes a helper to convert ShareGPT-style conversations into the correct chat template for your chosen base model automatically.

from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
    tokenizer,
    chat_template="llama-3.1",
)

def formatting_prompts_func(examples):
    convos = examples["conversations"]
    texts = [
        tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=False)
        for convo in convos
    ]
    return {"text": texts}

train_ds = train_ds.map(formatting_prompts_func, batched=True)
eval_ds = eval_ds.map(formatting_prompts_func, batched=True)

Now configure the trainer using trl's SFTTrainer, which Unsloth patches for compatibility:

from trl import SFTTrainer
from transformers import TrainingArguments

trainer = SFTTrainer(
    model=model,
    tokenizer=tokenizer,
    train_dataset=train_ds,
    eval_dataset=eval_ds,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    dataset_num_proc=2,
    packing=False,  # True can speed up short-sequence datasets
    args=TrainingArguments(
        per_device_train_batch_size=2,
        gradient_accumulation_steps=4,
        warmup_steps=10,
        num_train_epochs=3,
        learning_rate=2e-4,
        fp16=not torch.cuda.is_bf16_supported(),
        bf16=torch.cuda.is_bf16_supported(),
        logging_steps=10,
        evaluation_strategy="steps",
        eval_steps=25,
        optim="adamw_8bit",
        weight_decay=0.01,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="outputs",
        report_to="none",
    ),
)

trainer_stats = trainer.train()

A few notes on this configuration:

• per_device_train_batch_size=2 with gradient_accumulation_steps=4 gives you an effective batch size of 8 while keeping memory low — tune these based on your VRAM headroom.
• optim="adamw_8bit" uses a quantized optimizer state, which meaningfully reduces memory overhead versus standard AdamW.
• 3 epochs is a reasonable starting point for a few hundred examples. Watch your eval loss — if it starts climbing while train loss keeps dropping, you're overfitting and should stop earlier or reduce epochs.

On an 8–12GB consumer GPU, fine-tuning an 8B model on a few hundred examples typically takes somewhere between 15 and 45 minutes, depending on sequence length and batch size.

Step 5: Merging Adapters and Exporting to GGUF

This is where Unsloth really shines for the "local deployment" use case. Instead of juggling separate merge and quantization scripts, it exposes a single function that merges the LoRA adapters into the base model and converts the result to GGUF in one step.

# Save merged model in 16-bit precision first (optional intermediate step)
model.save_pretrained_merged(
    "runbook-assistant-merged",
    tokenizer,
    save_method="merged_16bit",
)

# Export directly to quantized GGUF for Ollama/llama.cpp
model.save_pretrained_gguf(
    "runbook-assistant-gguf",
    tokenizer,
    quantization_method="q4_k_m",
)

Choosing a Quantization Method

GGUF supports multiple quantization schemes, each trading off size against fidelity:

• q4_k_m — The sweet spot for most use cases. Roughly a 4x size reduction from FP16 with minimal perceptible quality loss. Good default.
• q5_k_m — Slightly larger, slightly higher fidelity. Worth it if you have the disk/VRAM budget and are seeing quality issues at q4.
• q8_0 — Near-lossless but roughly double the size of q4_k_m. Use when quality is critical and you have the resources.
• f16 — No quantization, full 16-bit precision. Useful as a quality baseline, but defeats the purpose of local, low-resource serving.

For a domain-specific assistant answering factual, structured questions (like our DevOps runbook example), q4_k_m is almost always sufficient — the compression artifacts show up more in creative writing tasks than in retrieval-style factual responses.

After this step, you'll have a .gguf file (typically named something like unsloth.Q4_K_M.gguf) sitting in your output directory, ready to hand off to Ollama.

Step 6: Serving Your Model with a Custom Ollama Modelfile

Ollama models are defined by a Modelfile — a small text file that points to your GGUF weights and configures the system prompt, chat template, and generation parameters.

Create a file named Modelfile in the same directory as your GGUF file:

FROM ./runbook-assistant-gguf/unsloth.Q4_K_M.gguf

TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|>

{{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|>

{{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|>

"""

SYSTEM """You are the internal DevOps assistant for the Engineering team. Answer questions about deployment procedures, incident response, and infrastructure conventions precisely and concisely, following established runbook practices. If you are unsure or the question falls outside documented procedures, say so and recommend escalating to the on-call engineer."""

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 4096
PARAMETER stop "<|eot_id|>"

A few things worth calling out:

• The TEMPLATE block must match the chat template your base model actually expects (Llama 3.1's format is shown here) — mismatches here are a common source of garbled or repetitive output.
• temperature 0.3 keeps responses focused and deterministic, appropriate for a factual/procedural assistant. Bump it up (0.7–0.9) for more creative use cases.
• num_ctx should match or stay within the max_seq_length you trained with.

Now build and run the model:

ollama create runbook-assistant -f Modelfile

ollama run runbook-assistant

You can now chat with it directly in the terminal, or hit it via the local REST API:

curl http://localhost:11434/api/chat -d '{
  "model": "runbook-assistant",
  "messages": [
    { "role": "user", "content": "How do we roll back a failed deployment on the payments service?" }
  ],
  "stream": false
}'

If your training data was solid, you should see the model responding with the specific, structured, in-house procedures you trained it on — not a generic "here's how rollbacks generally work" answer.

Real-World Example: From Support Tickets to a Specialized Assistant

To make this concrete, consider a SaaS company that fine-tunes a model on two years of resolved customer support tickets, formatted as question/resolution pairs. After running through this exact pipeline:

• The base model (a general-purpose 8B instruct model) answered common product questions with plausible-sounding but often generic or slightly incorrect guidance.
• The fine-tuned version, trained on ~800 curated ticket resolutions, reproduced the company's actual troubleshooting steps, referenced correct internal terminology (feature names, error codes), and matched the support team's tone.
• Running as a q4_k_m GGUF model via Ollama, it served responses in under a second on a mid-range GPU, with no external API calls, no per-token costs, and no customer data leaving the local network.

This pattern generalizes well beyond support: legal teams fine-tune on contract templates and past redlines, engineering teams fine-tune on internal API documentation and code review comments, and content teams fine-tune on brand style guides. The pipeline doesn't change — only the dataset does.

🚀 Pro Tips

• Start with a rank-16 LoRA and short runs. Don't jump straight to rank-64 and 10 epochs. Train fast, evaluate, and only scale up capacity or training length if you're clearly underfitting.
• Use Unsloth's unsloth/ pre-quantized checkpoints where available — they're specifically optimized for this pipeline and avoid redundant quantization steps.
• Log a handful of held-out prompts before and after training and eyeball the outputs side by side. Loss curves are useful, but nothing beats manually reading real generations.
• Version your Modelfiles alongside your training scripts. Treat the system prompt and generation parameters as part of the model artifact, not an afterthought — they materially affect behavior.
• Test multiple GGUF quantization levels if disk space allows. Keep q4_k_m and q5_k_m side by side and A/B test them on your actual eval set before deciding which to ship.
• Use ollama show <model> --modelfile after creation to confirm your parameters actually took effect — silent misconfigurations here are easy to miss.

Common Mistakes to Avoid

• Training on too few or too repetitive examples. Fewer than 100 examples, or examples that are near-duplicates of each other, tend to produce a model that memorizes phrasing rather than learning a general pattern.
• Skipping the chat template step. Feeding raw, unformatted text into the trainer instead of applying the correct chat template is one of the most common reasons a fine-tuned model produces incoherent or run-on outputs.
• Ignoring eval loss entirely. It's tempting to just run for a fixed number of epochs and call it done. Without an eval split, you have no signal for when the model starts overfitting.
• Mismatching the Modelfile template with the training template. If you fine-tune using the Llama 3.1 chat format but define an Alpaca-style template in your Modelfile, expect broken, repetitive, or nonsensical generations — the model was never trained on that format.
• Over-quantizing for the task. Aggressively quantizing (e.g., Q2 or Q3) a model meant for precise, factual output can noticeably degrade accuracy. Reserve the most aggressive quantization levels for cases where raw inference speed matters more than precision.
• Forgetting to freeze the base model correctly. If load_in_4bit isn't set before attaching LoRA adapters, you lose most of the VRAM savings QLoRA is supposed to provide.

Best Practices Checklist

• ✅ Curate a clean, consistent, deduplicated dataset of at least a few hundred examples
• ✅ Hold out a validation split and actually monitor eval loss during training
• ✅ Start with modest LoRA rank (8–16) and scale up only if needed
• ✅ Match your Ollama Modelfile's chat template exactly to your training template
• ✅ Benchmark at least two GGUF quantization levels before choosing one for production
• ✅ Keep a small, fixed set of test prompts to manually sanity-check every new model version
• ✅ Store your dataset, training config, and Modelfile together as a single versioned artifact

Conclusion

The gap between "playing with an open-source LLM" and "running a genuinely specialized model tailored to your domain" used to require serious infrastructure. In 2026, that gap has essentially closed for anyone with a decent consumer GPU. QLoRA makes fine-tuning memory-feasible, Unsloth makes it fast and painless, and Ollama makes the resulting model instantly usable — no cloud dependency, no per-token billing, no data leaving your machine.

The workflow you've walked through here — dataset formatting, QLoRA training, GGUF export, Ollama serving — isn't a toy exercise. It's the same pattern used to ship real, production-facing internal tools. The dataset is the only thing that changes between "a fun weekend project" and "a model your team actually relies on," so if you take one thing away from this guide, let it be this: invest the bulk of your effort in curating clean, representative training examples. The pipeline will do the rest.

From here, natural next steps include experimenting with larger base models if your VRAM allows, exploring DPO (Direct Preference Optimization) on top of your SFT adapter for finer behavioral control, or wiring your new Ollama model into a lightweight RAG layer for cases where the domain knowledge changes faster than you want to retrain.

References

• Unsloth Official Documentation
• Unsloth GitHub Repository
• QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers et al.)
• LoRA: Low-Rank Adaptation of Large Language Models (Hu et al.)
• Ollama Official Documentation
• Hugging Face TRL (Transformer Reinforcement Learning) Library
• GGUF Format Specification (llama.cpp)

📌 Key Takeaways

• QLoRA combined with Unsloth makes it realistic to fine-tune 7B–8B parameter models on a single consumer GPU with as little as 8–16GB of VRAM.
• Dataset quality — clean, consistent, deduplicated examples with a proper validation split — matters more than any hyperparameter choice.
• Merging LoRA adapters and exporting straight to GGUF via Unsloth removes the need for separate, error-prone conversion tooling.
• A correctly configured Ollama Modelfile, with a chat template that matches your training format, is what turns a raw GGUF file into a reliable, locally served assistant.