Fine-Tuning 27B–32B LLMs on a Single 24GB GPU: Unsloth and 4-Bit QLoRA Production Guide

Fine-tuning 27B to 32B open-weight large language models on a single 24GB GPU is now achievable in production using Unsloth and 4-bit Quantized Low-Rank Adaptation (QLoRA). While 8B models fit easily on consumer hardware, they frequently struggle with nuanced domain reasoning in legal analysis, financial extraction, and cybersecurity threat classification. Conversely, 70B parameter models require expensive multi-GPU distributed clusters. The trade-off is laid out in Qwen 2.5-32B vs Llama 3.3-70B sizing. Architectures in the 27B–32B tier—such as Google's Gemma 2 27B and Alibaba's Qwen 2.5 32B—occupy the sweet spot for enterprise reasoning density. Standard 16-bit fine-tuning of these models demands upwards of 120GB of VRAM, but custom Triton kernels, manual backpropagation rewrites, and 4-bit NormalFloat (NF4) quantization compress the entire training pipeline into 20.8GB of VRAM.

This guide provides an end-to-end implementation blueprint using Qwen/Qwen2.5-32B-Instruct as the reference architecture, covering dataset tokenization, response-only loss masking, hyperparameter tuning, and export paths for vLLM and Ollama.

Hardware Profile and Memory Allocation

Fitting a 32B parameter model into a 24GB envelope requires strict VRAM budget management. Base parameters quantized in 4-bit NF4 consume approximately 14.8GB including quantization constants. Confining gradient updates to low-rank adapter matrices ($r=16$) keeps optimizer memory under 1GB, while activation checkpointing bounds memory spikes during forward passes. The general arithmetic is in VRAM math across quantization formats.

Dataset Structuring and ChatML Alignment

Domain adaptation requires strict adherence to the tokenizer's chat template. Qwen architectures utilize ChatML syntax with explicit <|im_start|> and <|im_end|> delimiters.

[
  {
    "messages": [
      {
        "role": "system",
        "content": "You are an enterprise cybersecurity analyst specializing in CVE classification and MITRE ATT&CK mapping."
      },
      {
        "role": "user",
        "content": "Analyze the following vulnerability: CVE-2024-3094. Provide the affected component, attack vector, and mitigation."
      },
      {
        "role": "assistant",
        "content": "### Vulnerability Analysis: CVE-2024-3094\n- **Component:** XZ Utils (liblzma), versions 5.6.0 and 5.6.1.\n- **Vector:** Malicious backdoor injected via M4 macros during tarball compilation.\n- **MITRE ATT&CK:** T1195.002 (Supply Chain Compromise).\n- **Mitigation:** Downgrade to XZ Utils 5.4.x stable releases immediately."
      }
    ]
  }
]

To prevent catastrophic forgetting and maintain conversational flow, training must compute gradients strictly over assistant completions. Unsloth provides prompt-loss masking that maps all system and user prompt tokens to the standard ignore index (-100).

from unsloth.chat_templates import get_chat_template

tokenizer = get_chat_template(
 tokenizer,
 chat_template="qwen-2.5",
 mapping={"role": "role", "content": "content", "user": "user", "assistant": "assistant"},
)

dataset = dataset.map(
 lambda x: {"text": [tokenizer.apply_chat_template(c, tokenize=False, add_generation_prompt=False) for c in x["messages"]]},
 batched=True,
)

Step-by-Step QLoRA Implementation Blueprint

The training pipeline initializes Qwen/Qwen2.5-32B-Instruct in 4-bit precision, attaches trainable LoRA matrices across all seven core linear projections, and runs supervised fine-tuning via Hugging Face's SFTTrainer.

import torch
from unsloth import FastLanguageModel, is_bfloat16_supported
from unsloth.chat_templates import train_on_responses_only
from trl import SFTConfig, SFTTrainer

# 1. Model Initialization in 4-bit NF4
max_seq_length = 2048
model, tokenizer = FastLanguageModel.from_pretrained(
 model_name="Qwen/Qwen2.5-32B-Instruct",
 max_seq_length=max_seq_length,
 dtype=None,
 load_in_4bit=True,
)

# 2. Attach PEFT LoRA Adapters across all 7 linear projections
model = FastLanguageModel.get_peft_model(
 model,
 r=16,
 target_modules=[
 "q_proj", "k_proj", "v_proj", "o_proj",
 "gate_proj", "up_proj", "down_proj"
 ],
 lora_alpha=32,
 lora_dropout=0,
 bias="none",
 use_gradient_checkpointing="unsloth",
 random_state=3407,
)

# 3. Configure Supervised Fine-Tuning Loop
trainer = SFTTrainer(
 model=model,
 tokenizer=tokenizer,
 train_dataset=dataset,
 args=SFTConfig(
 dataset_text_field="text",
 max_seq_length=max_seq_length,
 dataset_num_proc=2,
 packing=False,
 per_device_train_batch_size=1,
 gradient_accumulation_steps=8,
 warmup_steps=0.05, # a float below 1 is read as a ratio of total steps
 num_train_epochs=3,
 learning_rate=2e-4,
 fp16=not is_bfloat16_supported(),
 bf16=is_bfloat16_supported(),
 logging_steps=1,
 optim="paged_adamw_8bit",
 weight_decay=0.01,
 lr_scheduler_type="cosine",
 seed=3407,
 output_dir="outputs_qwen_32b_lora",
 report_to="none",
 ),
)

# 4. Enforce Response-Only Loss Masking
trainer = train_on_responses_only(
 trainer,
 instruction_part="<|im_start|>user\n",
 response_part="<|im_start|>assistant\n",
)

trainer_stats = trainer.train()

Hyperparameter Matrix and Stability Boundaries

Adapting attention and feed-forward projection layers simultaneously allows the network to learn domain vocabularies and syntax without destabilizing baseline knowledge.

Context Window Limits and Diagnostics

Running a 32B model inside 24GB VRAM imposes strict context window trade-offs. While 8B models can process sequences exceeding 32,000 tokens on a single GPU, 27B–32B models are practically constrained to sequence lengths between 2,048 and 4,096 tokens due to activation tensor scaling. Inference has a similar limit; see why long context costs more VRAM than weights.

• Out of Memory (OOM) Errors: Lower max_seq_length to 2048, verify use_gradient_checkpointing="unsloth", keep per_device_train_batch_size=1, and increase gradient_accumulation_steps to achieve target effective batch sizes.
• Loss Collapsing Below 0.35: Signals dataset leakage or repetitive template boilerplate. Confirm train_on_responses_only is active so static system instructions are excluded from backpropagation.
• Loss Spikes to NaN: Caused by numerical underflow in mixed-precision FP16 on modern architectures. Enforce bf16=True on Ampere/Ada/Hopper GPUs and reduce learning rate to 1e-4.
• Healthy Loss Convergence: Training loss should start around 2.5–3.2 and steadily plateau between 0.75 and 1.10 over 3 epochs on a 2,000–5,000 sample domain dataset.

Adapter Merging and Production Serving

Trained LoRA adapters can be merged directly into full precision weights for enterprise API clusters running vLLM, or converted to GGUF format for local deployment in Ollama. Engine choice is covered in SGLang vs vLLM vs TensorRT-LLM.

High-Throughput Serving with vLLM

Merge adapter weights into standard 16-bit floating-point weights:

model.save_pretrained_merged(
 "qwen-32b-specialist-16bit",
 tokenizer,
 save_method="merged_16bit",
)

Launch the high-throughput vLLM inference server:

vllm serve ./qwen-32b-specialist-16bit \
 --host 0.0.0.0 \
 --port 8000 \
 --gpu-memory-utilization 0.92 \
 --max-model-len 4096 \
 --dtype bfloat16

Quantized Local Deployment with Ollama

Export the fine-tuned model directly into an optimized GGUF format:

model.save_pretrained_gguf(
 "qwen-32b-specialist-gguf",
 tokenizer,
 quantization_method="q4_k_m",
)

Create a local Modelfile:

FROM ./qwen-32b-specialist-gguf/unsloth.Q4_K_M.gguf

TEMPLATE """<|im_start|>system
{{ .System }}<|im_end|>
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

PARAMETER stop "<|im_start|>"
PARAMETER stop "<|im_end|>"
PARAMETER temperature 0.2
PARAMETER top_p 0.9

Initialize and execute the model inside the local runtime:

ollama create qwen32b-specialist -f Modelfile
ollama run qwen32b-specialist "Analyze CVE-2024-3094 attack vectors."

By pairing 4-bit base parameter quantization with memory-optimized Triton kernels, engineering teams can fine-tune high-reasoning 27B–32B parameter models on accessible 24GB GPUs without operating multi-node compute clusters. For serving the result, see the 24GB hardware blueprint for local 27B models.

Related reading

• The RTX 5090 (32GB) Local LLM Deployment Blueprint: NVFP4 vs. FP8, SGLang vs. vLLM, and Context Limits
• Open-Weight Coding at the Frontier: Evaluating Qwen2.5-Coder-32B for Enterprise Software Engineering