23 May 2023

QLoRA enables finetuning of 65B-parameter models on a single 48GB GPU

Tim Dettmers and colleagues showed that a 65-billion-parameter language model could be finetuned on a single 48GB GPU without measurable quality loss on the benchmarks tested, by combining 4-bit quantisation with low-rank adapter training.

Learning methods Language models Deep learning Transformers and attention Foundational method Demonstrated

Background

Finetuning a large language model means taking a pretrained model and training it further on a specific task or set of instructions. For small models this was already practical. For very large ones, 33 billion parameters and above, it was not: the memory required to store the model, its gradients, and the optimiser state during training simply exceeded what a single GPU could hold. Finetuning models at that scale was infeasible without access to large clusters of high-end hardware.

One partial answer was LoRA, short for Low Rank Adapters, which inserts small trainable matrices into a frozen model and trains only those. That cuts the number of parameters being updated. But the frozen base model still had to sit in GPU memory, and at 65 billion parameters even a frozen model is too large for a single card at standard 16-bit precision.

What happened

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer introduced QLoRA on 23 May 2023. The core idea is to store the base model in 4-bit precision rather than 16-bit, dramatically shrinking its memory footprint, and then backpropagate gradients through that frozen quantised model into LoRA adapters that remain in higher precision during the update step.

Getting 4-bit quantisation to work without degrading quality required three technical contributions. The first was a new numerical format called 4-bit NormalFloat, or NF4, described as information theoretically optimal for normally distributed weights. The second was double quantisation: quantising the quantisation constants themselves to reduce the average memory footprint further. The third was paged optimisers to handle memory spikes that occur during training without crashing.

Together these changes meant a 65B parameter model could be finetuned on a single 48GB GPU without measurable quality loss on the benchmarks tested. The authors used QLoRA to finetune more than 1,000 models, covering 8 instruction datasets and model types including LLaMA and T5, at scales that would be infeasible to run with regular finetuning.

Their best models, a family they called Guanaco, were evaluated on the Vicuna benchmark. The top Guanaco model reached 99.3% of ChatGPT’s score under GPT-4 evaluation while only requiring 24 hours of finetuning on a single GPU. The paper also provides analysis of chatbot evaluation methods, finding that GPT-4 evaluations are a cheap and reasonable alternative to human evaluation, and arguing that current chatbot benchmarks may not reliably evaluate chatbot performance.

All models and code were released, including CUDA kernels for 4-bit training.

Why it mattered

Before QLoRA, finetuning models of this scale required hardware far beyond what a single researcher could access. The method made it possible to finetune more than 1,000 models across a range of scales, including 33B and 65B parameter models that would be infeasible to run with regular finetuning. The resulting Guanaco models reached 99.3% of ChatGPT's performance level under GPT-4 evaluation on the Vicuna benchmark after just 24 hours on a single GPU, though the authors themselves caution against treating that evaluation method as definitive.

Sources

QLoRA: Efficient Finetuning of Quantized LLMs. arxiv.org. Primary source
QLoRA: Efficient Finetuning of Quantized LLMs. arxiv.org. Primary source

Cite this page

AI Achievements. (2023). QLoRA enables finetuning of 65B-parameter models on a single 48GB GPU. Retrieved 2026-08-29, from https://achievements.ai/milestone/qlora-enables-finetuning-of-65b-parameter-models-on-a

@misc{achievements_qlora_enables_finetuning_of_65b_parameter_models_on_a,
  title  = {QLoRA enables finetuning of 65B-parameter models on a single 48GB GPU},
  author = {{AI Achievements}},
  year   = {2023},
  url    = {https://achievements.ai/milestone/qlora-enables-finetuning-of-65b-parameter-models-on-a}
}