Starting April 29, 2025, Gemini 1.5 Pro and Gemini 1.5 Flash models are not available in projects that have no prior usage of these models, including new projects. For details, see Model versions and lifecycle .

LoRA and QLoRA recommendations for LLMs

This page gives you configuration recommendations for tuning large language models (LLM) on Vertex AI by using Low-Rank Adaptation of Large Language Models (LoRA) and its more memory-efficient version, QLoRA .

Tuning recommendations

The following table summarizes our recommendations for tuning LLMs by using LoRA or QLoRA:

Specification

Recommended

Details

GPU memory efficiency

QLoRA

QLoRA has about 75% smaller peak GPU memory usage compared to LoRA.

Speed

LoRA

LoRA is about 66% faster than QLoRA in terms of tuning speed.

Cost efficiency

LoRA

While both methods are relatively inexpensive, LoRA is up to 40% less expensive than QLoRA.

Higher max sequence length

QLoRA

Higher max sequence length increases GPU memory consumption. QLoRA uses less GPU memory so it can support higher max sequence lengths.

Accuracy improvement

Same

Both methods offer similar accuracy improvements.

Higher batch size

QLoRA

QLoRA supports much higher batch sizes. For example, the following are batch size recommendations for tuning openLLaMA-7B on the following GPUs:

1 x A100 40G:
- LoRA: Batch size of 2 is recommended.
- QLoRA: Batch size of 24 is recommended.
1 x L4:
- LoRA: Batch size of 1 fails with an out of memory error (OOM).
- QLoRA: Batch size of 12 is recommended.
1 x V100:
- LoRA: Batch size of 1 fails with an out of memory error (OOM).
- QLoRA: Batch size of 8 is recommended.

LoRA and QLoRA recommendations for LLMs Stay organized with collections Save and categorize content based on your preferences.

Tuning recommendations

LoRA and QLoRA recommendations for LLMs