LoRA — llm-kit

lora

// Low-Rank Adaptation — how to fine-tune a huge model cheaply

💡 The core idea: A 7B model has 7 billion numbers (weights) to adjust. LoRA says: "instead of changing all 7B, add a tiny side network of ~10 million weights on top. Freeze the big model; train only the tiny adapter." You get 99% of the benefit at 1% of the cost. Like adding sticky notes to a book rather than rewriting the whole book.

How LoRA works

The original model has a weight matrix W (huge). LoRA adds two small matrices A and B. Their product (A×B) is much smaller, and only A and B are trained.

W (frozen)

⬛

4096 × 4096

16.7M params

NOT trained

A (trained)

▬

4096 × 16

65K params

B (trained)

▪▪▪

16 × 4096

65K params

The "rank 16" adapter

130K total trained

vs 16.7M original

= 0.8% of params!

Why does A×B work? Research discovered that the changes needed to fine-tune a model have "low intrinsic rank" — meaning the effective change can be expressed as a product of two small matrices. This is the key insight. The rank (16 in this example) controls the expressiveness of the adapter.

What is "rank" in LoRA?

Rank controls the size (expressiveness) of the LoRA adapter. Rank 4: very few parameters, fast, works for simple style/tone changes. Rank 16: good balance for most tasks. Rank 64+: large adapter, close to full fine-tuning quality but still much cheaper. Most practitioners use rank 8-32.

RANK 4-16 FOR MOST TASKS0.1%-1% OF ORIGINAL PARAMS