Quantization

Why a 140 GB model can shrink to 35 GB and still answer well

Quantization stores each model weight in fewer bits. A 70-billion-weight model in 16-bit floats (2 bytes each) takes about 140 GB. At 4 bits per weight it takes about 35 GB.

You divide a group of weights by one scale s and round each result to the nearest integer the format can hold: −127 to 127 for 8-bit, −8 to 7 for 4-bit. When it runs, the model uses s · q in place of w.

What you lose is the rounding error, the gap between w and s · q. The fewer bits you use, the coarser the grid of values and the bigger that gap.

Quantization

How Ollama runs an 8-billion-parameter model on a laptop

Ollama and llama.cpp run open models like Llama on your own computer, usually from model files quantized to about 4 bits. An 8-billion-weight model shrinks from 16 GB to around 5 GB, small enough for a laptop.

The file names show it: a llama.cpp model file tagged Q4 or Q8 holds weights at about 4 or 8 bits. Apple has said the model built into its devices for Apple Intelligence averages under 4 bits per weight, so it fits in a phone's memory.

Quantization

Smaller weights mean less memory to read for every word

It also speeds things up. To write each token the model has to read every weight from memory, and that reading, not the arithmetic, is usually the bottleneck. A quarter of the bytes means far less waiting.

Quality holds up because each neuron sums thousands of weight × input products. The tiny rounding errors point in random directions and partly cancel, so the sum barely moves.

Outliers are the danger: one weight of 10 among weights near 0.1 stretches the scale until the small ones all round to zero. So each group of 32 to 128 weights gets its own scale, and rare outliers can stay at higher precision.