// How to run a 70B AI model on your laptop โ by using less precise numbers
๐ก The core idea: A 70B model stores each weight as a 32-bit float (like a very precise decimal number). Quantization rounds those weights to 8-bit or 4-bit integers โ less precise, but 4-8ร smaller. A 70B FP32 model = 280GB. A 70B INT4 model = 35GB. Same model, fits on a $500 GPU. Quality drops a little. Size drops a lot.
Compare bit widths
Click a card to see how the same number 3.14159 is stored at each precision level โ and how many unique values are possible.
Pick a model size and bit width to see the memory required:
Model parameters: 7B
Bit width: INT8
The precision-size tradeoff
Every AI model weight is just a decimal number. FP32 stores it precisely (32 bits = ~7 decimal places). INT8 rounds it to one of 256 buckets. INT4 rounds to one of 16. The model "loses" some precision, but research shows 4-bit and 8-bit quantized models perform surprisingly close to full precision โ because most weights don't need extreme precision.
INT4 = 8x SMALLER THAN FP32GGUF / GPTQ / AWQ = COMMON FORMATS