Quantization

Quantization makes AI models smaller and cheaper to run by representing their numbers with fewer bits. A large language model consists of billions of parameters, each normally stored as a 16-bit number. Quantization converts these to 8-bit, 4-bit or other lower-precision formats. A model quantized from 16 to 4 bits needs roughly a quarter of the memory, fits on fewer or smaller GPUs, and can process requests faster, because moving data through memory is often the bottleneck in AI inference. The trade-off is some loss of precision, which may slightly reduce quality.

Quantization is one of the most widely used techniques for lowering the cost of running AI models, especially self-hosted open-weight models. Many model developers publish quantized versions alongside their full-precision models, and providers use quantization behind the scenes to serve models efficiently. For IT financial management, quantization can reduce the hardware needed for self-hosted AI by half or more, making it a key lever in the economics of running models in-house and on smaller devices.

Definition of Quantization

Quantization, in the context of AI, is the conversion of a model's weights, and sometimes its activations and KV cache, from high-precision numerical formats to lower-precision ones. It reduces the number of bits used to represent each value.

The term also has a broader meaning in mathematics and signal processing. In AI economics, it refers specifically to reducing model precision to lower memory use and computational cost.

Precision Levels

Models are commonly trained and served at 16-bit precision. Quantizing to 8 bits halves memory use, usually with little or no measurable loss in quality.

Quantizing to 4 bits reduces memory to about a quarter, with a small but often acceptable quality loss for many tasks. Lower precisions save more but risk greater degradation.

Newer GPUs include hardware support for low-precision formats, which allows quantized models to run faster as well as use less memory.

Methods

Post-training quantization converts an already trained model, using a small sample of data to calibrate the conversion. It is fast and widely used for open-weight models. Ready-made quantized versions of popular models are often available, so many organizations never need to quantize a model themselves.

Quantization-aware training builds lower precision into training, which can preserve quality better but requires more effort.

Quantization can be applied to the whole model or selectively, keeping the most sensitive parts at higher precision. Mixed approaches like this often recover most of the quality lost at the lowest precisions.

Importance of Quantization in IT Financial Management

For self-hosted models, GPU memory is often the main constraint on cost. Quantization lets a model fit on fewer GPUs, or on cheaper GPUs with less memory, which directly reduces infrastructure cost.

It also frees memory for serving more requests at once, raising tokens per second per GPU and lowering cost per token.

Quantization can make it practical to run capable models on laptops, phones or edge devices, shifting some AI costs away from central infrastructure altogether. That opens up uses where cloud inference would be too slow, too expensive or unsuitable for the data involved.

Quality Trade-Offs

The effect of quantization on quality varies by model, task and precision level. Some tasks, such as classification, are robust to lower precision, while others, such as complex reasoning or precise calculations, can be more sensitive.

Because of this, quantized models should be evaluated on the organization's own tasks, not only on general benchmarks published by developers. A model that loses little on a public benchmark can still struggle with an organization's particular documents or formats.

The right precision is the lowest one at which quality remains acceptable for the use, which is a business judgment as much as a technical one.

Quantization and Model Choice

Quantization interacts with model size. A larger model quantized to 4 bits can sometimes outperform a smaller model at full precision while using similar memory.

This expands the options when choosing models for self-hosting, and it should be considered alongside model distillation and smaller models.

For API users, quantization is largely invisible, but it is one of the reasons providers can lower prices over time.