Quantization

Reducing model precision to decrease memory usage and speed up inference.

Quantization compresses models from 32-bit to 8-bit or 4-bit with minimal quality loss.

Example

A 70B model requiring 140GB at FP16 fits in 35GB at INT4.

Frequently asked questions

At 8-bit, loss is negligible. At 4-bit, acceptable for most tasks.

The process of further training a pre-trained AI model on specific data to specialize it.

A memory-efficient fine-tuning method that trains injected parameters instead of the full model.

Explore more

Book a free 30-minute strategy call. I'll show you how Quantization fits into a real growth strategy for your business.