If you skim the ggml source for the first time, the part that looks most foreign to a PyTorch reader is the quantization code. Functions named quantize_row_q4_K, vec_dot_q4_K_q8_K, structures called block_iq2_xxs, names like IQ4_NL and Q5_K_M — the vocabulary alone takes a while to settle in. By the end of Part III you will read all of this fluently. This chapter is the part that makes the rest learnable: the arithmetic and the design space of quantization, before any of ggml's specific choices.
We will not write a single line of ggml-specific code in this chapter. The point is to build the picture against which Chapter 7's specifics make sense. Once that picture is in place, every name in the quant zoo decomposes into a small number of axes — bits, granularity, symmetry, codebook — and the choices become legible.
At its core, quantization is a function from a real number to a discrete one. If you have a floating-point value x and you want to represent it using fewer bits, you choose a finite set of representable values {q_0, q_1, ..., q_{N-1}} and map x to the closest one. The information you lose is the difference between x and its chosen q_i. This difference is the quantization error. Everything else in this chapter — every scheme, every clever variant — is engineering around the question: how do we choose the set of representable values so that, on real model weights, the average error is small enough?
For a uniform quantizer, the representable values are evenly spaced:
q_i = scale × (i - zero_point) for i ∈ {0, 1, ..., N-1}
Where scale is the size of one step, zero_point is an integer offset, and N = 2^bits is the number of representable values. To quantize a value x:
quantize(x) = clip(round(x / scale) + zero_point, 0, N-1)
dequantize(q) = scale × (q - zero_point)
The quantize-then-dequantize round trip introduces an error bounded by scale / 2 for any single value (half a step, by triangle inequality). The total error across a tensor depends on how scale is chosen relative to the tensor's distribution.
Two specializations of this scheme show up everywhere.
A symmetric quantizer fixes zero_point = 0 (or equivalently, zero_point = N/2 for unsigned representation, with the symmetry centered there). The representable values are symmetric around zero, so q and -q are both representable. The scale alone parameterizes the quantizer; the dequantization is just multiplication. This is what ggml uses for most weight quants — the names ending in _0, like Q4_0, Q5_0, Q8_0.
An asymmetric quantizer keeps both parameters: scale and zero_point (often called minimum). It can fit a distribution that is not centered at zero — say, all-positive values, or values with a mean offset — without wasting representable codes on a region that has no data. The cost is one extra scalar of metadata per group, and a slightly more expensive dequantize. ggml's Q4_1, Q5_1 are asymmetric variants of Q4_0, Q5_0.
Beyond uniform quantizers, there are non-uniform ones — the representable values do not have to be evenly spaced. For example, you could pick the 16 values that minimize squared error over a Gaussian distribution and store those in a codebook; a 4-bit quantized weight is then an index into that codebook. This is what ggml's I-quants (IQ2_XXS, IQ3_XXS, IQ4_NL, etc.) do, and we will see them in detail in Chapter 7. The non-uniform scheme can fit a known distribution much better than uniform, at the cost of a lookup table and more complex dequantization.
Quantization works on neural networks for a reason that is partly empirical and partly structural.
The empirical part: trained transformer weights are, to a first approximation, distributed close to a Gaussian with mean zero and a smallish standard deviation. There are outliers — usually concentrated in particular rows or columns of particular tensors — but the bulk of the weights look like noise around zero. A uniform quantizer matched to the standard deviation of such a tensor produces small per-element error, and the error is roughly noise-like rather than systematically biased.
The structural part: a neural network is a sum of many products. The output of a single matmul is y_j = Σ_i W_{i,j} × x_i. The quantization error in each W_{i,j} is, in expectation, zero-mean and uncorrelated across i, so the error in y_j averages out by the central limit theorem. A 4-bit quantized weight in isolation has errors on the order of 5% per element, but the dot product of 4096 such weights has an error closer to 5% / sqrt(4096) ≈ 0.08%. This is why models with millions of weights tolerate aggressive per-weight quantization that a single-step computation could not.
The structural argument has limits. It assumes errors are uncorrelated; for some weights, they are not. It assumes errors are zero-mean; for biased quantizers, they are not. It assumes the activations x_i are not adversarially chosen against the quantizer; for some inputs they are. The K-quant and I-quant work in ggml is, in part, a response to these limits — picking quantizer parameters to keep errors uncorrelated and zero-mean even on adversarial activations.
Note: the activation problem.
Weights are stable; you compute their distribution once, at model-conversion time, and quantize accordingly. Activations — the values that flow through the model at runtime — are different. They depend on the input, they include outliers (some attention heads produce huge values for some tokens), and you cannot precompute their range. ggml mostly handles this by quantizing activations on the fly into a higher-precision quant (typically Q8_0 or Q8_K) just before each matmul, using the activation's own min/max for the current row. The dot product is then between an aggressively-quantized weight and a mildly-quantized activation, and the activation's per-call calibration absorbs most of the dynamic-range pain. We will see this dispatch in Chapter 9.
When you have one scale-and-zero-point for an entire tensor, you call that per-tensor quantization. When you have one per row, that is per-row (or per-channel, depending on which axis is the "channel"). When you have one per block of K consecutive elements, that is per-block or group-wise.
Granularity matters because the right scale depends on the values being represented. A single tensor's values may have very different magnitudes in different rows or different blocks; a single per-tensor scale has to accommodate the largest of them, which means the smallest values get represented with very few effective bits. Smaller groups → tighter scales → less wasted precision.
The cost is metadata. A weight tensor with M × N elements, quantized to 4 bits per-tensor, occupies M × N × 0.5 bytes plus one fp32 scale (4 bytes). The same tensor at per-row would have M scales — M × 4 bytes. At per-block with block size 32, you have M × N / 32 scales — M × N × 0.125 bytes, comparable to the weight bytes themselves. In ggml's K-quant formats, the scale storage cost is non-trivial and is itself heavily optimized: super-blocks of 256 elements share one fp16 scale, while sub-blocks of 16 or 32 elements share an integer scale that is itself quantized within the super-block. This nested structure is one of the most distinctive features of ggml's quants and the source of the "K" in the name.
The granularity trade-off, in one table:
| Granularity | Metadata overhead | Quality |
|---|---|---|
| Per-tensor | one scale per tensor | poor for unbalanced tensors |
| Per-row | one scale per row | good for matmul (rows = output channels) |
| Per-block 64 | one scale per 64 elem. | very good; ~6% overhead at 4-bit |
| Per-block 32 | one scale per 32 elem. | excellent; ~12% overhead at 4-bit |
| Per-element | one scale per element | "fp16 with extra steps" |
The sweet spot for inference, empirically, is per-block at 32 or 64 elements with a hierarchical scale layout. This is where ggml lives.
Once granularity is fixed, the next decision is how many bits to spend per element. The arithmetic is straightforward — B bits gives 2^B distinct codes — but the implications cascade.
8-bit (Q8_0, Q8_1, Q8_K). Almost lossless. Perplexity differences from fp16 are within measurement noise. Memory savings are 2× over fp16. This is the "safe" quant: nobody complains about quality. It is the format ggml uses internally for activations during the dot product, and is sometimes shipped as the weight format too for users who want maximum quality at minimum memory.
5-bit (Q5_0, Q5_1, Q5_K). Slightly visible degradation. Perplexity rises by a small but measurable amount over 8-bit. Memory savings are roughly 3.2×. Used when quality matters and 4-bit feels too aggressive.
4-bit (Q4_0, Q4_1, Q4_K). The sweet spot for most deployments. Perplexity rises by a measurable but acceptable amount; for chat models, output quality is usually indistinguishable from fp16 in casual evaluation. Memory savings are 4×. This is the most-shipped quant in the ecosystem.
3-bit (Q3_K, IQ3_XXS, IQ3_S, IQ3_M). The lower edge of "useful." Perplexity climbs noticeably. The I-quant variants here use codebooks rather than uniform quantization to claw back some quality. Used when memory really hurts.
2-bit (Q2_K, IQ2_XXS, IQ2_XS, IQ2_S, IQ2_M). Aggressive. Perplexity climbs a lot. The I-quants are essential at this level — uniform 2-bit quantization is unusably coarse. Used to fit larger models on smaller hardware (a 70B in 2-bit is roughly the size of a 13B in 4-bit).
1-bit and ternary (IQ1_S, IQ1_M, ternary quants). The frontier. Quality starts to matter for the chosen task. Recent work (BitNet, ternary models) shows that if you train with quantization in mind, 1-bit weights are usable; post-hoc 1-bit quantization of an unaware model is mostly a research curiosity.
The rule of thumb: every additional bit roughly halves the per-element quantization error, and on log-perplexity the improvement is approximately linear in bits down to about 3 bits, then less so. We will plot this in Chapter 7 with real numbers.
A small but important wrinkle that confuses readers on first contact: there are two independent axes here.
Symmetric vs. asymmetric is about whether the quantizer assumes the data is centered at zero. A symmetric quantizer maps zero to a representable value and uses a single scale. An asymmetric quantizer adds a zero_point offset.
Signed vs. unsigned is about how the underlying integer is stored. A signed 4-bit integer ranges from -8 to 7; an unsigned 4-bit integer ranges from 0 to 15.
These can mix. ggml's Q4_0 is symmetric and stores as offset-encoded unsigned: each 4-bit code is q_unsigned ∈ [0, 15], dequantized via value = scale × (q_unsigned - 8). The "minus 8" gives the symmetry. ggml's Q4_1 is asymmetric, also stored unsigned, with an explicit zero_point parameter as a second per-block fp16 (so each block carries a scale and a min). Some hardware-specific quants are signed; the I-quants are signed because their codebooks include both positive and negative values directly.
The choice has small but real performance consequences. Symmetric multiplication is one fewer add per element. Unsigned storage avoids sign-extension in some SIMD paths. Asymmetric is a better fit when the data really isn't centered at zero — which, for trained weights, is rarely true; weights are well-behaved. Asymmetric matters more for activations.
Tip: when you read ggml source, the suffix tells you the scheme.
_0(e.g.,Q4_0): symmetric, one fp16 scale per block._1(e.g.,Q4_1): asymmetric, fp16 scale + fp16 min per block._K(e.g.,Q4_K): K-quant — super-blocks with hierarchical scales.IQ*(e.g.,IQ2_XS): I-quant — codebook-based, with per-block scale.- Sizes like
_XS,_S,_M(e.g.,IQ2_M): the relative bits-per-weight within a family.By the end of Chapter 7 these will be second nature. For now, file them as a pattern to look for.
The most important decision in a uniform quantizer is the scale. The textbook choice — min/max calibration — sets the scale so that the largest absolute value in the tensor is at the edge of the representable range:
scale = max(|x|) / Q_max (symmetric)
Where Q_max is the largest representable code (e.g., 7 for signed 4-bit). This guarantees no clipping: every value in the tensor has a representable code. It is also pessimistic: if the maximum is an outlier — a single element 5× larger than the rest — the entire scale is set to accommodate it, and every other element pays the cost in precision.
Better calibration methods trade off clipping against quantization error. The choices, roughly in order of sophistication:
Min/max. Simplest, no clipping, dominated by outliers. Used as a fast default.
Percentile. Clip the top p% of values, set scale based on the new max. For example, percentile 99.9 ignores the largest 0.1% of values. Reduces dominance of outliers; introduces clipping for the few largest values, which may or may not matter depending on whether those values are signal or noise.
MSE-optimal. Choose the scale that minimizes the mean squared quantization error over the tensor's distribution. Solved by a small grid search or analytical expression for uniform distributions; iterative for arbitrary ones. The scale ends up smaller than min/max — sacrifices a bit of clipping for better average precision.
KL-divergence. Choose the scale that minimizes the KL divergence between the original distribution and the quantized one. This is the calibration most associated with TensorRT and similar production toolchains. Most relevant when the distribution shape matters downstream — e.g., in a softmax-fed layer.
Importance-weighted. Weight each element's quantization error by how much that element matters for the model's output. Concretely: the squared error (x_i - quant(x_i))^2 is multiplied by an "importance" w_i derived from how often x_i is exercised by realistic inputs and how much its movement perturbs the next layer. The scale is then chosen to minimize the weighted error sum. This is the basis of ggml's imatrix feature, which we will meet in detail in Chapter 8.
The cost of these methods scales: min/max needs no calibration data; percentile and MSE need a quick pass over the weight tensor; KL needs a small calibration dataset to estimate distributions; importance-weighted needs a real calibration dataset and a forward pass through the model.
Calibration is almost free relative to the cost of training the model in the first place — even importance-weighted calibration takes minutes to hours, not days — and it is the single biggest lever for quality at a given bit-width. The reason ggml's K-quants and I-quants outperform earlier per-block uniform quants is, primarily, better calibration.
Going Deeper: why MSE is not always the right objective.
Minimizing the squared error in the weights is a proxy for minimizing the error in the model's output. For most weights, the proxy is fine: a small weight perturbation produces a small output perturbation. But not all weights matter equally. A few weights — typically those in the input embedding, the output projection, and the first/last few transformer layers — have outsized influence on the final logits. Quantizing them aggressively for the sake of MSE-optimal scales over the whole model degrades quality more than the MSE number would suggest. Production quant recipes (including ggml's) handle this by leaving certain "sensitive" tensors at higher precision. We will see exactly which tensors get this treatment in Chapter 7's discussion of
Q4_K_M's "M" suffix.
If you measure the distribution of activations as they flow through a transformer, you find something striking: most activations are small, but a few are very large — sometimes 10× or 100× the typical value. These outliers tend to concentrate in specific channels, and once you know where they are, they are reproducible across inputs.
This is a problem for activation quantization in particular. A min/max quantizer of an outlier-laden distribution sets the scale based on the outlier and wastes most of the bit budget. The resulting precision on the typical values is bad enough to noticeably degrade model output. Three families of solutions have emerged in the broader research literature, and ggml borrows ideas from each.
SmoothQuant observes that outlier activations multiply weight values, and the outlier "load" can be partially shifted from activations to weights without changing the math. Specifically, if Y = X · W and X has outliers in channel i, you can scale that channel of X down by some factor s, and scale the corresponding row of W up by s, producing the same Y. After the shift, X is more uniform (easier to quantize) and W has slightly louder outliers (still easy to quantize because weights have small intrinsic range). This rebalancing makes both sides quantizable.
AWQ (Activation-aware Weight Quantization) takes the dual view: keep activations unquantized (or in fp16), but pay attention to which weights are most exercised by the most important activations. Those salient weights — typically a small fraction — are kept at higher precision; the rest are quantized aggressively. Quality improves because the precision is spent where it counts.
GPTQ is a different beast: a layer-by-layer post-training procedure that quantizes weights one column at a time, and after quantizing each column, adjusts the remaining columns to compensate for the introduced error. The adjustments are computed from the inverse Hessian of the layer's reconstruction loss. The result is a quantization that is much closer to optimal in the squared-error sense than naive round-to-nearest.
ggml's I-quants and the imatrix feature are direct descendants of these ideas. The "I" stands for importance: I-quants are calibrated using an importance matrix collected by running a calibration dataset through the unquantized model and recording, for each weight, how much its perturbation affects the output. Weights that the model uses heavily get more representable codes nearby in the codebook; weights it barely uses get coarser representation. The codebook itself is hand-tuned to fit the empirical distribution of trained transformer weights. Chapter 7 walks through what an IQ2_XS block looks like in memory; Chapter 8 walks through producing an imatrix from scratch.
The K-quants, by contrast, are still uniform per-block quantizers, but with hierarchical scales (per-super-block float scale; per-sub-block integer scale) and quant-recipe choices that protect sensitive tensors with higher-bit variants. They are, in some sense, "what you can do with classical uniform quantization if you push it hard." The I-quants are "what you can do if you abandon uniformity for a learned codebook."
Both live in the ggml repository, both ship in the same .gguf files, both are loaded by the same readers. The choice between them is a quality/latency/complexity trade-off we will quantify in Chapter 7.
Quantization replaces a real-valued tensor with discrete codes plus enough metadata to decode them. The design space has four axes: bits per element (sets the floor on per-element error), granularity (smaller groups = tighter scales = less wasted precision = more metadata cost), symmetry (asymmetric handles off-center distributions; symmetric is cheaper), and scheme (uniform spacing vs. learned codebook). Calibration chooses the scale (and zero-point, for asymmetric) by some objective: min/max, percentile, MSE, KL, or importance-weighted. Activations bring an outlier problem that pure weight quantization does not face; the modern toolchain handles it with rebalancing, salience-aware quantization, or per-call activation calibration. ggml lives at the end of this design space: per-block, hierarchical-scale, mostly symmetric, with both uniform variants (K-quants) and codebook variants (I-quants), calibrated with min/max for K-quants and an importance matrix for I-quants.
That paragraph contains every concept needed to read Chapter 7. With it in hand, the byte-by-byte walkthrough of block_q4_K is no longer a wall of acronyms; it is a particular point in a design space you already understand.
We have, in this chapter, covered:
- The arithmetic of uniform quantization and what determines its error.
- Why neural network weights tolerate aggressive quantization (the central-limit argument) and where that argument breaks (correlated and biased errors).
- The granularity axis: per-tensor, per-row, per-block, with the metadata cost of each.
- Bit-width as a continuum from 8 bits (lossless) to 1 bit (frontier), and where ggml's offerings sit on it.
- Calibration objectives, from min/max through importance-weighted.
- The activation outlier problem and the three families of solutions that influence ggml's design.
The next chapter takes every one of these axes and shows where ggml chose to land. We will walk through block_q4_0, block_q4_K_M, block_iq2_xxs byte by byte. We will see how the per-super-block, per-sub-block scale hierarchy actually gets stored. We will look at the dequantize and dot-product code that consumes those layouts and understand why they are shaped the way they are.
By the end of Chapter 7, when someone tells you they shipped their model in IQ4_NL, you will know exactly what bytes that produced and how big they are.
Note: post-training vs. quantization-aware training.
Everything in this chapter is post-training quantization: you take a trained fp32 (or fp16) model and quantize it after the fact, with no further training. The alternative is quantization-aware training (QAT), where the model is trained — or at least fine-tuned — with the quantization in the loop. QAT can produce strictly better-quality quantized models, especially at very low bit-widths, because the optimizer can route around regions where quantization error hurts. The cost is a training run. The ggml ecosystem is overwhelmingly post-training: a community of users converts published fp16 weights with no training resources of their own. Chapter 19 discusses where QAT may matter for ggml going forward, and the BitNet-style models that change the picture.
Tip: read perplexity reports critically.
The standard quality metric for quantized language models is perplexity on a held-out text — typically WikiText-2 or a similar corpus — reported as a single number. Lower is better. Take any single perplexity number with caution. The metric is sensitive to the exact tokenization, the chunk size, the dataset, and the position-encoding regime; a difference of 0.05 in perplexity on WikiText might be measurement noise rather than quality difference. The honest comparison is "the same model, same data, same chunking, with quants A and B, run side by side." We will use this discipline in the Chapter 7 quality-vs-size table.
Going Deeper: the role of the lookup table.
A non-uniform quantizer needs a lookup table at dequantization time: the integer code is an index into a small array of fp16 (or bf16) values that you load into a SIMD register and shuffle. This lookup is, on modern CPUs and GPUs, almost free — the table fits in L1 cache, and table-lookup instructions (
pshufbon x86,vqtblon NEON) execute in a single cycle. What it costs is kernel complexity: a uniform dequant is(scale × (q - zero)); a non-uniform dequant islookup[q]. The lookup version is fine for the hand-written CPU and CUDA kernels in ggml; it becomes painful in environments where you cannot easily express a table lookup, like some Vulkan compute shaders that have to emulate it with a chain of conditionals. This is part of why I-quants land on different backends at different times — the math is the same, but the kernel pattern differs.
-
Quantize a Gaussian, by hand. Generate 1000 samples from a standard Gaussian. Compute symmetric, signed 4-bit min/max quantization (
scale = max(|x|) / 7, code asround(x / scale), clipped to[-7, 7]). Compute the mean squared error between original and dequantized values. Now repeat with percentile-99.9 calibration. Compare. Then repeat with MSE-optimal calibration (try a small grid of scales around min/max and pick the best). Tabulate. -
Granularity vs. quality, on a synthetic. Generate a
[1024, 1024]tensor where each row has a different standard deviation drawn uniformly from[0.1, 10]. Quantize three ways: per-tensor 4-bit, per-row 4-bit, per-block-32 4-bit. Plot the per-row reconstruction error histogram. The pattern you see — per-tensor bad on small-magnitude rows, per-row uniform, per-block somewhere between — is exactly what you'll see on real weight tensors. -
The CLT, demonstrated. Draw two random vectors of size 4096 from a standard Gaussian. Compute their inner product
y_true. Now quantize each to 4-bit and compute the inner producty_quant. Repeat 1000 times and histogram(y_quant - y_true) / y_true. The relative error should be small even though the per-element error is around 5%. This is the central-limit argument from Section 6.2 made tangible. -
Find an outlier. Load any open-weight model in PyTorch (TinyLlama is small enough). Pick the matrix
model.layers[0].self_attn.q_proj.weight. Compute its per-row max absolute value and plot the distribution. Now do the same for activations: run the model on a few sentences and capture the input toq_proj. Compare the activation max distribution to the weight max distribution. The activations should have noticeably heavier tails. This is the activation outlier problem from Section 6.7, in your own data. -
Predict the metadata cost. For a Llama-3-8B model, compute the storage cost of per-block-32 quantization metadata, assuming one fp16 scale per block: how many bytes of scales for the entire model? Compare to the weight bytes at 4-bit. The ratio should be a few percent — the "metadata overhead" of ggml's K-quants. We will verify against a real file in Chapter 7.