Defines the key distortion measures for quantizer analysis: MSE, NMSE (= E[δ²]/E[X²]), MSRE (= E[(δ/X)²]), SQNR (= 10·log₁₀(1/NMSE) dB). Also defines auxiliary quantities α, β, γ, ρ with their algebraic identities, and the centroid condition (E[δ|X ∈ Sₖ] = 0) which simplifies β=0, α=NMSE, γ=1−NMSE.
The mathematical framework for measuring how much error a quantizer introduces. These measures are used throughout the QF8 paper to analyze and compare number formats for machine learning arithmetic.
Let X be a positive random variable with pdf f supported on [a,b] ⊂ ℝ₊. A B-bit scalar quantizer is a map Q: [a,b] → X̂ with N = 2^B reconstruction points partitioning [a,b] into cells {Sₖ} with codepoints {cₖ}.
Let δ_X = X − Q(X) be the quantization error and ε_X = δ_X/X be the relative error.
| Measure | Formula | Interpretation |
|---|---|---|
| MSE | E[δ_X²] | Mean squared absolute error |
| NMSE | E[δ_X²] / E[X²] | Normalized MSE — fraction of signal power lost to quantization |
| MSRE | E[ε_X²] = E[(δ_X/X)²] | Mean squared relative error — how large errors are relative to the value |
| SQNR | 10·log₁₀(1/NMSE) dB | Signal-to-quantization-noise ratio — higher is better; each +6 dB ≈ halving noise power |
Why NMSE and MSRE matter for ML: MSE is dominated by errors on large values and misses whether small values are accurately represented. NMSE and MSRE capture relative accuracy, which is what determines whether a quantized neural network produces the same outputs as an unquantized one.
These appear in the product error decomposition, which describes how quantization error propagates through multiplication.
| Symbol | Formula | Name |
|---|---|---|
| α_X | E[X·δ_X] / E[X²] | Signal-error correlation |
| β_X | E[Q(X)·δ_X] / E[X²] | Reconstruction-error correlation |
| γ_X | E[Q(X)²] / E[X²] | Quantized power ratio |
| ρ_X | E[X·Q(X)] / E[X²] | Signal-quantized correlation |
A quantizer satisfies the centroid condition if E[δ_X | X ∈ Sₖ] = 0 for every cell — meaning each codepoint is the conditional mean of its cell. This holds for Lloyd-Max quantizers and approximately for well-designed quantizers in the high-resolution regime (many quantization levels).
Under the centroid condition:
These simplifications are what enable the clean product error formula (1 − NMSE_prod) = (1 − NMSE_X)(1 − NMSE_Y).
For general quantizers, NMSE ≠ MSRE. However, for the log-uniform quantizer, the equalization property ensures NMSE ≈ MSRE ≈ ε²/12 for any source density — they coincide to leading order. This is a unique property of log-uniform quantization and part of why it is minimax optimal.