TinyGPT-2 QAT: QF8 Matches FP32 Validation Loss

Can a model actually train in QF8 (QuakeFloat8, an 8-bit log-domain format where multiplication is integer addition) without losing quality? This experiment validates format viability using quantization-aware training (QAT) on a small GPT-2 variant, comparing QF8 against full-precision FP32 and the industry-standard FP8 E4M3.

The +6.6 dB SQNR advantage over FP8 suggests QF8 should train at least as well, and the results confirm this: QF8 matches FP32 to within noise while FP8 shows a small but measurable penalty.

Configuration

Parameter	Value
Model	TinyGPT-2 (d_model=128, heads=4, layers=2, d_ff=512)
Vocabulary	256 (byte-level), seq_len=128
Dataset	Shakespeare (character-level)
Training	500 steps, batch=4, lr=3×10⁻⁴, cosine decay
Quantization	Block-scaled round-trip + STE (straight-through estimator), block_size=32
Device	CPU

QAT method: Forward pass quantizes weights and activations to QF8 (or FP8) and dequantizes back to FP32, simulating the round-trip error. The STE passes gradients through the non-differentiable quantization step unchanged. This is the standard QAT approach used in production systems.

Final Results

Model	Final Train Loss	Final Val Loss	Δ_val vs FP32
FP32 (baseline)	2.5388	2.5450	—
FP8 E4M3	2.5401	2.5478	+0.0028 (+0.11%)
QF8	2.5388	2.5445	−0.0005 (−0.02%)

QF8 matches FP32 validation loss to within noise (−0.02%), while FP8 E4M3 incurs a small but consistent penalty (+0.11%).

Training Loss Trajectory

Step	FP32	FP8 E4M3	QF8
1	5.3532	5.3568	5.3520
100	2.7647	2.7656	2.7645
250	2.5749	2.5753	2.5753
375	2.4425	2.4443	2.4430
500	2.5450	2.5478	2.5445

All three formats track closely throughout training. The slight rise from step 375 to 500 is expected overfitting on the small dataset.

Limitations and Open Questions

This is a small-scale validation only:

• 2 transformer layers, 128-dimensional embeddings, 500 training steps
• Demonstrates format viability, not scaling behavior
• The 0.11% FP8 penalty could widen or narrow at scale

Priority next experiments (listed as open questions in the QF8 paper):

1. GPT-2 Small (124M parameters, 300k steps) — does the advantage hold?
2. GPT-2 Medium (355M) — do any instabilities emerge?
3. Comparison against MXFP8 with per-tensor scaling
4. Gradient quantization with a u4.3 variant (wider range, lower precision) for backward passes