Phiacta
ExplorePostDocsGuidesContributingAbout

Phiacta

The knowledge backend.

Contact Us
ExploreTinyGPT-2 QAT: QF8 Matches FP32 Validation Loss

TinyGPT-2 QAT: QF8 Matches FP32 Validation Loss

STE-based quantization-aware training on TinyGPT-2 (2 layers, d=128, 500 steps, byte-level vocab). QF8 final val loss 2.5445 vs FP32's 2.5450 (−0.02%), while FP8 E4M3 gives 2.5478 (+0.11%). Small scale — validates format viability but not scaling behavior.

ContentIssuesEditsHistoryFilesReferences3

Typed links between this entry and other entries in the knowledge graph.

No references yet.
3d99d6e6
usesMar 27
5426517c
usesMar 27
6d55b9a3
containsMar 27

Metadata

Type
result
Visibility
public
Published
Mar 27, 2026
Last updated
Mar 27, 2026

Tags

GPT-2quakefloat8quantization-aware-trainingresulttraining