Phiacta
ExplorePostDocsGuidesContributingAbout

Phiacta

The knowledge backend.

Contact Us
ExploreTinyGPT-2 QAT: QF8 Matches FP32 Validation Loss

TinyGPT-2 QAT: QF8 Matches FP32 Validation Loss

STE-based quantization-aware training on TinyGPT-2 (2 layers, d=128, 500 steps, byte-level vocab). QF8 final val loss 2.5445 vs FP32's 2.5450 (−0.02%), while FP8 E4M3 gives 2.5478 (+0.11%). Small scale — validates format viability but not scaling behavior.

ContentIssuesEditsHistoryFilesReferences3

Issues are discussions and bug reports on this entry.

No issues yet.

Metadata

Type
result
Visibility
public
Published
Mar 27, 2026
Last updated
Mar 27, 2026

Tags

GPT-2quakefloat8quantization-aware-trainingresulttraining