Demystifying Manifold Constraints in LLM Pre-training
How do weight constraints change LLM training? An interactive exploration of MACRO, RMSNorm, weight decay, and low-precision training.
Research notes · Interactive articleBlog
Notes on optimization, training dynamics, and the ideas behind my research.