@VictorTaelin Not true, weight decay explicitly forces sparseification of parameters during training.

Source

• URL: https://x.com/o1lo01ol1o/status/2078125981114433630
• Author: o1lo01ol1o (@o1lo01ol1o)
• Posted: 2026-07-17 14:33:38

Branch

1/ @o1lo01ol1o

@VictorTaelin

Not true, weight decay explicitly forces sparseification of parameters during training. This for forces the “grokking effect”

Related

• Spine: 2026-07-17-read-this-and-your-next-model-will-be-10x-smarter