Context: Quoting @hayden_prairie: "We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters. Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, @hayden_prairie: "We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters. Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, https://x.com/hayden_prairie/status/2044453231913537927/photo/1" Tweet: The dynamical system view gives very clean conditions for looped transformer to be stable @hayden_prairie: "We’ve been thinking a lot about scaling laws, wondering if there is a more effective way to scale FLOPs without increasing parameters. Turns out the answer is YES – by looping blocks of layers during training. We find that predictable scaling laws exist for layer looping, https://x.com/hayden_prairie/status/2044453231913537927/photo/1"