Nous Research releases Token Superposition Training with 2-3× speedup
Nous Research released Token Superposition Training (TST), a modification to the standard LLM pretraining loop. The method processes contiguous bags of tokens by averaging their embeddings for the first third of training, then switches to conventional next-token prediction. It produces a 2-3× wall-clock speedup at matched FLOPs without changes to model architecture, optimizer, tokenizer, or training data. The announcement was shared directly from the company's account.
@itsclivetime basically better initial weights? feel like its basically a warmup run of one query/key epoch over whole dataset
this is quite remarkable
Check out our researchers' latest paper that introduces Superposition, a potential path to multiplying training speed during pre-training.
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.
Awesome. LLM mixup weirdness.
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.
this is quite remarkable
Today we release Token Superposition Training (TST), a modification to the standard LLM pretraining loop that produces a 2-3× wall-clock speedup at matched FLOPs without changing the model architecture, optimizer, tokenizer, or training data. During the first third of training, the model reads and predicts contiguous bags of tokens, averaging their embeddings on the input side and predicting the next bag with a modified cross-entropy on the output side. For the remainder of the run, it trains normally on next-token prediction. The inference-time model is identical to one produced by conventional pretraining. Validated at 270M, 600M, and 3B dense scales, and at 10B-A1B MoE. The work on TST was led by @bloc97_, @gigant_theo, and @theemozilla.
someone in the comments pointed out similarity to this prior work, which i've never heard of before:
tl;dr:
3x fewer steps iso-data, by pre-pre-training on a new objective: - segment of 8 tokens (mean pooled embeddings) ==> next segment of 8 tokens (multi-hot cross-entropy)
this is quite remarkable