MIT researchers introduce ELF continuous diffusion model for text
MIT researchers introduced ELF, or Embedded Language Flows, a continuous diffusion approach for text generation that remains in embedding space until a final shared-weight network maps outputs to discrete tokens. The method extends prior continuous diffusion efforts including the 2022 Self-conditioned Embedding Diffusion paper and follows performance patterns seen in LangFlow. Code is available in a public GitHub repository.
Another day, another continuous DLM paper!
This one feels like a spiritual successor to SED (https://arxiv.org/abs/2211.04236), bringing the approach into the modern era with significant improvements -- much like LangFlow (https://arxiv.org/abs/2604.11748) did for CDCD (https://arxiv.org/abs/2211.15089).
I'm a simple man, I see a Kaiming He paper, I click. ELF: Embedded Language Flows This is very interesting, getting continuous diffusion models working for text! "Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network." @sedielem you might like this one!
This paper from yesterday also follows that pattern, with Analog Bits (https://arxiv.org/abs/2208.04202) as the main inspiration!
1/?) As promised to Sander Dieleman (@sedielem), we’re finally excited to share: Towards Closing the Autoregressive Gap in Language Modeling via Entropy-Gated Continuous Bitstream Diffusion We show that continuous diffusion can achieve very strong language modeling performance when operating directly on bitstreams, outperforming masked and uniform diffusion baselines, and essentially matching autoregressive models under our evaluation settings.
👉 New preprint on revisiting continuous-space diffusion models!
Language is discrete. Language models don’t have to be. 🧚Introducing ELF🧚♀️: Embedded Language Flows—a class of diffusion models in continuous embedding space based on continuous-time Flow Matching 🧵
I'm a simple man, I see a Kaiming He paper, I click.
ELF: Embedded Language Flows
This is very interesting, getting continuous diffusion models working for text!
"Unlike existing DLMs, ELF predominantly stays within the continuous embedding space until the final time step, where it maps to discrete tokens using a shared-weight network."
@sedielem you might like this one!
