Thinking Machines introduces Interaction Models for real-time collaboration
Thinking Machines introduced Interaction Models, an AI system that engages users through simultaneous real-time talking, listening, watching, thinking, and collaborating. The company posted its technical approach, early experimental results, and a video demonstration on its blog at thinkingmachines.ai/blog/interaction-models. The demonstration highlights live visual generation and multi-stream exchanges. The work addresses a shift in AI bottlenecks from raw compute or intelligence toward human-AI bandwidth, positioning the models as full-duplex multimodal systems that enable continuous adaptation without turn-based constraints.
Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence.
We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to intelligence/autonomy because they're harder to eval.
In the future, we think every AI system will have something like an interaction model as the outer user-facing layer, continually keeping the user informed and learning what they actually want.
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Seeing the demos come together over the last week has been awesome -- so many things that previously required a special-purpose model (e.g. real-time translation, event detection in video) turn out to be zero-shot instruction following once you have a general-purpose model with the right type signature -- continuous/simultaneous audio+video+text->audio+text
Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence. We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to intelligence/autonomy because they're harder to eval. In the future, we think every AI system will have something like an interaction model as the outer user-facing layer, continually keeping the user informed and learning what they actually want.
Thinky's secret plan:
1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world
We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
In the past few months, we had a lot of fun (and stress 😅) to produce 12 versions (+ many subversions) and 137 pages in our training run log book.
Turns out human-human collaboration is important to improving human-AI collaboration. 😊
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
@soumithchintala congrats Soumith!!
Thinky's secret plan: 1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
Very cool announcement from Thinky!
The model looks nice (they go into some reasonable amount of detail), and reading some parts of the blog you can definitely see that the infea guys had a lot of fun there!
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
@rown @liliyu_lili @saurabh_garg67 @AndreaMadotto Yes but 40 though 🤔
Our interaction model is the first general video+speech model that's visually proactive. It was super fun working on this with @liliyu_lili / @saurabh_garg67 / @AndreaMadotto and others - after countless versions it was amazing when visual interruptions suddenly worked!
@cHHillee Wait didn't you draw a picture almost like this for a blogpost sometime over a year ago?
In modern ML accelerators, FLOPS have absolutely exploded. Often though, the bottleneck is not FLOPS but memory bandwidth. Similarly, model intelligence has exploded, causing the bottleneck to be human<->AI bandwidth. At Thinky, we think that it’s important to solve this. 1/4
This is the demo that hits me as being genuinely different -- both model and user talking at once! Great stuff.
Congrats on the release @thinkymachines
With the model's simultaneous speech capability, Horace has gotten a lot easier to work with recently.
@natolambert @thinkymachines Yeah I think concurrency is actually a pretty fundamental property of human interaction! Model talking/listening + user talking, model talking/watching + user doing something on screen (e.g. sports commentary).
This is the demo that hits me as being genuinely different -- both model and user talking at once! Great stuff. Congrats on the release @thinkymachines
To stretch this analogy further, when accelerators (humans) are severely limited by bandwidth you have no choice but to move everything into SRAM (make everything fully autonomous). However, this prohibits say, keeping kv-cache in DRAM (having humans contribute). 2/4
In modern ML accelerators, FLOPS have absolutely exploded. Often though, the bottleneck is not FLOPS but memory bandwidth. Similarly, model intelligence has exploded, causing the bottleneck to be human<->AI bandwidth. At Thinky, we think that it’s important to solve this. 1/4
In modern ML accelerators, FLOPS have absolutely exploded. Often though, the bottleneck is not FLOPS but memory bandwidth. Similarly, model intelligence has exploded, causing the bottleneck to be human<->AI bandwidth. At Thinky, we think that it’s important to solve this. 1/4
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
@giffmana
@cHHillee Wait didn't you draw a picture almost like this for a blogpost sometime over a year ago?
@johnschulman2 Congrats!
Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence. We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to intelligence/autonomy because they're harder to eval. In the future, we think every AI system will have something like an interaction model as the outer user-facing layer, continually keeping the user informed and learning what they actually want.
Haven’t tried this but it seems very neat…
Yet all of the demos (except maybe one) are the model being fun and/or annoying by correcting or reminding in real time. There are obvious uses for this sort of model in meetings, education, training, etc. Why not demo valuable cases?
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Great series of demos overall, but why are we spoiling a movie lol?
The Babel fish is here
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
We are so back!
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Our interaction model is the first general video+speech model that's visually proactive. It was super fun working on this with @liliyu_lili / @saurabh_garg67 / @AndreaMadotto and others - after countless versions it was amazing when visual interruptions suddenly worked!
We’re interested in AI systems that can collaborate in real time, without relying only on artificial turn boundaries. For audio, this feels natural: listen, speak, interrupt, update. For video, we think an important version of this is visual proactivity — models that respond when something happens visually: “Tell me when I start slouching.” “Count my pushups.” “Say stop when the person stops doing X.”
@liliyu_lili @saurabh_garg67 @AndreaMadotto If you're interested in working on realtime video+speech specifically, or human AI collaboration more generally, please reach out!
Our interaction model is the first general video+speech model that's visually proactive. It was super fun working on this with @liliyu_lili / @saurabh_garg67 / @AndreaMadotto and others - after countless versions it was amazing when visual interruptions suddenly worked!
super cool interleaving!!!
While Lilian is telling a story, the interaction model can track when she is thinking, yielding, self-correcting, or inviting a response; there is no specific built dialogue management system.
@lilianweng so cool
In the past few months, we had a lot of fun (and stress 😅) to produce 12 versions (+ many subversions) and 137 pages in our training run log book. Turns out human-human collaboration is important to improving human-AI collaboration. 😊
@soumithchintala That is so awesome. Can't wait to try it.
Thinky's secret plan: 1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
So many people were bearish on Thinky Machines... they were just quietly building!
Really cool work, TML continues to be one of my favorite neolabs!
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
almost exactly two years ago lol
Making a Thinky compilation thread for today's announcement.
Making a Thinky compilation thread for today's announcement.
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence. We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to intelligence/autonomy because they're harder to eval. In the future, we think every AI system will have something like an interaction model as the outer user-facing layer, continually keeping the user informed and learning what they actually want.
In the past few months, we had a lot of fun (and stress 😅) to produce 12 versions (+ many subversions) and 137 pages in our training run log book. Turns out human-human collaboration is important to improving human-AI collaboration. 😊
Thinky's secret plan: 1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
We started Thinking Machines to advance human-AI collaboration, and this is our first bet on what that looks like. Most labs treat autonomy as the goal and interactivity as scaffolding around a turn-based core. We think the way we work with AI matters as much as how smart it is. Interactivity has to be in the model, and it has to scale with intelligence rather than trail behind it. https://thinkingmachines.ai/blog/interaction-models/
We’re interested in AI systems that can collaborate in real time, without relying only on artificial turn boundaries. For audio, this feels natural: listen, speak, interrupt, update. For video, we think an important version of this is visual proactivity — models that respond when something happens visually: “Tell me when I start slouching.” “Count my pushups.” “Say stop when the person stops doing X.”
In modern ML accelerators, FLOPS have absolutely exploded. Often though, the bottleneck is not FLOPS but memory bandwidth. Similarly, model intelligence has exploded, causing the bottleneck to be human<->AI bandwidth. At Thinky, we think that it’s important to solve this. 1/4
We strongly believe in the bitter lesson. Making interactivity an integral part of the model will outpace any harness-based approach and will lead to a better experience working with AI models. This is a step in that direction.
@soumithchintala Dude.
You just posted your secret plan!!
Thinky's secret plan: 1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
@soumithchintala please. my commute just got longer. fix it
Thinky's secret plan: 1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
@soumithchintala Great to see a compelling vision articulated. @humansand is pushing for a similar goal. I'm glad that humans will have a role in the future :)
Thinky's secret plan: 1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
@johnschulman2 super cool - videos on the blog are neat.
Sharing our work on full-duplex multimodal models -- real-time interaction that's natural and intuitive without compromising on intelligence. We started Thinky in part to differentially advance capabilities for human-AI collaboration, which are underemphasized relative to intelligence/autonomy because they're harder to eval. In the future, we think every AI system will have something like an interaction model as the outer user-facing layer, continually keeping the user informed and learning what they actually want.
very cool
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
we are excited to share our latest work on interactive human-AI collaboration!
as intelligence increases, we think progress will be bottlenecked by the ability of AI to work *with* humans -- thereby enabling AI to positively impact the long tail of human experiences
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Early days but what’s most impressive is how natural the interactions are becoming with these omnimodels. Real-time, low-latency interactive AI models unlock applications that are very hard to imagine today. Brace yourselves!
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
We need this asap
With the model's simultaneous speech capability, Horace has gotten a lot easier to work with recently.
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Working on the interaction models is a lot of fun at TML! I can't imagine doing that in a turn-based world. Building it from scratch makes a lot of things so much easier. I am very excited about the future of natively multi-modal, multi-stream, multi-task models.
We strongly believe in the bitter lesson. Making interactivity an integral part of the model will outpace any harness-based approach and will lead to a better experience working with AI models. This is a step in that direction.
Working on the interaction models is a lot of fun at TML! I can't imagine doing that in a turn-based world. Building it from scratch makes a lot of things so much easier. I am very excited about the future of natively multi-modal, multi-stream, multi-task models.
Wait, TML is actually cooking??
Interesting model for higher bandwidth human-machine comms
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
@soumithchintala interesting
Thinky's secret plan: 1: Increase Human<->AI bandwidth 2: Raise ceiling of human+AI intelligence 3: Help humans continue as main-characters in the new world We are at Step 1. Interaction Models are great real-time collaborative tools for humans. Here's a preview:
notable
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Working on pretraining, it often feels like we are building an intricate physical system complete with dynamical laws and dimensional scaling. I did not expect that using our models would also start to feel like interacting with a dynamical system
(1/2)
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Interaction models continuously absorb information from the world and think and respond in real time. They are aware of the passage of time, know when to listen and can interrupt based on audio or visual cues. I am so excited by all the work being done here at Thinky
(2/2)
Working on pretraining, it often feels like we are building an intricate physical system complete with dynamical laws and dimensional scaling. I did not expect that using our models would also start to feel like interacting with a dynamical system (1/2)
@lilianweng @clarejtbirch Nice. Is this your version of Her?
In the past few months, we had a lot of fun (and stress 😅) to produce 12 versions (+ many subversions) and 137 pages in our training run log book. Turns out human-human collaboration is important to improving human-AI collaboration. 😊
I want more proactivity like this
Could I have it track pigeons that walk by my place and then call the police if it’s more than 10 pigeons?
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
holy shit they made Her
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Subtle point: building excellent AI models and agents will likely require us to reconsider organizational structure and communication patterns (of humans!)
In the past few months, we had a lot of fun (and stress 😅) to produce 12 versions (+ many subversions) and 137 pages in our training run log book. Turns out human-human collaboration is important to improving human-AI collaboration. 😊
If you care about building an AGI future where humans are not left behind, come join us.
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
It's refreshing to see an AI lab go in a different direction! But also, as someone who's obsessed over getting my inverse planning algorithms down to milliseconds of latency, I feel like I'm being gaslit when they caption those 1.5s to 3s delays as "instant reactions" 😵💫
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
The real world is physical, the real world is continuous, the real world is real-time
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
Thinking Machines launch should be @prozd and @cHHillee using the model to swap places giving talks at GDC and Jane Street
With the model's simultaneous speech capability, Horace has gotten a lot easier to work with recently.
are you getting it yet? now youre thinking in microturns
last fall, I read Walter Ong's Orality and Literacy twice and on my 40k step walks from Potrero Hill to the Presidio, I couldn't stop thinking about the "Some psychodynamics of orality" section:
- additive rather than subordinate - aggregating rather than analytic - close to the human lifeworld - agonistically toned - empathetic and participatory rather than objectively distanced - homeostatic - situational rather than abstract
then I had about 4 months of When We Cease to Understand the World level psychosis, where I repeatedly accused anybody I could of not being responsive enough and not being collaborative and being too objective and not touching the world in a high frequency high fidelity way, lost in their modern plato's cave of literary sauce.
in January, I explained my job as the guy that makes the sous vide machine do weird things it wasn't meant to do so that the chefs I work with can make the best dish of their lives and that someone told me actually that role exists at Lazy Bear and it was what created their asparagus dish:
turns out sous vide machines are designed assuming they would only ever be used with water:
- the motor expects certain viscosity - no way to clean insides this tool design constrains the chef; he cannot sous vide asparagus in asparagus juice
historically - immersion blenders, vitamix -> era of purees - cheap nitro -> foams
in the same way training runtimes are designed shapes the path of AI: - chat is turn based, now training is turn based, there's no synchronicity, there is no time, reality freezes - chat is turn based, what can you scale? ok scale the model turn -> cot -> o1 - and now here we are, sitting on our thumbs, waiting for claude
the shape of a tool is what it enables a creator do an intelligence transcends those limitations
so in a desperate attempt to end my psychosis, we went katabatic, wrote a bunch of rust, argued a lot with @_alex_kirillov_
and saw a bunch of the best chefs in the world begin eliciting flavors and textures I have never experienced before
here are some
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
What if working with AI felt less like a chat box and more like talking to another person? Today we're sharing a preview of how we think about humans and AI collaborating together at @thinkymachines, with full-duplex models that handle interactivity natively.
It's still early, and we have a lot more to do ahead of us!
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
“natively multi-modal, multi-stream, multi-task models”, good direction.
We’re interested in AI systems that can collaborate in real time, without relying only on artificial turn boundaries.
For audio, this feels natural: listen, speak, interrupt, update.
For video, we think an important version of this is visual proactivity — models that respond when something happens visually:
“Tell me when I start slouching.” “Count my pushups.” “Say stop when the person stops doing X.”
Tessa's quality of life has improved a lot with some nagging.
My go-to test for visual proactivity in realtime systems is live finger counting.
It sounds simple, but it requires the model to watch continuously, track visual changes, and respond at the right time.
Other models we’ve tried could not do it.
We’re interested in AI systems that can collaborate in real time, without relying only on artificial turn boundaries. For audio, this feels natural: listen, speak, interrupt, update. For video, we think an important version of this is visual proactivity — models that respond when something happens visually: “Tell me when I start slouching.” “Count my pushups.” “Say stop when the person stops doing X.”
Find some joys while pushing Human-AI Collaboration.
and "green tea is ok".
P.S. The demo is basically my life at thinky: I start to cut coffee, @liliyu_lili is visually prompt-injecting my human intelligence with sweet snack every day, and I've gained weight since joining TML.
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. https://thinkingmachines.ai/blog/interaction-models