Sponsorship and business inquiries: business@latent.space
Transcript
0:01 · [music] Okay, we're in a remote studio with our new head of editorial, Richard. Say hi.
0:08 · [laughter] Hi, everyone.
0:10 · Dialing in from the UK. And Raine, uh, CEO of Liquid. Welcome.
0:14 · Thanks for having me.
0:16 · Most people when they've heard about Liquid, they they they know about LFM or they've heard of LFS. They maybe haven't personally tried it. The other interesting I guess public data point is that it is very inspired by worm DNA or something. I never really pay much attention to that because obviously I think that that's an analogy but you can tell us how seriously to take it.
0:36 · Yeah, absolutely. Well, you know like when we when we started uh our kind of research about 12 years ago, we wanted to see if we can bring inspirations from brains back to machine learning. So that's that's where everything got started. And then when we started looking into human brain, we realized like when I say we, my current CTO and myself, we started back in Vienna University of Technology. It's called TUNI. And then so I started my PhD. My co-founder was the a master student there.
1:03 · And then what we wanted to do, we wanted to understand how we can um bring in like continuous time math inside machine learning kind of system. And that's what we did. So so we we basically look into neurons, how neurons exchange information with each other.
1:18 · And then um this was governed by very simplistic f first order differential equations and that's that's kind of how how the nervous system dynamics are actually working at the level of curve neurons and synapses and then uh our first focus we wanted to start with human brain but human brain has like a spiking neurons and and it's very difficult to actually learn like we still don't know anything about human brain how how competition actually happens.
1:42 · We know a lot more uh now about a fruit flies brain but we don't know that much about human brain but uh but I can tell you that um we got to this animal because we wanted to do it first principles we we got to this animal C elegance worm which has uh only 302
2:00 · nerve cells in its body and it's like it has like two it's 2 mm long and uh one of the nice thing about the worm was the fact that its neurons behave very similar to artificial neurons they don't they don't spike You know, so that graded kind of behavior in these neurons of this animal, the it's called in in neuroscience they call it electroonic behavior of a neuron. You know, like the neurons do not spike.
2:22 · And that behavior was very very interesting for us because if you build systems inspired by the brain of this worm, you would be able to build like very continuous time kind of systems and they're differentiable. So you can you can do back prop and all sort of other things on top of these things. This was this was the model organism that we were looking at at that time. The most fascinating thing for us was that with 302 neurons it could control 95 muscle cells better than any robotic systems that we had on the planet you know and this is 2016 and 2017.
2:54 · So then we thought that okay so and also another thing another interesting thing about the worm is the fact that 600 million years ago we got a split humans actually got split from this worm in the tree of evolution you know so it's actually like whatever whatever neuroscientists are discovering about the brain of this worm or the cell structure of this worm is actually transferable like to to humans as well to some extent 78% similarity to human genome you know so it is actually a very very important uh uh worm for us and so far it has won four nobel prices for humanity, you know.
3:25 · So, like you can imagine like scientists are really lucky to study this animal. So, it's a very very popular animal, you know, and um and that's why like that's our our research. We started even working with neuroscientists, you know, themselves that are working on this uh brain of these neurons. So, long story short, we mathematically modeled how two neurons exchange information based on the inspirations that we know and the things that we know about the brain of the worm. And then we got those mathematics and we built something called liquid neural networks. liquid time constant neural networks.
3:55 · Liquid was like basic liquid for flexibility because one of the things that we realized is that these are highly nonlinear systems governed by differential equations in a forward pass and they are differentiable. So you can actually run them backward pass and you can basically learn them like really nicely because of the multiple feedback structures that they had. They are really representation learners you know like they learn representation from data really really well.
4:21 · One of the things that we realized is that with a small kind of subset of these neural networks, you can control robots like you know very similar to how the the worm itself was doing it and uh when I say small subset we were talking about tens to hundreds of neurons you know like we talking about very very small subset of neurons and then we we we got very excited so we started applying these things to controlling robots on the robot you don't have that much compute so you bring like int the fact that you can actually pack that much intelligence to control autonomously a kind behavior inside a
4:53 · robot that was like a very attractive feature for us. So efficiency of this uh computation and bringing intelligence into like smallest kind of unit of compute that has always became kind of a a kind of research topic for us and then uh 2017 um professor Daniela Roose of MIT cell so she's she's my uh so she she reached out to my professor in Vienna and then she's one of my co-founders as well. She's the mother of robotics arguably. So you know she's like she has done tremendous contribution to the field of robotics and distributed robots and soft robots as a whole.
5:23 · She started learning about our research and then she she was like blown away and she said okay you guys would you guys consider coming to MIT and continue this research in real world like really applying this technology into let's say cars and drones and many many different kind of vehicles and we were like both Matias and I we joined MIT. So since 2017 I've been like um working with Daniela. I met my force co-founder uh uh Alexander Amini. So Alexander is although was was a PhD student at that time at MIT.
5:49 · So Matias and I were both there and the four of us since 2017 started working on these topics of um again continuous time dynamical systems and continuous time learning systems that and and then the scaling of all of these things applying them into many many different domains.
6:07 · predictive like sequence models like sequence modeling became kind of our thing you know like we started like doing contributions and then uh you know like in this field of continuous time it became a little bit better because a lot of other researchers from different places like from Yashu Benji's lab David Duban lab like from uh from uh University of Toronto and then there was some people from Stanford uh uh everybody was contributing to this field of continuous time dynamical systems you know and this was like imagine recurrent neural networks that are discretized
6:37 · kind of dynamical systems, you know, that are basically computing one step at a time computation, but the time difference between one step to the other step, it doesn't kind of change that much. But this this type of continuous time dynamical systems allows you to have like a little bit of control also over the when when events are happening in a in a in a discrete way, you also account for the time between the events that are happening. So sequences that are continuously evolving you have like a lot more adaptability in the time dimension as well.
7:07 · You see so that kind of aspect of recurrence and feedback kind of calculation. This was kind of the intuitions that we brought to machine learning. again not only us from the MIT side but also like a lot of other kind of people were simultaneously uh working on this with the topics such as neural odes state space models you
7:27 · know like you can think about like all these uh other names that people started like putting on top of like these dynamical systems as a whole you know so and then um what we tried to do then then it was time in around 2020 everybody was talking about scalability you know at the time everybody like from the MIT side we were comp we were competing with um with the open eyes of the world and and and Google's of the world like from a research point of view like we were putting algorithms that are efficient and then perform really really well in in new learning scenarios in the
7:57 · robotics environments because if you remember between the 2017 and 2020 OpenAI was very active in reinforcemental learning and also like doing a lot of robotics like OpenAI gym like for example like some of the things that they were putting out. So we were trying to like get creative around like controlling of robots with these type of continuous time algorithms and showcase how you can actually take learning to
8:18 · out of distribution you know like how how can we have learning systems that are small but at the same time they can take they can do a lot better in out of distribution generalization you know so that's was like the whole thesis that we were but then we started thinking about a scale because everybody has started thinking about a scale as well because we want to we wanted to evolve from this worm inspired kind of intelligence like to something that is like more like complex and get into human understandable signals you know like with audio vision and and and text being like a corpus of data that we cared about.
8:49 · So as soon as we start scaling continuous time and recurrent kind of highly nonlinear loops uh as soon as you start scaling them you realize like how difficult it is to perform computation you know like it becomes really really uh computationally not tractable. So it becomes very difficult to compute. So you need to get creative around how to scale this uh kind of systems. Recurrent neural networks have been like a their major blocker has been sequential computation you know. So you kind of like parallelizing recurrent operators when they are nonlinear.
9:20 · It's really difficult especially on GPUs. This is one of the beauties of transformer architecture because you they're unbiased. They are basically matrix multiplications and you can actually like scale them like it's the hardware lottery. That's the beauty of the transformer architecture for scale like and do for for unlocking like general purpose behavior.
9:37 · But the recurrence itself is like so attractive because the computation doesn't explode because if you think about like attention when when you when you consume more and more data you are exponentially kind of the quadratic cost of attention is something that people have paid attention to. So we've always realized that recurrence is actually bringing this computation lower and they make it sub quadratic or linear you know closer to linear kind of computation as you consume more data. So these algorithms are more efficient but this is only theoretical.
10:06 · So if you can actually if you cannot really scale and do the operations in parallel you cannot scale these systems you know so one of the things that we tried was that one one way to solve for these kind of nonlinear systems to become linear is to linearize them. When I say linear, you dynamically like you reduce all the nonlinearities of the system. So you can apply linear algebra tricks on the system so that the system actually becomes uh uh computable. You you can perform operations linear algebraic operations in parallel you know.
10:38 · So that's that's something that for example state space models are linear version of recurrent neural networks and they lost kind of that nonlinearity nonlinear relationship between the parameters. The reason behind why do you want to do that because you want to make a scalable computation.
10:57 · So SSMs are becoming very popular version of a dynamical system or a learning algorithm because you can now parallelize some other form of operations as well you know like like not just attention but you can have like parallel scan and you can apply linear algebra tricks as I said to scale another format of operations like very much So linearizing this dynamical system allowed us to scale right.
11:20 · So we if you think about the SSM view but the problem again what what arises is that like there's no free launch in computer science right so when you linearize a a complicated dynamics you're losing expressivity so you the system is not going to be as expressive at this nonlinear form then we realize like when we when we apply let's say this state space models like to solving language problems they are having like troubles to learning like dependencies at long context they can compute really efficiently but they cannot they express very efficiently.
11:52 · So the the expressive power of these neural networks always fall short of transformers. So with all the nonlinear kind of mathematics that we learned from liquid neural networks, we we learned from dynamical systems, we learned from SSMs, we learned from convolution neural networks, you know, there's many different variations of operators that you can bring in together to build a general purpose computer.
12:15 · With everything that we learned, we thought that okay, so you know what?
12:18 · Let's not put ourselves into a biased way to go and say, "All right, so our architecture is like the holy grail and everybody has to start like building an alternative architecture and then put a bet on a single architecture." What we thought day one at Liquid, this is like about 3 years ago. What we did at liquid
12:36 · we we started bringing avengers of alternative architectures like all in one place you know like we had like as I told you like people from yashu binges lab from Stanford from MIT like they're all inventors of various types of technology there was hyena hierarchy there was a state space models like Jimmy Smith of Stanford and ourselves like on liquid neural networks so we brought all of these teams into one place and we said let's let's build an unbiased way let's build a meta AI
13:00 · system that searches through the operation space so we build a meta system that searches through operators of interest given the deployment environment. Imagine you want to deploy these solutions on a GPU or NPU. So we applied basically this massive search algorithm on top of building hybrid architectures that allows you to not allowed you to optimize four things. Number one, no sacrifice on quality.
13:27 · So there are a bunch of quality metrics like reasoning capability, language understanding, you know, knowledge. There's so many different kind of things that we want the model to do. Like there are 100 different criteria. We thought that no sacrificing quality against the pure transformer model. Minimizing the memory consumption, minimizing latency and maximizing the speed of computation.
13:50 · These are like the four criteria that we optimize a searchbased algorithm. the algorith like there's a paper that we published on the just the science of this thing it's called the star you know it's called the synthetic architecture of um synthetic tailored architecture like synthetic design of tailored hybrid architectures so I think if you if you look at this just say star liquidi you can you can actually find it so I think that that's kind of the science behind like how systematically we are designing kind of architecture so that framework
14:21 · that meta algorithm is kind of a recursive self-improving kind of system. If you think about it, like three and a half years ago, we applied it, we applied it to really uh have like a search algorithm to go through various hybrid architectures for a given given chip. And then this system became kind of the computational graph of different generations of liquid foundation models.
14:42 · Liquid foundation model version number one was the first instance that came out of this architectural search. It had some elements of convolution, some elements of attention, some elements of liquid kind of original liquid. It has like a it was a mess of everything, you know. So, and that was like the first generation. The second generation, we optimized it specifically for running really really well on CPUs. The LFM2 instances of the models, they are, if you look at them, they have a very elegant and simple architecture.
15:12 · So the architecture is basically 80% doublegated convolutions you know 1D convolutions plus some uh group query attention basically 20% of query. So the architecture is just a very simplified very fast and and high quality kind of foundation model that you can run them on a CPU really really well.
15:31 · Now the beauty of this um uh uh you know architecture search is that our imagine a new innovation or new invention in the architecture space or fundamental building blocks of intelligence happens we add them to this massive meta AI search.
15:48 · So every generation of liquid foundation model that we put out it might have it would definitely have like an improvement over the previous version but the operators are from operators of that we choose and and the target device the target device implies basically what should be that hybrid architecture. So that's how we started designing these liquid foundation models. So what you see today we are at the LFM 2.5 instances of the models. We are a foundation model lab. We're building everything from scratch ourselves as you know like we pre pre-trained these models.
16:18 · We run the scaling laws of these models. We scale these models from tens of millions of parameters now to 70 billion parameters. Like that's kind of the range that we have usually like running like scaling laws on top of models. And then we so far released uh the smaller instances of the LFM2 and LFM 2.5 instances. And the next generation would be LFM3 which is like a little bit slightly modified kind of version of these things so that we can have like added more quality added more stability added more kind of better latency control on set of uh deployment frameworks that we care about.
16:51 · You just gave us a PhD uh you know abstract of what uh what liquid has. Um okay I I have so much to dive into uh you know um you know neuro architecture search um you know gradient free evolution all these things uh but just for the people also listening on the business side um let's also give an update you know I I so uh you know the last time we covered
17:16 · liquid on the podcast was with male from Shopify who's a big fan of you guys I'm sure there's a lot more so I think only you would know and you would know [laughter] and and you sort of bring bring people up to speed on uh this is not just It's a research project, right?
17:30 · Like this is actually like you're not like people are actually effectively using this. The benchmarks are not just, you know, uh uh goal seeks like they're they actually reflective of the world.
17:41 · Absolutely.
17:41 · Absolutely. So yes, so I have to say like Mikuel is one of my mentors now. Like Mikuel is like one of the biggest uh people like in the industry. He's like one of those no BS kind of type of people. I really like like how he gives his advice very very directly, you know. And then so one of the that he he he told us is this you
17:59 · guys you know like we have like a very competent team at Sha and we're building models right so if your model if you're telling me like you you have like models I have to I have to see I have to our team he's a mercenary he just cares about the input that's it you you bring in the technology we will we will test it and we will tell you is it good or bad then they tested it first time and they were positively surprised like of the quality and also like the the efficiencies of our models you know so I think in the space of alternative architectures like we have really really good models.
18:26 · So back back to the business side of things when we talk about this range of models that we are operating and I told you we are optimizing for CPUs too. We are looking into outside bringing intelligence outside of data centers. We started working with enterprises that are caring about data that goes the deployments that are outside of data centers as well. So we started working with car manufacturers, with robot manufacturers, with like u uh let's say laptop uh uh producers, we work with mobile companies.
18:57 · So they have like deployments of compute somewhere outside of data centers. So we go in there and our foundation models are really good and really like you can bring like the best of intelligence on top of like these devices for various type of applications. Examples of partners of ours would be like AMD on the AIPC side.
19:16 · 300 million AI PCs are getting generated onu on the planet like every year like these are laptops you know 300 million laptop 300 to 400 million laptops and I think uh from there um about 100 million of it could be AMD kind of uh powered basically like the core of the building blocks is AMD so we're working towards bringing our liquid foundation models directly running on the laptop enabling some sort of local uh uh deployment of
19:43 · intelligence you know and they can solve multiple different problems S this could be for privacy filters you know they can be in a smart way guardrail like what wants to get out of your computer. Imagine your your type you you want to send something to the chat GBT terminal or to your cloud code kind of agent.
20:00 · Those kind of bulk of sensitive data that you want to send you can pass them first through a liquid foundation model that uh you know like like you know basically shields out like sensitive information in a smart way and then you can share those information. That's like one thing. The other thing that you can do is uh you can have the small models to be a proactive agents directly running on your on your device.
20:19 · So imagine if you have like an agent that is uh having access to all sort of information that is like right now on your laptop and then performing some sort of analysis and gives you that analysis to yourself like everything that you can achieve in the cloud you you'd be want to bring it like as an experience also directly running on a on a laptop and that that would give you like also an offline capabilities also like airgapped capabilities.
20:41 · So you can imagine when you're going into manufact like what we are what we are trying to do we are trying to power the ondevice portion of intelligence today when you think about device intelligence there is a hybrid solutions out there is one model that is actually in the cloud and there are models that are running on the device right so and there's always like a collaboration of these two usually right now if you think about like the most u uh used phones like from Apple
21:12 · intelligence from Galaxy AI is like you know like of of all of these things 90% of the calls are going to the cloud right so these are 90% goes to the cloud 10% on the device the reason is because the device models are not powerful enough or reliable enough to really to really take more off of those cloud kind of use cases why do companies want to run these AIs on the device the first reason of this thing would be cost sheer cost if you have 400 million phones or
21:42 · let's say billions of clients that you have to support and give them a free intelligence access and while you have to serve that intelligence somewhere in the cloud that's going to become like really really prohibitive kind of cost right so what we do we we try to like really and these companies are really trying really hard to bring the AI directly workload of AI directly on the device because in that way like you're just creating margins for yourself right so that's like that's like the first and simplest kind of solution and then apart from at you would have active to you
22:13 · know as I mentioned offline features like privacy sensitive kind of topics and then uh many many different advantages that you can bring like on the device but it is very difficult building a model for the device and getting it to production quality is extremely difficult I can tell you like it's not like just downloading a model from open source and then trying to put it on the device and then now you're successful it goes a lot more than that you have to do terrmal control the behavior of a model in the long context is actually changes you know like you really have to have like degree of reliability to add that it's really
22:45 · difficult to actually get to that point you know so and and we have gone through like uh over the last two and a half years like we've gone through like many cycles of production and I can tell you like it is it is really difficult production grade AI is like way more than just benchmarking the some quality benchmarks on the device you know it goes beyond that and then uh another example of these things would be car manufacturers like Mercedes-Benz is one of our clients what we do in the car kind space we bring incar intelligence.
23:13 · So imagine we announced a partnership like last April that uh it actually happened really fast in the space of automotive is a significant kind of partnership that we thought that inside the car intelligence. So I want to give you like some some insight. So imagine you're sitting in your in your car, you can talk to your car, you can have like all sort of hey Mercedes features, you know, like you are communicating with your car and stuff. Those are all happening with the foundation model that goes directly inside the car.
23:35 · And this system like the the the model that we are powering it's a multimodel solution that can get connected to the to the cameras and it can it can actually do a voice uh interchange you know and also like it has like a reasoning powering through the text based kind of models and what it so this model itself is about 600 megabyte so imagine like if you have like a 600 megabyte intelligence that goes inside every car you can do an overthe-air update or OTAA you can do overthe-air update of every car on their on on the planet.
24:06 · The first deployments that we have on Mercedes, which is going to happen this year, it's going to be uh on all North America Mercedes-Benz cars that are generation 3. We are going to have like the first deployment at scale of uh AI in like actually inside the cars, you know, and this is running on a on a chip that is probably 100 bucks, you know, like in in cost. So what I want to say is that like it's absolutely insane to really think about bringing a really high quality kind of things production grade AI.
24:35 · It feels really really good to actually work towards this efficiency at the levels that we talk about and then taking these things and bringing them into production in the car kind of space and then I can tell you about millions of things that we've done with Shopify.
24:49 · You know Shopify has been an unbelievable partner for us. They are not doing ondevice AI but they care about latency. They care about latency and quality. So what they do they have private deployments that they have to serve like through the shop app the merchant facing and also the client facing uh products of Shopify across many different kind of uh uh things that Shopify does. We have a lot of liquid foundation models deployed in production. One of the examples would be if you go to the shop app today and type in something in the shop app you would actually uh uh send the signal to a liquid foundation model.
25:20 · So there is actually a liquid foundation model there that is serving uh across the stack which is which is very nice and uh and uh we are passing we are passing through like uh some amazing statistics exponential statistics as well in terms of use. It seems like clients are really enjoying those features and now um we are exponentially at over a billion uh
25:42 · request per month you know so that's like the amount of requests that goes through like the liquid foundation models like across like that shop app kind of just just that kind of metric just Shopify got it yeah it's just that and then in terms of popularity of the models themselves as you know we open way we open the we open we we deliver kind of liquid foundation models like to the to the community and We we really open source them and I think the models are very popular right now. They have an overall over 40 millions of downloads and we have like 1.5 about 1.5 million downloads per week.
26:15 · So the models are getting downloaded really much. A lot of developers are taking these things and putting them in production. The range of parameters of liquid foundation models that so far we released are between 100 million parameters to 24 billion parameters and these are a mixture of expert kind of architectures the the the bigger ones. And uh this range of models they're also like they're adding multimodal capabilities as I mentioned like the models could be like both uh processing audio vision and and and text at the same time as an input and then generating audio and text as an output.
26:46 · So that's kind of the the portfolio of things that we do.
26:49 · Was wondering about um robotics cuz that was kind of one of your inspirations when you started. And I noticed you've got like you've got multimodal um models. You've got like a vision language model and an audio or models and audio models. So are those being used for you know robotics purposes and can you maybe talk about some of those experiments? Absolutely. Yes.
27:08 · So when you have like the vision language models like you can you can actually solve vision action kind of problems you know like you can turn into robots seeing the world and then you take an action based on like the what robot sees you know and you can also use them as monitors. So imagine like in factories there are like cameras that are monitoring like the entire operations of a factory and then for we we usually are thinking about industrial operations of robots you know
27:32 · like not not mostly humanoids but also like like just industrial robots we worked with a company called rooe tech I think you should you should find if you just say roboc rootech AMD and liquidi you would find like there's a demo of like what we've done there there's another company called Gbionics I think it's in Italy and uh these companies also like they are having like modular robots and they're using our vision language model for some action space for the body of their robot.
27:57 · You know there's a camera actually like on the body and they are using these things for let's say instruction following kind of capabilities like basically vision action models. So use cases mostly goes inside uh inside factories and then there are like obviously like if you think about like multimodal intelligence as you as you mentioned the smaller instance of the models they for low latency applications in robotics they're really really popular. there's a lot of um new engagements that are coming our way but robotics has been something that we deliberately kind of put it on pause day one.
28:27 · The reason behind it was that because the go to market motion around robotic and also like really building a business around robotics is very difficult and and it is it is delayed because if you think about production grade robotics you need to go down the rabbit hole of verification and validation of the technology and then you get into like regulatory kind of aspect of it for deployment right so we always like because we are coming back from a robotics background ourselves we know how how it is like to really productize some robotics kind of vertical that's why we wanted to build like a horizontal layer intelligence on
28:59 · top of like uh you know places where we can have like OEMs first and then entering into the action world where where verification and validation becomes much more more sensitive as a later step of the company but I think now it is like a good time a lot of because our models are matured a little bit better now we can we have a lot more control over like data and and and and the post- training and customization process of these foundation models for certain type of applications I think it's a very good time for us to really also like start um the robotic vertical like much more seriously than before.
29:30 · Just on on this cuz I wasn't going to I wasn't expecting to spend that much time on robotics, but since you mentioned it and you said it's a good time, do you basically use the same LFM family? Uh I think a lot of people make a lot of a big deal about how different uh you know action models are or world models are for robotics. I don't know how much of that is just marketing versus fundamentally it is actually the same thing. That's a great point. So I will tell you something.
29:56 · So I'll tell you like innovations that happens in the space of foundation models you know that are getting used in any vertical like I would categorize them in three different classes you know like there is model architecture of intelligence then there is algorithmic aspect of intelligence like algorithms that are innovative and then the last thing would be data uh uh processing kind of mechanisms like what what kind of data work do you want to do so innovation across this tree what we've been focusing on like from our research point of view it was like what we saw so far talked about was about model architecture innovation, right?
30:30 · Like it was just about the architecture like efficiency of a model and the ability to become a general purpose computer basically. So that's got the model kind of perspective. Then there are algorithms. The way you train a model, you can train a model with an auto reggressive kind of loss. You can train a model as a you know like as a in diffusion processes. You can train a model in a word modeling context.
30:51 · So you can take that architecture and bring them into so I would say word modeling it's like an unsupervised way of learning uh systems you know like it's it to really build like a representation in between so that from that representation you have a world view of the world like you have like an understanding of the world so of course like I would attribute kind of word modeling on the algorithmic side of intelligence you know rather than on the model architecture architectural change into the so you could technically take
31:21 · some of these LFM architectures and use them in a word modeling pipeline and train them like to have like a better uh understanding of the physics of the world. You see, I would always argue that various algorithms allow you to learn uh the some physical structure of the world with different ways. Word modeling allows you to extract like a different set of primitives from the physical world compared to next token prediction.
31:47 · You wouldn't say next token prediction would create um would would not have a physical they have a phys physical understanding but it's very different than the humans way of understanding the physical world. So I would just say like that physical and there is still a debate between like what learning algorithms are going to unlock what type of what set of capability this is needed for having the chat GBT moments for robots that's like
32:12 · a big big question mark on the algorithmic side on the architecture side I would say you could use any general purpose computer like any any general purpose kind of algorithm to to architecture algorithm to be used in those kind of frameworks that I mentioned just so I understand the business model side before we go back to uh a bit more research. Um I assume these are all licensing agreements because obviously it's on it's you know airgapped, it's on device, it's all these things. How much time do you have to spend to customize a model?
32:41 · Let's say I I come to you with like a very special set of compute constraints or whatever.
32:49 · Is it you got your guys doing it? Is it a collaboration? How does this work?
32:54 · Fantastic question. So this is actually our early engagements is usually with design partners. you know like these design partners are the ones that we we would walk we associate kind of an applied ML team to work with the client and we charge also for that duration of kind of engagement right and then once the solution is built then we deploy the solution and then we charge licenses on a yearly basis on a device basis you know like depending on the depending on the where where this thing is getting deployed you know so and we are charging the kind of for models as a recurrent kind of cost then after we build a
33:27 · solution in a vertical then the time of service reduces for the next customer in that vertical. So imagine if Mercedes-Benz is our design partner and we build a solution around like our our products then the next customers that are coming in the automotive vertical the amount of service that we provide is like reduces like from our side and we it becomes just the solution deployment.
33:49 · So we make solutions in every vertical and those solutions are the ones that are actually like allowing us to sell faster. That being said, because as a foundation model company, we've gone through like the process of building a model from a scratch and deploying these models in production. We've gone through this process many many times. We're building now a platform for model development and that platform is kind of self-s served. So you can actually get access to this platform and then using this platform you would be able to automatically kind of do that job.
34:18 · So and this is something that we are building.
34:21 · Wow. as of it's a very difficult problem you know and you you can close the loop also with like recursive loops and everything but we want to give these capabilities of model building the quality that liquid builds models to the client space to the enterprises themselves is this platform fully rolled out is in beta I this is my first time hearing about this no it's in beta right now it's is in beta like we we're still like testing it out we're very excited about it is a very interesting kind of problem if you think about it like this customization and continual learning It's a very very
34:53 · important next step for the deployment of AI. Right now like 90% of the market is around inference tokens like Frontier Labs like if you think about Frontier Labs and even inference providers right now like Fireworks of the world what are they doing? They they host the models and then they basically they charge the inference right like the inference tokens are the ones that are making money.
35:12 · But I think the next wave of companies that you have you you you should know I can name a couple of them rei there's like core automation there's like thinking machines you think about um uh what else do we have trajectory you know like there's like many different companies like in this kind of category of customization so a lot of people are turning into customization tokens because now there's like a lot of
35:35 · good base open source models as well so people enterprises could actually get we are talking about enterprise AI by the way so you know like in the space of enterprise AI I monetization of customization tokens is something that I think people are going to capitalize on and it's going to be something that is very very um interesting and it's going to be the next uh next wave of um
35:55 · decacorns that are going to emerge from these kind of category which some of them are already decacorns like at at at the like Jeff Dean just started like a lab again like on the same type of topics you know so you can imagine like customization is going to become like a very and when I say customization don't only think about post- training a model.
36:13 · Think about the depths of customization from pre-training to inference. That's customization. You can do pre-training, you can do mid-training, you can do post- training, you can do data gen, you can do RL. All of these things fall into this category of customization of a model and then allowing agents to do that and then looping the whole thing so that you can actually have like a recursive loop that allows the customers to own their own intelligence.
36:36 · That's kind of something that we are planning to do and we are providing like this platform to customers so that they can they can really build liquid foundation models at the quality that we can build like that's number one and then once they put them in production they can keep them always up to date and always kind of to the level that they to the the level that they want them to perform. So just um before we head to the research um can I ask a little bit about your developer platform uh leap how are developers using this platform at the moment and what kind what kind of process do they use to um deploy your products?
37:08 · So if you think about um uh uh lip is like a a library we have a cookbook you know like at liquidity this is like an open source kind of place like you can go to our cookbook got a lot of attention there is so much going on in there you know so we literally like provide like even a lot of insights you know like like this is manual this is built for humans right lip is also built for humans like you're a developer you want to use liquid foundation model you want to fine-tune them you go there you can take like some of the like lip fine tune for example is one of those places where you can fine tune one of our models.
37:40 · Then there are like deployments you can leap allows you to also extract kind of a bundle that you can deploy like an inference kind of ready GGUF kind of format like a llama CPP kind of ready kind of system that can go directly on the device if you want to have like a let's say CPU deployment you can go to our leap platform and it allows allows you to extract one of these bundles that are ready to be deployed you see so that's what what what leap does and again it it think about our current
38:07 · platform that we are building right now it's literally the automated version of like what we have been doing like with leap right like there are some functionalities that leap has and some functionalities that it doesn't so the ones that we want to we want to complete the leap platform as tools and then be able to also provide enterprises with the capability to do that in a way that we we use those tools but those are kind
38:29 · of individual kind of segmented tools that you can use or give your cloud agent to really like take them and and use them to fine-tune an elephant that being said there are also like support of liquid foundation models in the you know very popular kind of places like onslaught and you know hugging face kind of they have like the TRL kind of fine tuning kind of you can use any of those kind of open source uh packages also because we we now try to like be really like connected to the open source community we love the hugging face guys we love the SG lang guys you know like we work very very closely with them and
39:01 · um and and and really bringing like support and the llama CPP project itself is an amazing effort to really deploy intelligence outside of data centers.
39:10 · Yeah, I I haven't heard TRL in a in a while is uh you know there's all these every year or two there's like a new hotness like right now it's unslaw but like you know we've also covered exoltole in the past as well. One thing we my eyebrows just went up when you said self-s serve.
39:25 · I think I I had expected that you would do a lot of services maybe maybe these days it's called forward deployed I don't know whatever the the the hot term is but is there a maturity or hierarchy of things right like so basically the way I think about it is like okay everyone's talking about continual learning but probably most people are not even ready you release for example quantization aware distillation right and like that I can do that that seems easy that seems like self-s serve like okay you know you give me some guardrails on on how to do this like I can distill big model owns a small model, right?
39:56 · Is there a stage of like, okay, you start here and then you work your way up?
40:00 · Absolutely.
40:00 · Yes. You touched on a very very important problem. I'll tell you a little bit a deeper perspective. We we work we worked probably with 200 of the Fortune 500 companies like so far like we really are like in touch with all the enterprises, you know, and what enterprises are exhausted. They went through the process of building their own intelligence for a long time. They try to have like applied ML teams themselves, you know, like we're talking about mature enterprises, right?
40:24 · We're talk these mature enterprises, they tried out like intelligence, downloading models from open source, building their own models from scratch, you know, like for their own kind of topics and stuff and this this this did not result in actual production kind of lines and they always defaulted back to the cloud solutions, you know, so they never got to the production quality that they want. some of these products are are out there but I can tell you the ratio is like 80% of the production grade kind of stuff doesn't the the PC is 80% of them
40:54 · do not hit the production kind of level so that's one of the exhaustive kind of things that I see is this second thing that I see is that there is massive demand for developers right now that they want to have access to the highest form of intelligence for development they want to wipe code stuff like everybody wants to token max you know like there was like these kind of things like it's It's it's such a virus that is in the hand of like in the head of like developers you know and everybody is and again when you do the value attributing
41:23 · value how much value was generated off of this wipe coded wipe coded repositories and stuff that is happening at organization mature organizations you will again end up with 90% of these tokens are useless tokens. So when you actually have enterprises see this and they want to also control the cost that they have there is a massive demand now for them for them to have a self-s served platform that allows them to really get to the point that they want to get as soon as possible to a value that that they want to receive you know while satisfying the demand of their own developers.
41:55 · So what we try to do we try to build the platform not as another UI another new type of uh set or something.
42:04 · So our platform is literally going to sit inside a client like very similar to how you interface with your cloud code very similar that you you take your uh uh you know your agentic harness and you just give it the tools of liquid AI and these tools would be able like your own favorite kind of agentic harness would be able to in a in a stepbystep manner
42:25 · walk you through developing and building and deploying a liquid foundation model production quality level that the way that we would do But this process is an interactive process with the developer at the moment. Human in the loop aspect of it is very important to the point that you mentioned. You mentioned like are they ready to do this you got to guide the customers right like you got to guide them the developers to really get to that point. So that that is the automated kind of stuff.
42:50 · You can also put it on a steroids you know like we have like you can put it on on auto mode like the whole platform and say click one click and then just deliver this. I don't want to see any intermediary stuff, but obviously the quality of the resulting model is not going to be as good as when you're having an interactive kind of a developer actually interactively build something with it, you know? So, we try to get ourselves Let me let me push you. Yes, let me push back on you, right? If I have good evals, why do I care what the process is, right? It should just auto research itself.
43:20 · Why do I have to look at the logs for the first time? I fully agree with you for the first deployment, right? So imagine imagine if you if you put the model in in production based on the evals that you had now you put the model in production then new data is going to come in you know like new data is going the model is in production actual production environment okay now imagine new data is coming in new requests like complete like all of a sudden like the
43:45 · whole distribute there's a massive distribution shift of let's say community wanting different things you know from your models how do you handle that you know you got to be able to handle these things like we because your evals are going to become obsolete.
43:58 · Static evals are going to become obsolete. So you need to have some sort of a dynamism like there and this would only happen if you have a continuously evolving system. You see evals cannot stay static. That's the thing for the first iteration. I fully agree with you you know but if you go to uh you know production then the evals are not going to stay the same you know you always have to change the criteria. I definitely agree on on that. uh I think to why we're moving towards environments and um you know harbor style tasks and all these things.
44:26 · What other trends are you seeing in like sort of post- trainining um our own environments? um any uh you know like uh there there's a lot there which obviously Maxim has has has talked about but effectively are we customers let's call it doing more and more of the post training and you are doing the architectural work is that is that the division of labor something like that yes I would say not just post as I said like don't forget about just post training I I talked about the whole model development from scratch we want
44:58 · to be able to everything including the the architecture search Yeah, if you're as sophisticated at let's say Qualcomm that wants to let's say design their own let's say like for the for the next generation of their chips they want to have like a dedicated computition graph that works the best on top of Qualcomm chips on an NPU that they're designing
45:17 · you know or there are like also some companies that are etching kind of weights of a neural network directly inside the chips you know like these these things are happening I covered talis uh yeah tales and etched before so as you know like there's there's so much in the in the silicon kind of world that is also because those guys would be interested in our architecture search more than anybody. So silicon is actually extremely staying close with silicon partners is actually very important for us because our architecture search is hardware aware.
45:42 · So why because they want they are putting their hardware or maybe next generation of the hardware in the loop of finding out what is the best architecture for the criteria that I care about. So imagine you launch a process like that. Silicon companies they really like this automated foundation design kind of uh uh platform that we are putting in front of them.
45:59 · So that technology and then as you mentioned I agree customization wants to get more and more to enterprises you know like enterprises would be able to own their own model development and deployment you see these are the two things that uh you like this is kind of the true meaning of sovereign AI you know like everybody would be able to own their own intelligence and then hopefully one step further would be all
46:20 · the customers like every single person in the planet would be able to own their own intelligence as well so that's kind of the next level personal intelligence you know hopefully we can unlock that and I just wanted to quickly touch on things because like you left so much in there. It's uh there's uh there's actually a lot to to dive into. Um in terms of neuro architecture search, you're the first I think maybe you're like second or third uh company. You know, we recently covered poolside where they talked about their model factory.
46:47 · I'm sure you've uh talked a little bit about that. And then also a bit back I was also very excited by the tooling for for search, right? Um Apple had this uh paper on Taria which is their sort of LM ops like layer bylayer inspection and ablation tool. What have you found in your new architecture search or model factory type things that you thought would work doesn't work or is like you know surprisingly effective like anything that you want to highlight from your process or research.
47:15 · When we do architecture search on in the space of foundation models, we run a scaling loss. Like imagine for every architecture candidate, we try to run a certain type up to a certain number of tokens, we run a scaling loss. Okay, so like this this being like more throwing more data at a certain fixed size of the model or increasing the size of the model, you know, like and and see like how well it can actually train given that candidate architectures.
47:40 · The things that we learned is that the larger the the neural network you are making the architecture it should be if you want to build general purpose intelligence it should be unbiased. So you have to unbias your system. So for example if you look at LFM2 architecture that came out this doesn't have any of these gated delta nets and all these things. It's like a very unstructur it's like 1D convolutions literally like 1D convolutions like this is like the simplest thing that you can actually do.
48:11 · It's like an unstructured way. Attention itself is like a very unstructured as I said when architectures are not biased like attention like transformers architecture. You would you would you would be able to scale them. So at scale I would say a lot less bias is needed inside the architecture. You don't need to add add features out loops at some sort of like you know so you you got to simplify the architecture the the larger you make them.
48:36 · On the other side of the spectrum, imagine if you want to solve problems at a very very small scale and you cannot afford to have like the uh uh you know like you have like memory limit let's say like 8 GB of memory limit you know there you can get more creative on the architecture side you can make the models to have multiple feedback loops you know you can add like a lot more complexity to the system in the small model regime the more biases you add to the system the more expressive dynamics you get in the larger scale you remove those biases.
49:08 · This is what what what we learn. Second insights that I would tell you is data modality. There are some data modalities like audio. In audio, you could use a continuous time dynamical systems or SSMs for example, you know, like they're very effective in audio, but on text, they suck. You know, like SSMs like they don't work like really well on text. And when you when you think about like let's say if if you want just want to have an audio model, throwing in a recurrent neural network at the at the scale that is needed for humanly like like having like a really good quality kind of audio output.
49:40 · This is kind of the place where I would say recurrent network would be very much shining you know again at small scale you see like when we talk about like this this category of models that we we search you know below let's say 20 billion parameters you know recurrence can be like really really effective there you see and [snorts] but I would call them a more biased architectures
50:00 · you know like they they have like more feedback you know like there's a structural gate you know like they have like a certain certain architecture on architecture archetypes like on top of them and then you remove those things as you scale the most that's kind of the the general kind of uh understanding that we have you know no surprise that uh I think Jason we had this um post where like at small scale you add in biases and then eventually you have to take them out um I also I I think layer
50:28 · looping or what I don't know if this is called layer looping do you call it layer looping or whatever the yeah yeah yeah looped loop layer yeah that one is a bit unintuitive why it works at small scale but not at larger scale It seems like it would work. It's mostly like a efficiency thing I guess. So maybe that's just my that's my explanation. I don't know. But I would say still you know looping it's again an a bias like any any format of looping.
50:56 · It is a bias that you're adding to your system. You know like you you make it complicated and then you you got to you got to find out an infrastructure for by the way. So we don't have the correct infra the current GPU infrastructure that we have is optimized for this type of workload. you know parallel matrix multiplication you know everything that you can parallelize like in a parallelizer do imagine if you have like photonics you know like I think this race of photonics would be like an amazing kind of a space like if you think about it if you can actually communicate with the speed of light now
51:26 · compute in a serial way what is the speed of serial computation when you're moving with the speed of light it's like humanly not perceivable right you know so what I'm saying is that like I think phutonic could be like an unbelievable way for us like to move if you think about like medium change you see so that that could also unlock like nonlinear operators I know companies that are working on like photonics and then the quantum computers like you know everybody's working in this in this space of alternative mediums for intelligence or or compute in general
51:54 · those are kind of the another opportunities that I see like opening up and I and I'm really excited about photonics because I think I think there's a lot of a lot to to be like because sequential computations this is something that from a linear algebra point of view we learned it for nonlinear systems because we cannot really parallelize nonlinear system. So, but imagine if you have like a speed of light computational kind of speed for for for nonlinear operators, you know, then you can actually build the next generation of architectures because then you're not limited by the by the substrate you're hosting intelligence on.
52:23 · So, one I'll give you one uh quote from I think it was from Christopher K who was writing like some book on neuroscience. I think I read it like about 14 years ago that he's not it wasn't a a a quote. He was at a conference and he was saying intelligence is a property of carbon. So because the substrate of human human is a property of carbon.
52:41 · So you he was saying like okay so natural life is happening on a different substrate and we're building intelligence right now on silicon right so the base of intelligence is very different like I think substrate is going to play a role into like even the next generation of architectures you know if you really think about it because it would unlock various formats of computation apart from like just parallel computation through GPUs that's also another space that I would just bring in okay we don't have a ton of time to mention this but you you brought this up so I have to ask
53:13 · uh I will say actually a surprising number of people in my circles in in SF not that excited by quantum computing if we did have at scale like a bunch of cubits do you actually think that it would change anything there's some people that disagree so I want to hear a few I think the only thing that it changes like massively it would be search because I think search is one of those algorithms that matters so much and imagine if you can have like parallel search you know and and what is optimization you know at the at the end of it like We we are uh
53:43 · you [gasps] know we're submitting ourselves to gradient descent you know right now because we don't know any better way to perform massive search at scale you know it is a search in a parameter space you know and and if you think about quantum computers like I think search is going to evolve massively that one hopefully that is not controversial but uh yeah it's it's very interesting that there are many other bottlenecks that matter to us that actually is not solved by econom absolutely but I don't think we are closed Because I think 2017 I had I was
54:14 · at one of these Nurips conferences and uh and I I was talking to um I think it was Microsoft or IBM like they were oh no it was IBM that they were they were bringing they were showing off their quantum beats like kind of the the thing that is like hanging from the ceiling you know and they were showing it at the at the conference and then I was talking to the guy and I said okay so how can when can everybody actually use something like this at that time 2017 he told me 2035 okay Okay. So, coming I think it's still too early so [laughter] to say anything.
54:44 · Uh yes. Yes. Uh okay. Coming back closer actually for listeners. Um we've done a bunch of related podcasts. Um we RWKV, you know, is is another like pretty widely adopted non-transformers architecture that's like sort of loosely inspired by RNN's. Uh so we've done a couple episodes on those things. We've actually had Tree Dow on.
55:06 · And I we haven't had Cartisia on yet, but like when you mentioned how SSMs are very attuned to voice, at first when I when Tree talked about it to me and Karan talked about it to me, I didn't understand like why voice to me actually like the simple explanation is there's just not that much state. Voice is like continuously inferencing anyway. So like this actually pretty ideal. Yeah, it's it's a continuous on sequence, you know, and sequences that are time series.
55:27 · You can say if you throw a dynamical system that is designed to handle time series data that's like the best version of thing that you can actually model time series right so that's why that's why SSMs are pretty effective SSM and RNNs and continuous time systems they are very effective in in in basically sequential kind of data you see you brought out time series the other thing about my background that I bring into this is I used to be an options trader right where uh we actually would do have to do stocastic calculus uh and solve
55:57 · solve those kinds of os uh which is another kind of continuous time finance which is what what we we talked about there.
56:04 · Yeah.
56:04 · I I just I just think like it's weird that like this hasn't been a priority till now and like now it seems like once you explain it it's like relatively obvious that we this is a thing that we should exploit. It's like a dimension that we should exploit. Are there any others in your mind that you know obviously you're scaling out this you're maybe like let's call it 10 years into this journey but like okay if we're wrong about continuous time what else?
56:26 · That's a good question. So I think there's a lot of uh you know like I when when we started like liquid as as an idea so we were thinking about efficiency of intelligence and adaptability of intelligence. I think we we haven't explored too much on the adaptability of intelligence like forward pass for us is like basically forward pass to a to like a model like you know a lot of a lot of forward looking competition is like literally like just a forward pass competition. So what if like adaptability on the go?
56:55 · A lot of people would come would tell you that okay so in context learning is solving that problem because in context like systems are actually like when you bring more data at the test time they would somehow kind of solving for some sort of a back pro like you know like a pseudo very very similar kind of to back back prop situation. It's like a list square algorithm like as as a in context kind of learning is happening on the on the test time compute.
57:20 · But in general like I think I think approaches where we can combine forward pass and backward pass at the same time. This would be a new generation of architectures and I think there will be an evolution of this thing. I know that I've heard that SSI is also working very on the similar architectures to the original ideas of liquid neural networks you know. So that's what I heard. But we will see we'll see how have they not reached out like come on like what's going on?
57:46 · But but in general like you know like me I know that all the found you should also also already assume that all the foundation model labs you know entropics and open eyes and everybody else is like massively invested like they have teams that are working on alternative architectures. Another thing that I want to say is that again why do you want an
58:05 · al alternative architecture right this is the question you have to ask the first thing for us would be efficiency and everybody else is now seems to be like focusing on post hawk optimizations or really like a speculative decoding for example like like one of the spaces like try people try to make things quantization you know like it's another
58:22 · like quantization of training you know like these are these are the kind of places where people try to pack information and you can do a lot in there you know because these are kernel science and and I think with the with with the with better AIs, you know, I'm I'm hoping for the next generation of AIS to get like a lot better at what they do today, they can uh people can really optimize kind of the current architectures to get efficiency, you know, out of the models. Now, if you want to unlock the next generation of capabilities, that's a different story.
58:47 · You know, that could be a meta algorithm. So, think about it also like right now, how does our workflows of coding today look like? there are like there's an orchestrator and there are like a lot of agents that are so in the future multi- multi- aentic kind of architectures like that's another thing that I can think about like we can think about as the atom level or we can think about as a system level right so you can also build like new systems that are composed of smaller kind of units of
59:14 · compute like like let's say like a unit of compute could be neurons and synapses or it could be an LFM you know or it could be a small model itself and then you can use and bundle and you can build basically these systems of systems, right? So that's kind of another direction that architecture and I would call them cognitive architectures, you know, where you have like multiple kind of agents that you can actually bundle them together to build like a whole full-blown uh system as a whole. So I think a bunch of directions I I I pointed out that I think could be an exciting uh future step.
59:41 · If I were to paraphrase maybe you know agents should be able to train models casually as part of their job and obviously LFM would be would be quite adaptable and it's not actually the LFM it would be uh leap because you need the full platform as well in order to adapt to whatever the the situation you you're deploying in.
1:00:00 · Definitely you all these named uh directions uh probably you're you also have like sort of more near-term research directions.
1:00:06 · What what would you say is the current research agenda for liquid? We are very much focused on massively multimmodal systems you know like we thinking about like training every all the data modalities at the same time and at the same time like uh longer horizon kind of reasoning of course like this is one of the directions that we want to do.
1:00:25 · Imagine if you want to do reliable ondevice intelligence kind of services like assistant you know like if what all of those things require like higher degrees of uh reliability for them for the models. So we work a lot on that direction. We work a lot. We have a department for efficiency you know like we work on massive like layers of efficiency you know like you can think about like kernel design it goes to inference this goes to infrastructure RL infrastructure that is like much faster you know like making like all of our processes that are happening like from pre-training to uh delivering a model to
1:00:56 · production how to make this all the infrastructure like more efficient you know that's like the focus of the company then um from a research point of view we're looking also non-human understandable data for example think about signals, you know, you can think about biology as well. Like we have some biomodels as well. Like for example, we building DNA foundation models. Like your DNA with your eyes, you kind of like really read it. They're long context data, you know, like they're like not dense as like let's say in the vocabulary, they're like four vocabs like that. You can actually like letters that you're dealing with.
1:01:26 · But I mean from the sequences length point of view like they're very large. There are a lot of applications that we are excited about in that kind of realm. And I think our type of technology is like naturally pushed us into like looking into this uh alternative type of data as well as I told you like our architectures we have talked about dynamical systems and longer sequences of data. So DNA data could be like one of those places where where we can bring value and then as I mentioned multimodal research is extremely hot right now at liquid.
1:01:56 · Yeah.
1:01:56 · Yeah. Yeah. Uh you said uh video audio in and video audio out. Video, audio, text in, audio and audio and uh text is assumed. Text [laughter] is assumed. Exactly.
1:02:08 · The the in is fine. The out is new.
1:02:11 · Right. Correct. That that that is like typically I would never expect someone like you to care. Is it realistic?
1:02:18 · It is to to to have a you know image out audio out video out from a 7B model that like is usable. I don't think I don't think it is visible to get a video out of that of that like a proper like very very high quality high fidelity kind of videos but um but but directionally if you want to solve physical problems you know like and if you want to have like a vague understanding of the world imagine you you're training these LFMs of let's say in the range of 7B below in this
1:02:48 · kind of range you want to train like a with a with a word modeling kind of context like you want to train a liquid foundation model you would be able to encode some sort of uh physical knowledge inside the model but but the model would not be able to output something very high fidelity you know but it would add to its own understanding but it would not be able to output that you know it can basically
1:03:08 · like have like a little bit of a better representation internal you see like if you retrain them like with the word modeling especially for video applications where video generation matters you know like not just for entertainment but we are talking about like grounded industry work of a robot that is like operating in a factory you see yeah you I mean video understanding is actually so underrated. Uh there's still not enough uh mostly it's just interled images but like we we want more.
1:03:34 · I'll tell you in the space of multimodal intelligence it's really really a lot to be learned like so far because it's it's really not that many u highly multimodal systems you know like this is this is like a very very open uh research topic like for foundation models as a whole. It's going to enable like something completely different. One thing I wanted to double check is the um something that's adjacent to your space which is modular.
1:04:00 · Uh they just got bought went to Qualcomm. They they just released their the uh you know 1.0 open source. You guys famously switched from uh Jax to PyTorch. [laughter] Did you did you explore um you know Mojo? It is supposed to be for you guys like that you are doing custom kernel work.
1:04:19 · You are python centric. uh if it didn't work why I mean modular is basically operating on a space of kernels like they're they're one one layer below the foundation models you know and they are they they try to be also like us like Switzerland working with all the hardware providers but they now got locked in into Qualcomm I was on a panel with Chris actually like uh some uh weeks ago actually like on an AMD panel like uh and and I can tell you
1:04:47 · like I have so I mean he's one of my heroes you know like Chris Chris Latner like he's He's unbelievable. Yeah, he he's probably listening.
1:04:55 · Yes.
1:04:55 · [laughter] But what I what I can tell you is that um one thing that from first principles we try to do at liquid, we don't want to reinvent the wheel. You know, if if the fundamentals is actually like I'll tell you for example, it's so there's a lot of inference companies that are trying to replace Llama CPP or building rappers around Llama CPP for ondevice AI. almost every single of
1:05:16 · these things that are not locked in into like one ecosystem and they try to be like a general purpose language for all hardware that is actually out there all of them are worse if you dig in they're worse than the original llama CPP so we don't want to change base you know so that's what I want to say there are some successful ones that are locking into one ecosystem and I feel like Mojo is going to go into the similar direction you know now that they're getting locked in into like the Qualcomm ecosystem I
1:05:44 · think there this there's going to be like a lot more success but but the platform itself is extremely makes things easier to bring like intelligence like at that layer you know for for hosting intelligence you like so so what they provide is actually like useful but we haven't we haven't been uh using those for for our uh for our own deployments yet you say llama CBP is you
1:06:04 · know better as a as a baseline but like is it just production hardening or is this something fundamental about the approach I'm trying to figure out it's a compiler so it It's like very optimized C++ kernels, you know, that's what it is. I mean, yes, but but Chris Ladner knows C++ like this is not a [laughter] this is not an issue. As I said, you could it's extremely difficult to unify it. I don't I don't believe for example in the uniformity of the Mojo kind of platform. You need to do custom work and what this is the work that the modular team actually put into.
1:06:34 · So I think the version that of Mojo that runs on let's say AMD you have to design like certain kernels for Mojo so that they're AMD specific or let's say Qualcomm specific and all those kind of work that they do that underlying work is like a very solid work and if anyone can pull pull it off is like basically Chris's team you know like that's what I would say but I would consider those like of course like what Chris does what Jonathan Ross did like with TPUs and and and all those kind of things that they they're doing. So I will tell you like these are kind of the very fundamental works that they do.
1:07:05 · But I will tell you like we would stick to the the the most native kind of language, the most native platform that allows us to touch the computer like as as close as possible. You see ecosystem always wins. You know it's it's just really brutal answer. It's not it's not fair to new entrance.
1:07:22 · It's not fair. Um it's it's just a hard problem. And I would and I would also encourage like people like you know a lot of entrepreneurs these days like you know like I'm now four years in into my our entrepreneurial journey a lot of people my friends and and people are coming and asking hey you know like how's how's the journey I would just say like if a problem is hard to solve it's actually worth it makes it more attractive you know it should make it a lot more attractive for you you know because easy problems everybody else is going to go and solve it you know so maybe the the the question I was going to end the episode with is mostly you
1:07:54 · know it's going to be the 10 year anniversary of the transformer next year. Uh is attention all you need?
1:08:00 · Attention is definitely one element that you need you know and I think I think it will be a long time that we will be living with attent attention based system. So attention is definitely going to be inside like any of the it's like you know how how do you see you know how do you see how how's your eyes are working you know but you you need your body is uh I I randomly noticed like uh
1:08:19 · actually you're not seeing the whole view your eyes are just bouncing around all the time and your brain is compositing right yes you you see the difference between two frames by the way this is what you see so it's basically you see the differential oh left and right eye no like you're you're basically the the way that humans see you don't see one frame You see two fra the difference of two frames.
1:08:40 · I need to go look at look this one up. Look this one up.
1:08:43 · So so attention is is one component.
1:08:46 · Attention is definitely one component.
1:08:47 · But then again hardware awareness customization. Yeah.
1:08:50 · Embodiment like there's another there another aspect of so I would say like intelligence that wants to get deployed in the society outside of data centers you got to have like some sort of embodiment. agents that are actually inside the virtual world, they have also an understanding of their virtual physical kind of environments that they are operating in, you know. So embodiment is also true for virtual agents by the way. So you know, so you got to you got to have like those kind of aspects in there. So embodied intelligence and em embodied reasoning is like some something is very important. And then adaptive intelligence is another another level that we we really want to add.
1:09:22 · Is that routing related or what do you mean by adaptive? Oh, adaptive I mean like systems that can always evolve like continual learning, continuously kind of improving systems. Yeah.
1:09:34 · Okay. Well, uh that was a a really good deep dive. Uh is there anything that we should have asked you that we didn't cover?
1:09:40 · No, I think that was a great great session. Thank you so much. [music]