Transcript
Hook
0:00 · We've actually changed our SDK team to think about agent experience instead [music] of uh developer experience. And and we think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that's not even typed when it can [music] basically make a couple changes in a decorator and it gets this sort of self-provisioning runtime of uh being able to see [music] its changes live in action.
0:29 · Before we get into today's episode, I just have a small message for listeners.
0:32 · Thank you. We would not be able to bring you the AI engineering, science, and entertainment content that you so clearly want if you didn't choose to also click in and tune into our content.
0:42 · We've been approached by sponsors on an almost daily basis. But fortunately, enough of you actually subscribe to us to keep all this sustainable without ads, and we want to keep it that way.
0:52 · But I just have one favor to ask all of you. The single most powerful, completely free thing you can do is to click that subscribe button. It's the only thing I'll ever ask of you, and it means absolutely everything to me and my team that works so hard to bring the Inspace to you each and every week. If you do it, I promise you, we'll never stop working to make the show even better. Now, let's get into it.
Introduction
1:17 · [music] We're here with Ashhat of Moto, CTO of Moto, together with Vivu.
1:23 · Congrats on your CC.
1:25 · Thank you.
1:26 · Uh your party yesterday was amazing.
1:29 · Yeah.
1:29 · All the photos and all the swag.
1:31 · We we had a bunch of art installations which uh is kind of fun seeing like our products on pedestals next to like Rodan. [laughter] Very nice. Very nice. Um when you started it was not the GPU inference company. I mean maybe it was in your mind. take us back to the origin story.
Modal’s origin and why Kubernetes wasn’t enough
1:53 · Uh I actually first met Eric who's the CEO through an investor and um back then Eric was already thinking about building uh a new kind of runtime and he he got there thinking through why are workflow orchestration products so hard to use?
2:09 · It's because you have to run them on Kubernetes. Kubernetes is hard to manage. It's not built for burstiness and um custom images and has a terrible developer experience. And I I'll inject for for listeners uh who are new. Uh we interviewed Eric two years ago and there's a bit more of the story there from Spotify and all those things. And I I actually came across Eric through data council because he he did that talk on the the sort of serverless container stack that that you guys did which was like that was my first like okay I need to take models very seriously moment but it was still very unclear like do I actually need all this for just my data pipelines? [laughter] Yeah, I mean initially what we were thinking about was if we build a better runtime. It's a very useful primitive in itself. It's um there's a lot of things that um get solved by serverless functions like you can do uh ETL stuff, you can do job cues, you can do all this like bursty processing which it turns out every company had needs for. Um but then we also were thinking about this as like um this is a primitive that we can build a whole collection of products on um which are very virt. So perhaps data engineering would have been the first one but we were thinking about inference back then it was more classical inference like computer vision stuff and running XD boost and whatnot but we we added GPUs to the product a year before Chat GPT came out. We just didn't think it would be that big of a deal.
3:36 · Yeah.
3:36 · Just like add A100. Was there any like early key problem that really sparked off why you built it? Yeah, primarily it's just um none of the tooling that that was out there was built for um one a really great developer experience and also there's a general trend of um a lot of the workloads that we were seeing were very we this is I wish there's a better word for it but compute heavy like they need um um one like need a lot more resources so you need to burst up and down a lot versus like Kubernetes designed for like slow scaling and uh more for like web server use cases And also there's just a lot more specialization in like what kinds of environments these workloads run in. Like we sometimes they need accelerators, sometimes need different kinds of images and this is just like a consistent thing that we saw across a lot of companies. That would be the the next step.
4:27 · Yeah.
4:27 · Be be nice. I don't know how much this factored into the early story, but I wrote a post when I was at Temporo about infrastructure software defined infrastructure or something like that.
4:36 · The self-provisioning self. Yeah. Yeah.
4:38 · I couldn't even remember my own post and then you put me on the landing page.
4:42 · Yeah, we we really like [laughter] uh the term and and so we we stole it because you had the insight that everything can just be in decorators next colloccated with the code, right?
4:52 · Was that a big part of the original story or was just like a DX layer? that that that was uh really important because we really didn't want people to spend um so much time uh writing YAML and it seemed like you could really condense the surface area of what you're doing uh put it in code so you can actually operate on it just like you can operate in other code like build stuff that's more expressive and dynamic and so yeah that that was always a very important part then the push back is this is a DSL it's you're close source uh I am locked in to modal.
5:26 · Yeah, we we never really got push back for that because the nice thing about modal is you can bring whatever code you have and sure the DSL is that the sort of configuration layer for uh what hardware you're using, how you're scaling things up, but you still own the code. Uh and that's that's been an important um part of our story even as we do inference now.
Developer Experience → Agent Experience
5:46 · Yeah.
5:46 · How much of it do you think still stays the same today? Like if you were to build something today, DevX obviously very important, but I feel like you know a lot of this has kind of been changed with just hook it up to an agent, have cloud code, have codeex implement a tool. Um there's very agent native primitives that are kind of different than if I'm doing this myself, right?
6:08 · We've actually changed our SDK team to think about agent experience instead of uh developer experience. And and we think that the same benefits that apply for DX also actually apply for AX, which is why would you have an agent read through hundreds of Kubernetes files and like write YAML that's not even typed when it can basically make a couple changes in a decorator and it gets this sort of self-provisioning runtime of uh being able to see its changes live in action. Um yeah, it just seems from from the customers we talked to um they actually find the model is way faster used for agents versus operating on a different substrate.
6:48 · Yeah. Because like you you again you collocate the infrastructure requirements to the code that runs it.
6:52 · Well, the the the negative thesis now is that nobody's looking at their code anymore. So there's no point. [laughter] Yeah. I mean people aren't looking at code. Uh one thing we actually still see is is really important is observability.
7:05 · Yeah, how good is your dashboard? And of course, like we have um we push a lot of it to the CLI so the agents can do their own investigation, but you still need humans to go interpret what's going on and uh you know, make judgment calls and whatnot. Uh and that's I feel like um maybe more important now than looking at the code itself.
7:26 · Yes.
7:26 · because like you know it's you can try to treat the code as a black box and then use sort of see the observable action that comes out of it and then just prompt a change. [laughter] Yeah. So I I think actually think it it takes a bit of restraint to not specialize to say I want to ship a new primitive and then just just be general purpose. People ask you what are you for you're like I don't know we can do this we can do that. Uh yeah, I'd [laughter] be curious to say you know like okay if if we were to ask you like what is modal for even at a high level there's a lot you guys do sandboxes GPUs uh everything how do you answer modal is a cloud platform that's built for uh where we've built the primitives from scratch for AI applications uh and right now it basically covers um inference training batch crossing uh and sandbox workloads but we're building a lot more and I noticed you didn't say web server so there there is still a role for like the always on large scale Kubernetes type of things.
Modal’s AI cloud primitives
8:23 · Yeah, absolutely. We're not trying to compete with the renders uh of of the world. Uh because yeah, we think the differentiator for us is the um our other workloads that need specialized compute, need to scale up and down a lot. Um yeah, they're shaped differently.
8:40 · I think you're building a lot of it alongside the startups, right? They're innovating quite a bit. Um even in your like latest blog post like even even in the series see the the customers that you mentioned here uh the cognitions decagons ramps and whatnot they're they're innovating with you right and [clears throat] that's not something AWS is doing directly with yeah absolutely I think I mean this is again classic we're small team we can move really fast uh our engineers are working with their customers and figuring out my first week at cognition I walked in there was someone wearing a moto shirt I was like what are you doing here.
9:14 · They're like, "Yeah, I just I I am embedded inside of COG." [laughter] Yeah, I think that was Payton. We we sent him over because uh the latency of communication was too high otherwise. [laughter] Um yeah, you know, distributed node. You have to put you have to place one in collocate.
9:30 · So actually, so I had a direct personal experience, right? So I worked on small developer 3 years ago. Uh it was inspired by cloud one. I think you onboarded me at some point like just before and I was like oh like I need some bursty compute like I was just going to try using modto and it was it was pretty pleasant experience. Uh apparently I showed up in the board meeting like the analytics.
9:53 · Yeah, you blew up on hackernews and and we got a big traffic spike. Uh I actually I think the way you you use small developer was modal functions for for running stuff. uh which which was like that was a good use case.
10:08 · Yeah, that was so to me that was protocognition, right?
10:10 · If only I had like stuck to it like that that was like if you just like draw the tech tree out was like yeah like probably this will happen.
10:16 · Yeah, like he was so close to [laughter] um but but the funny story there is at the same time uh we were talking to a bunch of customers who needed something like sandboxing. This is like 2023.
Sandboxes, agent loops, and proto-Cognition
10:30 · Uh so we need a new API right after that. Yeah, like we built sandboxes in May of 2023 before anyone was even knew this was going to be a thing. And the first example we published was uh we took small developer and put in a loop uh so the agent can iterate on itself.
10:46 · Uh loops are hard these days. [laughter] Loops in when was this 2023?
10:52 · Yeah, small chick.
10:54 · Yeah, it's like mid 2023. I mean so I mean those sort of obviously for listeners like the problem was the models are not built for any of this, right? like you you're trying to like they're not postrained to understand like you know looping and like self correction and and tool calling was there but like also not that great.
11:09 · Yeah, I [laughter] don't remember if you use tool calling in this one but yeah the models would just diverge after like 10 iterations and not produce anything meaningful. Yeah, but like then so I mean okay like now talking to myself three years ago the answer is collect all the failures, build benchmark and then collect all the you know examples build the oral environment sell it for like $10 billion to Meta uh and then and then also train a model and then sell that for $60 billion to to Elon and this is a money machine like it's like it's actually not that hard. [laughter] I mean, it's hard to have that kind of inherent conviction that this stuff will get that much better in retrospect. [clears throat] It's so [ __ ] obvious. Like, [laughter] fair enough. Like, what else were we doing back then? I don't know. Um, anyway. Yeah. So, so, so I mean, so that that was the start of your sandboxing journey, right? Uh, I feel like it didn't blow up blow up until like last year.
12:04 · Yeah.
12:05 · So, there was like a couple years of quietness.
12:06 · Exactly.
12:06 · Yeah.
12:08 · very underrated product value like um my experience with modto Charles before he had joined Moto met this guy at a hackathon and he really insisted we wanted to run some small model you know not hosted anywhere and he's like ah there's this cool company modal they'll like spin up a GPU sandbox so we can throw it on there it'll take a hugging face link and you know like there's so much value just right there right like instant hosting spin it up spin it down it'll stay cold but you know we run the demo a few days later it'll come back up and All this stuff in retrospect like is still what we needed like today.
12:41 · Yeah, I mean it's still needed today.
12:44 · Obviously workload shapes have changed a lot as um we we run stuff for people with really massive production scale and uh there it's it's not about scaling from 0 to one but it's how do we scale really elastically uh from like,000 to 1500 GPUs very quickly in in a given region. It's the same shaped problem.
13:04 · Okay.
13:04 · So you look at say cursor composer right they had a we'll do RL on a model every couple hours um you guys have a whole version of RL inference gym and whatnot when you look at workloads like that you're basically doing train runs where you need to scale up scale down every hour thousands of GPUs right um that's the example for we do need it right yeah well actually I'll I'll take a step back and um maybe talk about like how how people use model today uh because our Our biggest use case actually is uh elastic inference and the thing we first found product market fit uh with was inference for custom models. So we kind of stayed away from the LM space. Uh and we were serving companies like Sunumo for audio, runway for video, robotics, uh comp bio companies that train their own model elsewhere. But modal is the best blackbox that for deployment um scaling to however many GPS you need as your traffic pattern changes. And we saw all of them actually um like have a very unpredict predict predictable uh traffic pattern. It's like dial it's um some days like the company will do launch and you know they'll need like u way more and it's not just one model that they deploy they all these companies deploy uh lots of different models in different regions and so the autoscaling problem becomes even harder because then you have to scale within a certain region and those cycles sort of are offset so different times you need to scale up in different regions. So that's like our sort of and that you know that that in and of itself is a huge category. There's a bunch of inference providers which you know provide this fireworks does this as a service together whatnot base 10 um that's kind of carved into its own niche for language models at least right now.
Elastic inference, GPU snapshotting, and 100,000 sandboxes
14:50 · Yeah, I mean the thing that we we have actually specialized in is the autoscaling aspect. uh because we we we found that it's not universally true that everyone else can autoscale and we've gone deeper into it on the tech side but we've incorporated GPU snapshotting into the product so you can actually uh take the GPU state uh like your torch compiler model snapshot it and next cold starts way faster and so going back to your question it's um that's why you need a lot of burstiness for inference but then people also do a lot of on demand training like for RL stuff your rollouts are bursty as you said people also do a lot of batch jobs.
15:25 · So we'll see uh a lot of companies uh before they have a training run, they'll need thousands of GPUs to run encoding or something like that. And I think those things are much more bursty than I agree that agents are are not that bursty. Uh sandboxes are uh except when you're doing RL. RL is insanely bursty.
15:43 · Yeah.
15:44 · Yeah. Like when when you're doing uh rollouts uh you you sometimes need 100,000 sandboxes.
15:52 · Yeah.
15:52 · I'm curious if you've seen early sparks of continual learning. There's some people like our friends engram recently announced this. They're they're trying to do training. That also seems like a different workload, right? If you're doing training 24/7, per se, there's a very weird dynamic of how you're using GPUs between people and whatnot, but seems like something you guys would work for. As you said, we're we're fortunate to work with a number of uh customers at the frontier and grab some of our customers uh and and they are taking the primitives we have um and trying to use them in very interesting ways like continue learning. It's possible as the stuff gets better some of that will be part of uh you know our offering as well if you know more people need it. Um but we're just waiting to see how all this shakes out. Is there a primitive that you added after sandboxing that was the next step in the story?
DeFlash, speculative decoding, and Auto Endpoints
16:46 · I guess we've been going much deeper into LM inference because we realize that some of the advantages we have with like autoscaling again especially in different regions and whatnot uh are um not present elsewhere. And and the place where we had a gap was we weren't uh working on the model layer itself like we were a black box and uh we realized that uh we actually can get to frontier level model performance um uh with you know by having great people who who work on uh all of this and uh we've actually been open sourcing a lot of our work uh in terms of um recently we um shared our work on Dlash which is a block based uh uh speculator and we've open sourced uh all of it. So um you can get by using open source v flash you can get the same performance as you would with one of the proprietary providers. And the next thing we're thinking about here I thought this was actually an interesting blog post as well right like uh I think in here you make a claim that you know not a claim just that how how effective speculative decoding really just gets you anything you want to point out from this around you know what people should know.
17:53 · Uh yeah absolutely I mean the high level summaries um would it help to describe what speculative decoding is? Yes, I think like we so we've covered like ego and all this like hydra and all those things but it was like two years ago I think it doesn't hurt right the speculative decoding is you have a smaller model uh called a draft model predict tokens ahead of the bigger model and then you have the bigger model uh verify all of this all the tokens are predicted and the reason it's faster is if you're predicting uh one token at once you're kind of u bound by memory bandwidth but if you can batch the verification of uh the draft model uh then you're much more efficient using compute and it's faster and as long as your draft model is producing a lot of tokens that can get accepted which is called the accept length you can get u speed up that's multiple times of um you know the original model speed uh and that's what we highlight here it's um like people talk a lot about we made these kernels faster and whatnot but improving kernel only give you like a few percentage points of improvement and increasing accept length literally is a multiplicative decrease in like 2 to 4x without much head-on performance.
19:04 · Yeah.
19:05 · Yeah.
19:05 · I think it maybe I mean you you are running a second model, right? So maybe it's like more expensive in in the compute but I meant quality but yeah I mean I think [laughter] so there's no drop in quality performance because you're always you're never accepting a token that better or same.
19:20 · Exactly. Yeah.
19:21 · Right.
19:21 · Yeah. And so we've been working a bunch on Dlash which is a block based speculator. Uh so it's um uh instead of predicting uh one token at a time it's predicting a block and we we've been open sourcing our work with it. The the next thing for us here is for for helping people train speculators and custom models. Uh it's it's something that traditionally is very FTE driven support deployed engineer driven like you work with customers and help them do that. And our vision for this is why we launched auto endpoints is we want to make frontier level performance available to everyone. And so uh we mentioned this announcement we kind of teased it. The next thing we're we're launching is basically as you run an auto endpoint uh we we shadowed traffic and do you want to explain auto endpoints are high level? Yeah.
20:12 · Yeah.
20:12 · So um this is I guess going back to your modal is is you touch the code but uh sometimes people actually don't want to touch the code and they want to get started with an endpoint that works and has all the great performance and uh scalability that modal has. So we've made that easier with basically way to create an endpoint from our UI from the CLI um that has all of our optimizations that we talked about uh like the deflash stuff already baked in and there's full transparency. So we give you the code uh you can go run it yourself and if you want you can eject out into the full modal experience uh which we see as people get sophisticated they they do want to tweak the models uh they want to fine-tune stuff you you can still do all of that it's it's not a black box and yeah the next thing as we tease later in the post is how do we give you value even beyond this in terms of having your draft models evolve as your data distribution evolves again without having to talk to a person and um yeah I guess just to kind of understand it directly I mean you know obviously you you have the GPUs you have an endpoint that's compatible you serve open model if someone was to do this themselves what's the delta that you guys provide so you do a lot of open- source great work on effective inference um how does it compare to say I take the same model GLM 5.2 2 FB8 take offtheshelf inference engine VLM SG lang um you know get compute of similar capacity similar cost what's the kind of delta that plugging into something this like this offers outside of the benefit of you know scaling it's kind of interesting because we we've taken the approach of open sourcing our contributions and upstreaming them we work closely with the SG lang team we we actually want the improvements that our team uh comes up with to be uh there and openour or for others to use even outside of modal. The benefit to us is we have a team that has significant expertise in terms of if you do have something that is not there our team can help you get that performance uh first. Um the the other thing is with these endpoints we are way more elastic as you said uh than uh anyone else and you have true scaling to zero uh you have true uh burstiness and in practice that matters a lot more to people than just finding uh the GPU and uh running model code on yeah and I I will say it's actually not that straightforward to just like what I said is easier said than done, right?
Production-grade inference beyond raw GPUs
22:41 · Um, it's I think still for the average person still hard to just gut check using different there's there's quite a bit of combinations you can make there. Um, the trade-offs aren't really known at face value.
22:54 · Yeah, I mean it's it's not just that. I think it's it's that running production grade inference is a hard infer problem. uh even if you subtract out the autoscaling is uh controlling things like tail latency and um making sure every uh request is delivered at least once and whatnot.
Background agents, Ramp Inspect, and the agent lifecycle
23:14 · There's a lot of innovation that you can do here. I think uh it's very interesting that you're starting to encroach on like as you become a full cloud, you're starting to encroach on other people's turf.
23:23 · Mhm.
23:23 · [laughter] What will you not do? Well, we want to follow our users and um make sure they uh they get like a platform that has everything that works well together. Uh so right now we're kind of focused on the model life cycle and the agent uh life cycle. Uh so both like going from data prep to training to inference and then also if I want to deploy a background agent let's say you know sandbox to persistent storage a whole bunch of other stuff.
23:53 · We talked to Cole uh who did uh open inspect. Yeah.
23:56 · Yeah. And obviously real inspect also is on modal.
23:59 · Yeah.
23:59 · So ramp inspect was was a a great example of a background agent that was really successful because they uh were able to use some of the primitives like snapshotting and fast scaling to this has something that feels really reactive and uh and works well.
24:16 · Yeah, that's the new CTO of uh RAM right there.
24:19 · Yeah, Rahul. [laughter] It was really really fun. Um yeah, I mean you know I I think uh all very bullish like you know one of my reflections was also I did not originally cuz obviously when I met you guys you weren't that much in the GPU game and now you're all about uh inference and one of the points that I hinged on for Jensen's keynote at GTC this year was what we're calling like the inference inflection right that let's say in AI workloads or machine learning workloads it used to be like let's call 8:1 GPU to CPU and now it's more like 1:1 which is like a interesting like because of how much agents basically are blocked or call out to to CPU heavy stuff the actual like limiting factor like swings back and forth from GPU to CPU a lot more than it used to be all GPU and then occasional CPU GPU CPU and now it's like just constantly and you just have to collocate everything.
Modal’s 17-cloud supercloud strategy
25:22 · Yeah.
25:22 · And and that's one of the things that um actually again we see is uh something appealing about modal which is we've built this capacity pool that spans uh 17 cloud providers. So we're very good at running on various kinds of cloud capacity across the world.
25:39 · You don't have your own data centers.
25:40 · We don't have our own data centers. We we just run across a lot of NeoClouds and providers question.
25:45 · Yeah.
25:45 · Yeah. You're running the math and you're like what's the cut over point where you're like Yeah. It's a good question. I mean part of it is we we see our differentiator in in the software layer and um being capital light and focusing on the software helps us move really fast. Uh so far it's worked out well because there are so many other people building data centers that we're able to work effectively with them and again focus on what makes us uh special.
26:09 · Yeah.
26:10 · 17 gets you into like the local providers sometimes like the [laughter] who's the most interesting one. There actually a lot more Neo clouds than you expect and they all have various degrees of um various levels of reliability and uh that's why something we've invested a lot of time in is actually uh building our own reliability layer on top. Uh so if the GPU falls off the bus or something happens we uh user workloads are not affected and that that actually lets us use a lot more capacity than um you know you as a user would be able to.
26:46 · It's a useful thing to have because like now everyone knows like what layer you are and like you you sort of optimize for being the super cloud of all clouds.
26:56 · Yeah, that's that's uh that's the idea.
26:58 · Uh and so I guess when you mentioned collocation that's that's another interesting thing where um one thing we've seen is people come to us when they want uh very specifically located uh CPUs or GPUs uh like they want oh the pin it in like EU exactly or EU US data locality thing or performance or what it's either data locality or latency.
27:21 · Yeah, like you want your they're running sandboxes in modal. They want them to be right next to that's a you know that is important in in in all those things and so like you've kind of accidentally I don't know if it's accident but like you've built the perfect primitive for agents to to express themselves and then you know like it's almost very funny how every extra development just involves more file system just involves more CPU uh just like the things that you already have. I don't know much about if there's any like networking usages that are interesting, but you've also done some good work on networking.
Networked sandboxes, private IPv6, and RDMA
27:54 · Yeah, I mean that's exactly right. Like we we're sort of just taking compute, storage, and networking and building stuff on that layer uh for again the stuff people need.
28:04 · We we see a few interesting network things coming up. Um one is people actually want sandboxes. Uh so we have a for like a Docker cluster type thing.
28:13 · Sorry. Uh Docker swarm. [ __ ] So what is it called?
28:16 · Compose.
28:17 · Compose type thing.
28:19 · Yeah.
28:19 · So actually if you want docker compose uh our sandboxes now support uh this thing called sidecar. So you can a sandbox is actually a pod of containers and you can run multiple containers in uh the sandbox. Uh also useful because uh going back to networking people want a lot of control over um outbound networking from a sandbox. might want to run man in the middle proxy for like maybe logging stuff for RL or um controlling how egress can happen to a domain injecting credentials. Uh and yeah, so we've kind of had to build a lot of that stuff ourselves.
28:52 · Yeah.
28:53 · Uh but then also sometimes people actually want um sandboxes spanning multiple nodes to talk to each other. Uh which is an emerging thing we're seeing.
29:01 · We have support for that for for a different reason and yeah we'll see if that becomes safe like just a open socket it's a this is directly like NTLS we do support that which is you can uh expose a tunnel inside a sandbox and then you can either expose public internet or it can be um mine you can add like a HTTP uh odd layer above it but we have this thing called I6PN which we haven't talked about uh which is this like overlay network using IPv6 addresses. Uh so if modal containers uh within the same workspace uh when this enabled can actually address each other using this private IP v6 address uh and no one else can uh it's like sort of private networking uh for containers. We actually built it because we needed it as a primitive for our distributed training product. Uh so we have this other feature which is you can add a decorator to a function and you get a cluster of GPUs uh and they have RDA networking uh so you can run a dist training job uh that's truly serverless uh and we need the overlay network for that but then we've seen that people are using it for other reasons and I'm kind of intrigued to see [laughter] yeah what would people do with it build primitives and let people figure it out right [clears throat] they're like they read the docs we pick let me use that for [laughter] something that you never intended This is literally not even in our docs page. People somehow found it and they're using it.
30:27 · Uh I mean the way you portrayed it with like RDMA versus TCP like very well laid out, but just the transfer speed change at scale for RL like Yeah, you have it you have it built in.
30:40 · I'm sure someone found it found it to be a lot more efficient before you actually made a thing out of it, right?
30:46 · Yeah.
30:46 · And and not to split hairs, I guess the the overlay network actually is the TCP overlay network. Uh the reason we have that is you need that to do the key exchange for RDMA before you set up the RDMA network on top of that. But then people found the the TCP part.
31:02 · Can I tell you this is like a big aha moment for me because so I you know I reviewed 2,200 submissions for for the world's fair. Yeah.
31:11 · And then I got this from John Austerhalt who I don't know if do you know John Osterhalt by name. He published he's a well-known professor published a lot of interesting software design books and this is the talk he chose to submit it's on t it's on RDMA at imprint and I'm like you wouldn't think that this guy who is like kind of operating systems guy would care about RDMA [laughter] I I mean it makes sense to me because cloud right yeah like like the way you move around your KV cache and how efficiently you can do it how efficiently you move um your weights from your training GPUs to your inference GPUs and RL is there's a lot of degrees of freedom and it is basically a systems problem of moving memory around scheduling this shows you how primitive my understanding of networking stuff is is this like the domain of wire guard as well not quite so adjacent um so explain everything sure [laughter] sure uh how do we move memory around GPUs [gasps] oh sorry yeah that is memory sorry I I was talking more and maybe I was talking like 5 minutes back about the the private IPv6 uh addressing that you've set up is it's basically a VPN.
32:25 · Yeah, it's sort of like a VPN and yeah, WireGuard is is uh Yeah, you're right. It is it is um yeah, you already moved on to topics a similar in the same space wire guard is uh encrypted and and this is uh you don't need encryption.
32:38 · Yes, it's not encrypted. Uh that's the main difference. This is TCP and we have EBPF programs that will reject or allow the u the TCP connection based on whether you're allowed to do it. Used to involve a fullsize car but now you have EBPF in the Linux kernel.
32:53 · Yeah.
32:53 · Yeah. I don't know if this is a natural follow on to the topic of like my skepticism on distributed training is that well like people spend a lot of money on like cables to hook up GPUs and even that is not like fast enough and that's the bottleneck. Is your networking fast enough?
33:14 · Yeah.
33:14 · So I guess you're talking about sort of fully distributed training like a dialoc or something which is like cross that would be yes that's the extreme.
33:22 · Yeah. you're kind of in the middle and then other people would have like the Melanox cables up in in like their actual data center. when you run multeno training on on modal uh the RDMA uh I think Melanox uh is or infiniban is like a uh is you also use RDMA uh but basically it's a way to bypass the TCP networking stack and um transfer uh stuff much faster uh between one node uh to the other and and we we have I think like 3 terb per second uh internal networking which is the standard that's needed okay so I misunderstood what part of the stack of TCP.
34:00 · Okay. [music] Yeah. I mean, very impressive work. [laughter] So, effectively you're you're extending sort of like the modal philosophy to the the trading cluster like Yeah.
Multi-node training, post-training, and auto research
34:14 · Yeah.
34:14 · And we're not going for obviously like large scale pre-training runs. Uh the thing that we've built multiode training for is uh we see a lot of u smaller scale post-training. uh like uh people are post training like mediumsiz quen models uh so they can uh get higher quality on inference uh this is a perfect fit uh for something like that.
34:35 · Yeah, that is my impression of how a lot of these labs explore branches in post training and then eventually merge whatever they find in yeah the the other use case we've seen for mult training is even if you have a big cluster your researchers is still doing small runs and yes having elasticity there matters a lot more like this is actually like the current limiting factor for auto research which is like you you basically need to give your model some GPUs We have a blog post on auto research and modal is uh yeah like uh turns out to be a pretty good substrate for that.
35:13 · So my impression is auto research means many things like if anything coins right now still science fair right like not not not actually like I don't know how many people actually doing this. Yeah I taught the same thing.
35:25 · Yeah
35:25 · you would know we like our internal both training and inference teams actually use this sort of the general shape of this quite a bit. uh like we have this one internal repo called auto inference which is essentially we've automated our own FTE efforts using uh this harness which is um the agent will just spin up a sweep of different things. It'll even run like um Nvidia inside profiler and it'll like tweak configs and it'll arrive the right thing. Uh it'll change your GPUs from H200 to B200 and actually works really well.
36:01 · Nice. So, by the way, I enjoy that your FDE is so technical that you have to do these things. [laughter] It's very different from FD from other people.
36:08 · Yeah. Yeah. For our FDA team is uh essentially they're like applied inference researchers or applied training researchers.
36:16 · Someone told me like they have to be able to build but they also have to be able to sell. Do they have to sell or are they like they're good. This is like post sale type of thing.
36:24 · It does. Being able to talk to a customer and engage effectively with them matters a lot. the same thing, you know, but it's it's not really a sort of sales uh thing. We we pair them with we have solutions architects as well that are more on the pre-sales side.
36:38 · Okay, let let's spend a bit more time on auto research. This is a big focus for me for for for this year. Where does this go? You know, like um have have people explored enough like you know there's all these beautiful charts of like improve improve then it sort of level off a bit and then you find the next thing. Is this basically sort of one abstraction up from normal training?
36:57 · Is that how we think about it or do you think about it differently like model level training versus basically like AIdriven hyperparameter search?
37:05 · Uh some some people call it like neural architecture search or whatever, right?
37:08 · Like yeah, I mean I so the the stuff I've seen people do with it is nowhere on the architecture level. It's pretty much tweaking parameters, but it's it's basically a hyperparameter sweep that's guided by some sort of model intuition. So it's like much more efficient than um whatever other sweeper you would have.
37:26 · Uh yeah, I mean, you know, it's just it's just a question of where you want to spend your computer, you know, right?
37:30 · Cuz yeah, you can just throw infinite amounts of money on this and somehow you'll bang on Shakespeare, you know. [laughter] Yeah. Internet monkey.
37:38 · Uh yeah, I mean so like very good for modal uh and I think it's also very important that agents can spin up other agents. They can spin up their own infrastructure like very like very good for you. How good are LM at generating modal code? Like you know like the benefit of existing prel is that you are in the data. Yeah, they're they're actually surprisingly good. Uh I think like pre-Cloud 4 they were not and then now they're able to oneshot uh stuff out of the box. We're playing around with releasing like a modal bench for like the harder things uh that the LMS cannot do yet and and maybe what's an example of that? I think the the things that sometimes agents struggle with um without right guidance and a skill is um how to use the rest of our observability like how to something is failing like how do you look at the the logs and then update the right thing. It's sort of reasoning about that but they're able to oneot like yeah you can't just add a skill to it.
38:40 · Yeah.
38:40 · So, so we have a model skill now that which is kind of actually why we built this model bench. It's to find things like that so we can we we can address them in our tuning skill. Yeah.
Compute strategy, capacity planning, and batch tiers
38:50 · No, no, I mean it's good. Uh are you facing any shortages? Uh you know like we talk a lot about GPU shortages but also CPU also memory.
38:58 · Yeah.
39:00 · We have had a lot of growth which means that uh there's a we've had to be much better about proactive capacity planning.
39:09 · Yeah.
39:09 · Uh so we we have a which by the way like it's like an MBA's like dream job. It's like just planning this stuff. I think last time you and I talked about something maybe about this.
39:17 · Yeah.
39:17 · I mean we have a really competent team of people that we call um the roles called comput strategy. Uh so yeah if anyone listening here [laughter] wants to work on computer strategy. Yeah. Yeah.
39:28 · I like I mean the nor the normies call it FPNA or something.
39:32 · Well it's more it's it's not FPNA. It's it's um there's a lot of interesting financial questions of like uh like what is the blend between one year and three year reservations? How do we forecast our own capacity? Uh how do we basically especially since our capacity is very funible across different GPU types and different regions. Uh like you basically have to model a lot of it. Uh and you also have to have an opinion on how the supply chain is going to evolve and then you have to like take bets uh based on that. tokconomics.
40:05 · Yeah, this is like probably not a real point, but uh I was trying to think about like what other industries I was trying to think about like you know we cannot be first to like these kinds of problems and what other industries have had this and I was like airlines with with fuel and like they have to hedge their fuel and like I think for a long time Southwest because they made like a hero fuel bet they like were like super low cost because compared to everyone else.
40:28 · [laughter] Yeah, I had thought about that.
40:30 · We're having a fun time too, you know.
40:32 · Yeah, it's a lot of the the compute business in general uh for us is also about being very good about capacity management. That is how you have great unit economics uh but also over times how you can unlock more value for customers like um one of the things we're building now is like a way for customers to get um if they don't care about latency uh like get much cheaper pricing and they'll get results back in like next 24 hours or something uh like a batch tier essentially. Yeah. And those are levers we have because we control the whole stack and scheduling and whatnot to give people a sufficient Yeah. I feel like they're not as popular like those like the the frontier labs have all those APIs. They're not as popular as they should be.
41:14 · The demand that we see for something like that is actually not for LMS. Uh although sometimes people want to run evals and do synthetic data prep in and there it makes sense.
41:24 · Okay.
41:24 · Uh but it's uh from a lot of nonLM companies uh like people who are doing computational bio like they they haven't run really big batch jobs and they don't care about when they get it back.
41:36 · Yeah.
41:36 · And like they have a reasonable it's it's also like a cousin to the stopping problem of like will this finish in time. [laughter] Yeah. You you can you can bound it like you can give people on it.
41:49 · Yeah.
41:49 · Yeah. I think what's what's interesting is like the the next phase of modal like what you know do people expect from you uh now that you're sort of established and you're like well-known computer player among all these leading companies. You had an inference launch week and we talked a little bit about the launches like what else what else should people know?
Open models, real-time AI, and production agent infra
42:09 · We are building primitives that make our our users lives much easier. So um I think for example with LM inference thousands more companies are going to post train their own models and deploy open source models for inference. So we're thinking a lot about what is the best product shape for that and uh that involves everything from our training gym to uh then uh endpoints that get frontier level performance. um you know again without having to talk to anyone it looks somewhat different on other verticals like uh we're also seeing a lot of real time uh audio video stuff in there um uh which is why like uh we're working on things like regional routing uh with with fallbacks so you can you can get sort of GPUs that are as close to users as possible uh so you you get like low latency for video streaming and and whatnot and then on the agent side it's Um, we're still sort of working very closely with our customers because stuff is changing so fast in terms of what they need and uh I think beyond sandboxes and persistent file systems uh there's a lot of other things people need from this agent stack as they build production agents. So yeah, we're thinking about those other things that fit in there.
43:27 · I want to ask what what the other things are.
43:29 · Yeah, [laughter] probably share right now. I think okay so uh I do think a lot about the principal components of cloud and you do talk about comput storage networking because so far for me it's fine uh so far for I mean the first couple generations of cloud it's fine what's different qualitatively different about agents that you you need some new permission level like a lot of people you know obviously okay and I'll just kind of spew tokens at you until it like hopefully sparks something like the new level now is whatever cloud code does which is dangerously skip permissions or like allow list by command or like whatever, right? And and sometimes they're like, well, okay, well, we have like this adaptive thinking mode where like just just trust me, bro. I I will make the calls for you. Is that it? You know, like basically like sort of LM mediated permission.
44:17 · Now you're looping it with a goal and flooding row.
Hard guardrails, managed agents, and specialized sandboxes
44:20 · Yeah. I mean, I'm I'm skeptical of LM mediated permission for stuff that is at the sandbox level because you do want hard boundaries. Yeah, obviously someone can exfiltrate stuff.
44:34 · But like maybe maybe that's old school thinking. Maybe we're the dinosaurs.
44:37 · Maybe the AIOS or the LLM OS is really the kernel is a goddamn LM. [laughter] Like it makes you feel uncomfortable, but that's what frosting the LM is. Like imagine a spherical cow perfect LLM, right?
44:52 · Let it [laughter] maybe I want to test the boundaries, right?
44:56 · Obviously, I don't believe that, but I want to see where I'm that's the non-conensus.
45:03 · Yeah.
45:03 · I mean, I think you always need hard guard rails when you want um and you can pair those with softer guardrails, right? And ask a deal and mediate it.
45:14 · Yeah.
45:14 · I get end with a couple of your commentary on like the ecosystem outside of Moto. Manage agents. Everyone has one. Gemini OpenAI cloud uh very useful for you but also like it is their way of starting to edge into your space.
45:31 · Yeah.
45:31 · Uh what's going on?
45:33 · Yeah.
45:33 · I mean, we're um very excited to partner with Enthropic and some of the other foundation labs belting is a great place to start if you're starting out building an agent and uh but but then when you get to um building something more production grade like you're a company that's like RAMP that's building their own uh RAM also runs their accounting agent on us. of their external facing agent. You need a lot more control over uh your compute primitive on things like um what sort of how do you process different files that the agent has access to and how do you snapshot and restore how do you control the networking um maybe you want GPUs when you get to that point you kind of want um specialized sandbox provider uh that gives you those things and that's the role that we are trying to play. We don't really have an opinion on the harness whether it runs in it's a cloud manage agent and you hook it up to modal sandbox you run the harness in modal sandbox. We'll see where people converge with that.
46:40 · Yeah. Do you any opinions on like the meta harnesses? Um it just another layer on top of these things.
46:46 · You mean like the open pie and open pie is one. Um I think Versell had one which I can't remember the name of right now. Fred shot had one. Uh and then uh to me most recently was data data bricks that had Omnien. All these are sort of meta artists like kind of pseudo agent cloud type things.
47:04 · I personally have not played around with them and built into that.
47:08 · I mean everything's bullish modal as long as it consumes more infra. That's why we're focusing on the infra layer. It's um somewhere where our um relative competences and um also it's a hard problem to solve.
Why AI made infrastructure exciting again
47:21 · Yeah.
47:21 · I mean I will say like just generally reflecting on I don't know if you if if there's other topics on model but like just generally reflecting as an infra person not as intense as you but in that field this has like been the most exciting time in infra like it was boring actually for for for a while and you couldn't really get people excited about data infrastructure like Eric would get on data council everyone just watched the video and like it's like look at how many sandboxes I spin up and no one gave a crap. [laughter] Yeah.
47:54 · And like now everyone gives a crap.
47:56 · That's true. It is a very exciting time and I think a lot of that's driven by just uh the amount of scale all of this stuff needs.
48:04 · I think like a lot of your initiatives, a lot of your like product directions make sense in retrospect, which is like the best kind, but I wouldn't necessarily have thought about it myself which we need we need the predictions, you know? I mean, I think there's a lot that you just don't even see, right? like you have the batch, you have the voice, you have the multimodel. Um but what what else you know what else is coming up for us or where do you see things going?
48:27 · Yeah, I mean in general it's it's clear that there's a obviously there's a huge shift happening. I I think one thing that's not as obvious to people because LM inference gets talked about so much and is also we work a lot of companies that are doing things like drug discovery and computational bio like the chai discoveries world big things are probably going to happen there uh we work a lot of robotics companies that are actually putting robots in in like active deployments and getting good results out of them. Is there air gap modal? Is there is there a version that is like onrem airgapped whatever?
49:04 · No, we we cloud only. Yeah. Okay. But yeah, I mean uh so what you're saying is like because you're focused on primitives and they're good primitives, you find use cases and all these kinds of things. Actually probably diversifies you a little bit away from LM all the time.
49:19 · Yeah, absolutely. We our our goal isn't to only serve the L1 fist market just on the website. The audio the bolts on the bio images. Yeah. I mean there's a lot here. There's QITTS customizing. Oh, chatter box.
49:35 · Um you know there was a customizing whisper.
49:38 · Yeah. Yeah.
49:39 · This screen reminds me of a fallen competitor. Uh which replicate.
Model APIs, differentiated products, and agentic video
49:44 · Mhm.
49:45 · Um what's your postmortem on what happened? This is one thing we've kind of stayed away from is providing an API for models because I think providing model APIs is some of it ends up serving like a really hobbyist market which is much less sticky and we've always wanted to build for companies that are building sort of products and need sort of more flexibility that's not just an API and which you you can build an API for a model and this is clearly what it is but you can but you're saying you can wrap it into more fully functioning back end that you that you run.
50:21 · Yeah. So, actually all of our examples, it's not that spin up this model, here's an API token, use it. They're actually all code.
50:27 · Okay.
50:27 · And so, the point is that this is an example starter code.
50:31 · Yeah. But you can you can tweak it however you want. And if you're like a company building a product like computational bio, whatnot. Yeah.
50:40 · I guess I'm trying to tease out for listeners. Yeah. when does it stop becoming oh you're just an API call and you're just a wrapper on an API to becoming what you call a product right like what is that layer like what like you know obviously more lines of code but like beyond that what what is the substance that people add that qualifies it to be something more I think there's a little bit of like a selection effect of like a lot of companies who do want to get deeper into that level are probably building something that's more differentiated and um I think Um an example is like we with LM and friends originally we uh worked with companies that were building their own post- training frameworks or they were uh ramp actually early in the day was training their own tokenizer and like swapping out the tokenizer in lava and whatnot.
51:30 · I'm not saying that's that was successful um in that case.
51:34 · A better example is like um like let's say u because um does not use for training Mikey on a pod. Yeah. But they they use modal for all their inference and that's because they have like a custom they have completely custom model architecture and that means that they have to be at the code level and tweak things that are not um you know it's an API. It's interesting as well like we had Ethan most recently on the XAI Grog team make a prediction that actually like the next tier in videog model it's a better model or agent that orchestrates video models.
52:11 · language model backbone that can use tools and write code like yes I can make my sixcond video or my 10-second video from grock but actually I want my 6 minute video and I'm not going there through normal video gen yeah that's interesting I actually so we have GPU sandboxes and recently have seen a few companies doing sort of agents that do video manipulation or give it f like that's not you need to give it Adobe.
52:41 · Yeah.
52:41 · I hadn't put it together with like it would actually be a video production thing. Uh in in my mind, these things were going more towards editing and Yeah. I think about this a lot obviously. [laughter] Sorry.
52:55 · Um Luma Luma agent is a version of this for video production, but you know, it's a one-off.
53:00 · I was going to get your quick takes on uh on some other stuff that happens in recent news and see if you have anything interesting. GitPod very like somewhat like you know different market they they're in like sort of like the CI/CD market but actually technically very impressive. I don't know if you've like taken a real look at them.
CI, coding-agent infra, SDKs, and Modal Bench
53:17 · Yeah. Um we've um people on our team have talked to the Kipot team and they've they're technically very strong.
53:24 · Yeah.
53:24 · I I actually am we're very bullish at modal on the CI market as well because as there's more agents uh coding agents they're going to run a lot more CI and the primitives there can be much better.
53:37 · I think there's a lot of wasted CI.
53:39 · Yeah.
53:40 · So is it just like let's filter like what what is the highest order bit here in improving CI for agents? Well, there there's a lot of wasted time in CI on like preparing your artifacts and like you know getting you to the basically um preparing your dependencies and whatnot and um obviously like build systems help with that but like if you have primitives that are like memory snapshot and restore can you just run CI more efficiently?
54:10 · Oh oh okay okay interesting. Yeah. I mean another form of like you know on demand compute.
54:16 · Yeah. Exactly.
54:17 · Yeah.
54:17 · Uh it it needs the same again platform.
54:20 · Yeah.
54:20 · So so uh for those who don't know GitPod rebranded to owner. There was like there was this whole thing. I I got actually I I like sort of semi sounded the alarm at cognition. I was like you should take these guys seriously because they're infred very good.
54:31 · Yeah.
54:31 · uh and but you know then then they join OpenAI and uh presumably we'll we'll see codeex cloud from the owner team like which which I think would be very very strong to me like teams like that that can set up the networking and like the the secure boundaries for like and your like agents to have their own cloud each effectively is what you're doing kind of and I'm just trying to draw the analogy or or the differences if you have studied them like what is the philosophical difference you know My sense is maybe they didn't go after the right market at the right time because we I guess also got lucky with like aentic use case is really taking off and um dating um like more of like a sandbox shaped thing than like u uh my setting is yeah sandboxes work is sandbox [laughter] it's just like buildtime sandboxes versus runtime sandboxes and actually it turned out runtime was better right and the difference there is runtime sandboxes have a different configuration surface of like how you configure images, how you like attach like storage and yeah it's it's fascinating other people Astro also openai also like Python tooling ecosystem people are you still sort of bullish build building on top of Python also recently modular also got bought by by Qualcomm just any any of your takes there yeah I mean we we had Python as our first SDK language anguage because that was the language that people did data and ML in. I actually now have go and typescript SDKs as well and our runtime is completely language. It isn't in Rust but it's it's not tied to Python by any means. We haven't seen I think with like inference and training stuff people are still very Python and the interesting thing with like the agent stuff is people use our TypeScript SDK a lot more because they're not actually doing anything than use ML. I don't think we'll have to go beyond that super soon because Python and TypeScript is still dominant.
56:34 · The last two languages in the world.
56:35 · Yeah, that's [laughter] it.
56:37 · Well, English and prompting is English and prompting. I occasionally talk to people who try to build new languages. They're like uh even um what's it say Brett Taylor uh who's chairman of open like we we need we need a new language for for LM so no one has come across one and I keep looking you know Python and Typescript are you have a lot of data plus but then also they are very imperfect as just as languages themselves then my close is I think modal used to be a big bet on developer experience and you've pivoted the team to agent experience is it like the way now like do you do Can entire companies and unicorns, multiunicorns be built on just having better agent experience? Do you need something else?
57:19 · It's a big part of our identity. Uh it's not just the, you know, like the uh the very tactical how does an agent use the CLI? But it's also how easy is it to spin something up? Like what is your iteration time when you want to spin up a new service and uh you want to get something going in prod. uh in practice that matters a lot uh to people and uh I think it will continue to matter like uh people are building stuff even faster and if you give them ways to do it quickly and not have overhead then I think the debate for me has been do you do anything differently that is like very fundamentally different for developer experience versus agent experience you seem to be on the side of they're they they're like this they're like co have a blog post on that cosign similarity on like 0.9 or whatever Yeah, I mean pretty much it's the um the main shift for us has been as I said like we built this uh benchmark modal bench uh to see where agents are lacking and yeah actually literally add surface area to a product if if they're reaching for something like maybe this should just be a CLI they yeah they hallucinate their own features yeah and sometimes it makes sense like if they're reaching for this thing it's product feedback like give it to them and then yeah actually moving we used to only have like logs and metrics in our UI just moving all those things with CLI as well so they're accessible in in in that form.
58:40 · Simple as that.
Closing Thoughts
58:42 · Cool. Thank you so much. Yeah, this is a great update and uh I can see why you guys have succeeded so much. Uh it it is really focus but also really good execution.
58:53 · Thanks. I mean we have a long way to go.
58:55 · All right. Thank you. Cool.
58:59 · [music] [music]