Transcript
Cold open and intros
0:00 · So we set out looking for interesting examples and one of them was like weather modeling because the weather data is open source and so given that the data was there we were like okay let's just go try it right and that's the beauty of it whenever data is available it's really good news but a lot of weather scientists did caution us back then this was back in 21 and they
0:24 · said no no this is so difficult you know there have been like decades of like development in traditional weather forecasting and that's very careful bottom-up physics-based modeling right so assuming oh this is the fluid dynamics can you go predict the
0:40 · weather the next day and so on and so that's how a lot of the thinking was that AI is just not going to be able to beat the decades of work in weather modeling but to our surprise we just went ahead we trained them we used neural operators to be able to effectively capture the phenomena and then to our surprise we
1:00 · found that it's not only you know accurate it's almost as close to what the traditional weather models can do accurately but also tens of thousands of times faster so what would take a big supercomputer to run can now be run and we only needed a consumer grade like GPU like you know it was a small model it fit very well it's very fast and it's accurate and I think that just changed everybody's thinking
1:29 · welcome to lane space this is the AI for science section of lane space I'm Brandon I work on RNA therapeutics using AI at atomic AI I'm joined by my co-host RJ Honakee who develops spatial transcriptomics and is the CTO and founder of Miraomics today we're excited to be joined by Anima Anankumar the Brin professor
1:49 · of mathematics and computer science at Caltech Anima has done all sorts of really cool work combining AI with basically models of the physical world and has a really diverse background I don't think I could even remotely cover it but anyway I'll let Anima introduce ourselves thank you for coming on the show yeah yeah thank you Brandon and RJ it's a pleasure to be there
2:14 · and I really like the term latent space because very much figures in a lot of my work because it's really you know the world is latent but yeah just as a brief introduction you know I've been working in AI for more than two decades in a way you know before even deep learning when a lot of the theoretical foundations had to be built for probabilistic models I worked on
2:39 · them and then as deep learning started taking off I also had a foot until recently so I was at NVDI led AI research there and before that at Amazon Web Services helped found the cloud AI team and built the first cloud for AI products back almost a decade ago so you know like kind of having this one foot in industry in academia I think has given me a lot of interesting
3:05 · perspective of how to bring theory and practice together and think of AI at large scale but also AI that is principled a lot of your work has been related to you know the modeling of physical systems using certain types of physical systems which you model with differential equations and you help model them with using machine learning so
3:30 · maybe first let's go in and talk a little bit about that as a high level but we'll get to kind of the details about the single role operators and some of the applications like weather later but first I'm actually really curious to hear about TorchLean and how this recent work you've been doing connects with that larger research program.
Anima's thesis: AI for science, not language models
3:48 · To me broadly like you know my thesis is AI and science how we bring that together right so you know when I started at Caltech almost a decade ago that's when you know my passion was always science was physics and but you know I was doing AI so how to bring that together was where you know the first kind of foundations got laid there and to me like you know there are several aspects to that one is people have been thinking how to
4:21 · use language models for science yes you can do a lot of hypothesis generation you can have ideas but ideas are not enough right so you can have a lot of ideas the bottleneck is going testing and verifying that they work in the real world and so this aspect is where a lot of my recent focus has been on how
4:44 · do we ensure that we can build AI that has guarantees that it will work in the physical world or any aspects in scientific domains and one way to think about it is you know can we model and keep the physics correct and that's where neural operators come in the other aspect is can we verify symbolically certain aspects for instance you know if we claim that the theorem is correct we have to go verify that you know that's where lean as
5:17 · a formal language can be useful for verification so how do we bring that together with language models is where a lot of mathematical reasoning has been at the forefront and so torch lean kind of is in that realm where we say you know not only that you want to verify mathematical statements you may want to verify what neural networks themselves claim to deliver you know
5:45 · for instance if you are now using a neural network and you want to ask whether it's going to be robust say you want to use a neural network in a control loop you want to control you know whether it's a drone whether it's a nuclear reactor so all of this ultimately when we build AI systems with deep learning into
6:08 · control loops we want robustness and so now torch lean can help us do those verifications seamlessly so we can now have neural networks be part of the verification loop and have confidence that we can use them appropriately we have already discussed on the podcast lean and and everyone should probably be familiar with neural networks how but neural networks seem very unconstrained what kinds of proofs are you talking about are you talking about bounds on the outputs inputs
6:41 · what what can you prove with torch thing yeah so torch lean is a noble framework right so what it really enables is that you can now write neural networks essentially in lean so instead of writing in like pytorch it's like a pytorch like abstraction but you can like kind of you know write it in lean and so it can be fully formalized in lean and then there are several implementations you know we have algorithms for certified robustness like crown you know those are implemented
TorchLean: proving what a neural network will do
7:13 · under this framework so sorry was it like what like crown crown crown is one of the can we explain it a little bit so there are different ways to bound you know for certified robustness you know how tight those bounds can be it depends on the relaxation techniques and sort of without going into those there's many such algorithms but you know we are kind of like implementing them and enabling them in lean so we can seamlessly run both you
7:42 · know we both first kind of write down not torch like framework neural networks very simply right and then we can also make statements about them formally and verify them so all of that can be brought together in one framework so what's an example of a a bound that you could claim like so we're operating a nuclear reactor we don't want it to melt down what are the sort of guarantees that you could provide to the out inputs and outputs that would help that not know yeah I mean the natural one is this certified robustness that I
8:16 · mentioned so saying that if you are inputs are you know put up by a certain amount how much is the output going to be put up right this sensitivity analysis is another term and so having those kinds of bounds for different architectures so you kind of
8:33 · automatically get those bounds can then help us you know not only train neural networks to do well in a control loop but also worry about safety and robustness stability these are all part of control systems that people worry about so that's one example of an application so it's really more broadly the idea is you need verification uh in lots of scenarios that involve neural networks so control loops are one another example is you know we
9:04 · used physics inform neural networks to say solve partial differential equations or come up with systems that that are guaranteed to satisfy certain physical laws but we also want to verify for instance that our neural network is only trained in finite precision right so can we overcome those requirements and what happens when we are what are the shortcomings because we are using this finite precision can we also bound those all right so those are other kinds of bounds that work in torch lean
9:38 · so all aspects of like you know the effect of precision the effect of perturbation all of these we can you know we can have algorithms that are implemented in lean that can be seamlessly now part of the verification loop so and is the descriptive power of the torch lean is that sufficient to describe basically any neural network or is there are there constraints on that yeah so it's essentially a you know pytorch like in a framework right so you can just kind of nicely define neural layers in the
10:12 · same way but the back end having like lean now helps us formalize and prove it and for like transformer architecture for example is it reasonable to prove these kinds of bounds on a very large neural network so the you know there is the aspect of one is like kind of having the framework right the other is scalability low so lead still has a lot of shortcomings there it's cpu based and you know it's
10:39 · not like getting that onto the gpu has a lot of nuances there so you know a lot of work needs to be done so what we've started with is a framework you know making that more efficient especially at a very large scale requires still a lot of work to
10:56 · be done yeah but that's true broadly for lean as well so i'm just trying to understand like how i imagine this so if i were to take a numerical analysis classes you know on graduate level numerical analysis class you have a differential you have some discretization error or something and you bound like given
11:15 · these properties i can bound the solution right so solving some of these physics based or ai-based solutions to differential equations i think historically has been kind of the while less so i think you mentioned physics inspired neural networks um really cool idea it'd be fun to talk about that a little bit but
11:33 · i know that sometimes they they are particular and that people don't they don't always work and i think people don't always know when they will or won't work um i mean i'm not an expert but i'm just wondering if that's i've been your experience um and what i'm wondering is like has this helped you understand like the domain of applicability for pins or and is that sort of
11:56 · like the goal is like you can rigorously say like this solution will converge or is there not necessarily the same concepts of convergence in the controlled way for neural networks yeah so you know like physics form neural nets are about like saying that you know i write down like a pde uh partial differential equations and hopefully the optimization succeeds
Why physics-informed nets fail, and how neural operators fix it
12:19 · and i get the answer right and of course if like optimization was not at all an issue this would be universal you solve everything you know we're all happy but that's not the case and so optimization ends up being usually very difficult especially for problems that are time dependent meaning it's not just stationary you also have time and the time component in many cases could be turbulent like in the case of fluid dynamics you
12:50 · know you kind of like if you run it long enough it can become chaotic so you really you know have like very small fine scale effects matter and so in those cases just trying to solve a partial differential equation at all times is just hopeless like you know this is not an optimization landscape that you know i think we'll you know we can have any handle on and this is where
13:16 · the idea that from scratch we would be able to solve these equations using a neural net is not possible so pins don't work everywhere and our idea of neural operators came as a way to overcome this right so saying you know we can't rely just on physics constraints alone to come up with answers we have lots of data available you know i'll talk about the weather example
13:42 · where we even collect data right so we don't just solve equations and have synthetic data but we also have real data by observing the weather as one example so why not make use of all of the data available so we don't just rely on trying to solve partial differential equations and other physical problems from scratch because it's really the data-driven approach that makes it possible to get quick answers and so with neural operators we
14:13 · can bring both of them together we can have all the data that's available we can utilize it we can add physical constraints and then that overcomes pins face can you give a little bit more intuition on the difference there and why that is possible so i heard you mentioned you know in with pins you're basically just baking the physics constraints into the neural network but that this becomes unstable over time
14:39 · or other other variables whereas if you add a little bit of data like i can kind of intuitively understand why that might help but can you give a little intuition for what's going on what's the difference here so the pin like you know every instance of an equation you solve
14:56 · from scratch right at least in the classical sense so you start you take the specification of what equation you want to solve when you hope that the optimization landscape succeeds which many cases it doesn't whereas with the neural operators what we do is we you know have lots of data so we have a training phase we teach it how to come up with solution for different instances of equation and so just as in other supervised learning at test time you can now ask you know can you come up with an answer and
15:30 · you can still have physics constraints as a way to guide that so you know it can be both data driven and physics informed together but the benefit is because we have data you know it's like you're not stuck in an optimization landscape right so you know what the answers are during so you're now at a better chance to come up with the right answers even at this time my understanding is it a neural operator is is a function fit to
15:59 · to uh like to data or a neural network you know learns to fit functions to data is that a good intuition here yeah so you know neural operators are in that sense similar to you know it's the same as neural networks right you're learning on data but the difference is neural operators are you can think
What a neural operator actually is
16:21 · of it as a generalization of neural networks so with standard neural networks the inputs and outputs are of fixed sized so in language we have fixed vocabulary we fix what the input and output are and same with images in computer vision in videos we assume a fixed resolution and we always you know our inputs and
16:42 · outputs always at that fixed resolution we can't change it post-hoc whereas with a lot of this physical data the idea is our world is inherently multi-scale so you should not be like deciding beforehand what the resolution is you know maybe you have like weather data available only at coarse resolution but really the actual phenomena is happening at a finer scale right and maybe you want to after that incorporate either additional data at finer resolution or add in physical constraints at finer
17:15 · resolution so we should be having that flexibility and we should really think of the world not at these fixed resolution but one that's happening uh infinitely you know that one the real world happens at that infinite resolution and that's what neural operators enable because they model inputs and outputs as continuous functions that can be infinitely resolved that can have infinite discretization and now we can have you know at inference time you can give it now inputs and ask for outputs at
17:48 · any resolution so you're not just limited to the resolution of training that we see in standard neural networks and that's what neural operators enable so neural operators enable us to zoom in and out as we like so obviously that as stated that any function that that's under
18:06 · constraint right you could have many many functions that fit this fit the data it'll be easy to overfit so how do you regularize that yeah so certainly like you know you know if you're asking about making predictions at a higher than what's seen like what we call zero short super resolution you're kind of making some guesses right and and that's what these models are doing they're trying to regularize and kind of smoothly extend to higher resolution but of course if you now give it the model additional
18:39 · information in terms of let's say a physical loss so you could give it partial differential equation constraints conservation laws and you can now enforce them at a finer resolution than the data you have then there's more guidance in a way so that way it
18:56 · can now come up with the the right answers even at higher resolution because you're you know giving it constraints at higher resolution and so that's how we can ensure that these physics-informed neural operators can work at higher fidelity and higher resolution than even the training data that was available my understanding a lot of your work uses a particular kind of a
19:22 · neural operator a Fourier neural operator so Fourier is a dual domain it is extended across the entire domain of the inputs that's a lot of jargon maybe can you give some intuition for why
19:38 · why is that important how does that help what i mentioned neural operators as a class of models that allow us to have any resolution input and any resolution output right and learns the mapping between them so that's really called an operator so the mapping between function spaces so that's the reasoning behind the name neural operator and in a Fourier neural operator was one of the early setups we or architectures we came up with and the reason why that's been so
Fourier neural operators, and why Fourier
20:11 · successful is because it kind of strikes a nice trade-off between efficiency and expressivity right so why is the Fourier space a good one the Fourier space allows us to you know it's a dual space like you mentioned but it really allows us to capture
20:29 · non-local phenomena right so meaning something that's like uh non-local in the Fourier domain could be even efficiently captured and the lot of phenomena like we see in nature whether it's fluid dynamics material deformation quantum chemistry it's
20:47 · all you know there's a lot of them are non-local you know the differential equation like the derivative is local but the inverse of it is you're kind of doing essentially integration it's non-local right so the solutions are non-local and these models are able to capture that but at the same time doing Fourier transform is efficient and it kind of like nicely captures a lot of inductive bias we see in many of these natural
21:17 · phenomena but this doesn't mean that we are capturing the world entirely in the Fourier basis right it's not a linear representation in the Fourier basis which is what classical numerical methods do we add non-linearity just as a transformer and other neural nets in between Fourier layers and we also add residual collections so
21:42 · all of these architectural aspects that are inspired by other neural nets that work well in other neural nets bringing that together really kind of helps us get best of both the world so you can think of like if we were to use transformers and we require a very high resolution it would become untenable because
22:03 · of the quadratic complexity and all to all connections on the other hand if you did that with Fourier transforms we have like quasi-linear complexity and still we have global connections in a way we can model these non-local phenomena and so that's why
22:21 · it's a nice middle ground so that allows you to learn from what is happening on the other like it is talking whether what's happening in Chicago may have some impact on what's happening in San Francisco well maybe not but that's the idea yeah so that's the idea in time like kind of yes there is you know like and I think like yes you know at this point maybe local but eventually they have an impact in the other locations
22:48 · and yeah so both in space and time we want to capture that dependence yeah so what happens today in Chicago will happen will have an impact in a month in San Francisco or something like that yeah so you know so there is like both the short term and the long term effects so in a short term like we think about predictable weather but longer term we're talking about climate
23:11 · right so what happens you may not be able to say precisely you know what happens in Chicago what will happen in San Francisco that's like the butterfly effect on the other hand we can kind of give averages you know if there's heat wave in this kind of overall region you know we kind of have an idea that it's going to be higher than average temperatures so those are the aspects
23:35 · we can capture together from an architectural standpoint for all the AI engineers here are we just talking about doing all the work in the Fourier domain but it's basically the same neural network but I'm just operating in the Fourier domain or is there other other aspects that are that are required in order to do this properly so think of it I guess maybe the easiest way to think about it
23:59 · is you know you can if you think of a transformer architecture instead of like the you know attention map you now have the Fourier but you still have other non-linearities you have like you know the residual you have you know many other parts of the
24:15 · architecture still there that give it like expressivity you can we have lifting to higher dimension like you know your channel space to give it more expressivity so all of those kind of best principles are are still available but the Fourier helps us capture that all to all the you know for dependence without
24:36 · requiring very huge complexity that I see transformers that makes sense the other advantage is that it gives you the natural multi-scale what's sort of a implicit cutoff is you know the sort of if you have a signals background or physics background you might ask you know in linear if you're doing everything linearly there's a maximum frequency and you know pull you know above that you can't represent anything but how does that how does adding these
25:02 · other architectural changes in a non-linear domain actually affect you know your choices of frequency bounds yeah yeah no that that's a great question and that's where the expressivity comes in right otherwise if you're just taking a Fourier transform of a signal and trying to represent it you know that's what numerical methods have also attempted to do and
25:25 · that requires very fine discretization and that's why it's very expensive to do the simulations in a classical way and instead if you want to move away from that and say we want to learn the features which is what deep learning is all about then we cannot force it to be only in the Fourier domain we have to give it non-linearity to figure out what the right basis for you know the best basis to represent the signals are and so that's the kind of like nice kind of combination we have that it's like all
25:57 · these non-linearities will help it kind of you know find the right space no fun in tendence this is the right and so you know the and if you do Fourier in that latent space you know that may be a more efficient way to represent so that's one
26:15 · way of thinking because you know first of all we're lifting the signal to more dimensions even if the signal is two or three dimensions we are now lifting it to much higher dimension so in that space the idea is to learn and we're doing it as a non-linear lifting right so there's already a latent space there and then we are doing further non-linear transformations in between our Fourier transforms so that means we are saying yes you know maybe with these limited number of frequency modes it's
26:47 · not expressive enough but when I add non-linearities I can you know I can kind of more nicely capture them.
Old math meets deep learning, and why scaling breaks
26:53 · So you started your career back before neural networks were I guess taken off right so I think back then people really did think a lot about you know appropriate basis sets and you know function expansions and orthogonal polynomials or whatever how does that evolution from you know your research standpoint how like as the community has evolved from that to oh just crew it throw it all in it seems like you still believe in at least some
27:20 · of those concepts as being guiding principles do you think that that that there is actually so lessons to be taken from you know classical mathematical like rigorous mathematical techniques that you can use those techniques actually to help improve modeling of the real world even if you still are just throwing the kitchen sink at things.
27:41 · No I think it's a it's a nice I think there's a trade-off I mean it's funny my undergraduate thesis more than two decades ago now was on fractional four-year transform right and so yes I mean by themselves like you know that wasn't enough to do computer vision but I was curious okay what are these techniques and how
28:02 · well do they work and so you know I'm completely with you that we cannot just force ourselves to use stone age techniques or classical techniques right I mean so we have to have feature learning we have to have flexibility expressivity you know they
28:19 · have to be easily optimized so all of these aspects are very important with deep learning but when it comes to the physical world and physical data it's never going to be as plentiful as we see with language models because we are you know our weather model like had about like 50 000 samples right 50 000 samples of fairly high resolution like work global weather maps but it's nothing like what we see with language and in other domains it's
28:49 · even less because it's so expensive to simulate and the real data may just not be available and so here we have to think about the inductive biases more we have to add the physics constraints cannot be just reliant on data and that's where I think a little bit more thinking of the architectural design comes up the other aspect is computational complexity so think
29:13 · about language it's just one dimension and even there the context length you know we are getting to millions and we are struggling right I mean on the other hand now we are thinking about the not just 2d 3d even 40 you know 3d and time and if each of the dimension is even a few hundred grid points which is
29:34 · where you know industrial scale starts at like like a thousand grid points in each dimension we're talking like hundreds of billions to even a trillion context length right so forget ever having a transformer for anything of this scale all of the world's compute will not be enough and first of all they all have to be co-located to be able to ever do this so that's why
29:58 · we need other architectures but I would push back a little bit right we have the vision and video language models right and they use they basically learn a mapping but the resolution is very low that's the key like for the physical world the resolution what do I require I mentioned like thousand by thousand by thousand by thousand so there you know the if you count that that's like already in hundreds of millions billions yeah so so you know we are not doing that high
30:28 · resolution when we think about images and videos currently I have a friend and the video is also like autoregressive so it's essentially only like you only need to do the next right yeah but you're learning I mean like generally you're learning a code book right so you have you're kind of learning the bias of the latent space or the the real world to the latent
30:50 · space and and and so that if there is a like a compression that you can do from the physical world into the latent space then then um you know these autoregressive techniques have been successful yeah but the idea is you know a lot of these autoregressive and
31:07 · you know techniques for vision and video models are for mostly like you know looking good right so they are like not for very precise simulations and they're you know having that higher resolution and details is really important and so we need to at least take in the data of that high resolution right so we need
31:29 · to process that and reason over them and so this is where a lot of the bottleneck is because we you know cannot afford to just throw away everything and say oh let's just like have 100 grid points in each dimension or 50 grid points because there just isn't enough detail to correctly model phenomena like fluid dynamics plasma how materials deform so all of this requires high fidelity and for that we need high resolution my understanding you have a thesis that AI needs you know to
32:03 · incorporate the physical world into it in order to scale and be accurate going forward many people have this thesis you are somewhat unique in that you have several examples of applying our operators to the physical world in this way and then it seems like you're getting a thesis around your experience here so can you share with us some of the really interesting and exciting looking things that you've done using neural operators and other techniques?
Weather, and the 2021 skeptics
32:39 · Yeah I mean you know for us when we started with like neural operators for partial differential equations but also more broadly you don't even need to assume their partial differential equations right could be any spatiotemporal or data at multiple
32:55 · scales so we you know set out looking for interesting examples and one of them was like weather modeling because the weather data is open source it's available called the EDO5 from the ECMWF the European Agency for Global Weather Modeling and so given that the data was there we were like okay let's just go try it right and that's the beauty of it whenever data is available it's really good news but a lot of weather scientists did caution us back then this was back in 21 and they said no no
33:28 · no this is so difficult you know there have been like decades of like development in traditional weather forecasting and that's very careful bottom-up physics-based modeling right so assuming oh this is the fluid dynamics can you go predict the weather the
33:45 · next day and so on and so that's how a lot of the thinking was that AI is just not going to be able to beat the you know decades of work in weather modeling but to our surprise we just went ahead we trained them we used neural operators to be able to effectively capture the phenomena and then now we to our surprise we found that it's not only you know accurate it's almost as close to the what the traditional weather models can do accurately but also
34:16 · tens of thousands of times faster so what would take a big supercomputer to run can now be run and we only needed a consumer grade like GPU like you know it was a small model it fit very well it's very fast and it's accurate and I think that
34:33 · just changed everybody's thinking after that DeepMind Huawei many others followed us a year later released their own models we were the first to actually open source our weather model forecast net and do it permissively so that's what allowed companies weather agencies everybody to build on us and so you know it's been a really interesting revolution to see that the weather models are now out there weather agencies are adopting them and it allows us to now have small weather agencies in the
35:07 · global south for instance have the same kind of fidelity that very big agencies were in the past only able to do right so it's democratizing weather modeling and so that's just one example of where there's been very quick rapid progress and a paradigm shift in terms of saying that oh now we can have AI as a reliable way to do weather modeling I saw that you know the the models that you mentioned you made
35:35 · an insight that nobody else had and that people were able to devise other mechanisms to kind of follow behind you but there was some sort of shift in thinking that was required here and was it it was it simply we believe that there's enough structure in this data to
35:54 · learn and that people are just doing it wrong and people found other ways to learn the structure but that your method was very um so let me let me clarify right so there is you know first of all the very first work was to just say that you know look traditionally this has been done with trying to solve partial differential equations each time doing it again and again whereas AI from data learns patterns and can
One model for weather and climate: assume a sphere
36:21 · be just as accurate but fast yeah and then the next iterations was to say you know how do we make it even more accurate and there's the aspect that you know there is the short term weather like what is predictable for the next two weeks and then there's
36:37 · the long term you know going to sub-seasonal to ultimately climate modeling and traditionally what people did was to have different models for these different scenarios there's a different kind of system that works for short term another system works for long term but to me there's only one earth you know if you want a foundation model if the claim is that it should be able to do both very short term as well as very long term together and that's where in forecast net three the latest
37:08 · iteration of the model we're able to do both and that's because you also you know incorporate the spherical geometry of the earth and so with a lot of the architectures that have been getting fairly good accuracies for the short term weather when you run them for longer term when you run them for like several months to even year even before that it just very quickly blows up right because it assumes the world is a rectangle which it isn't and so incorporating all of the geometry and that
37:40 · information into neural operators means that we can faithfully run the same model also longer term and make this into a climate model this is where the Allen AI Institute has now built climate models based on our neural operator architecture and that's the
37:58 · only one that works as an AI emulator right none of the other architectures work for climate because climate requires us to assume the world is a globe and that if you are repeatedly rolling out you kind of keep that information whereas if it's like a narrow surrogate that's what I consider a weather model you just narrowly look at a few metrics many different architectures should do the job right but if you are asking one
38:28 · architecture do a range of different tasks like a foundation model that's where incorporating the geometry of the earth which is that it's a sphere and using neural operators an efficient way to do that enables us to accomplish that yeah so forecast net you said trained on 58,000 data points can we just talk about like what does this look like like what does the data input look like what are you actually trying to predict
38:56 · from here and then what is where the like the large scale you know you said you you're going from weather to climate what does it look like to do that because I could imagine if you have 50,000 data points these are you know some high resolution in North America then you know they might even depend on the local geography like like if you are always modeling you know Kansas does this kind of transfer to um you know let's say the swiss Alps or something and then does that transfer to uh you know to
39:26 · him so first of all to clarify training it on the global weather model right so we have all the information around the earth and that we are you know asking it to predict like given the current weather uh like say wind conditions humidity and so on what happens uh you know in an auto regressive way and it's every six
39:46 · hours so what happens in the next six hours and so on and you roll out and you train the model to predict and and so you know you can have potentially the same model predict forever right so but the predictability window is like the weather and if you want to go beyond if to do what we call ensembles meaning you have like a probabilistic estimate of what happens uh in several months to years and that's how you get a climate model okay so you have you actually have an ensemble of these local
40:19 · predictors and then you use the some sort of statistics on the ensemble to get a climate yeah so you kind of like have several rollouts essentially you have several trajectories of rollouts and then you get and so you can actually pretty key here yes and that's why that's the biggest bottleneck with traditional climate modeling that it's so expensive to do even
40:43 · one single run you have to do long trajectories very very high resolution and uh you know that's why we don't have a lot of very high resolution ability to do climate change predictions for instance so how do you validate climate so you do lots of rollouts and you look at sort of how things evolve kind of an aggregate i mean nicely said butterfly effect locally i think we can believe even a perfect climate model or perfect weather model would only
41:12 · give you maybe two weeks before it sort of it becomes non-computable so um you do lots of rollouts um there's some chaos you you average all these things how do you actually validate that this works over long enough times yeah and skills
41:27 · yeah and and you know it's a tricky question right so for instance you have to kind of ensure that you satisfy all of the physical constraints and if you just do a standard rollout that is you know likely not going to happen and so some of the ongoing research we're how do you kind of enforce the right physical constraints as you do the rollouts like you don't want to you know completely wash out the fine details because then it's not accurate uh but on the other hand if you keep them you
41:58 · may be physically they're invalid so this is still an open problem and that's what makes it difficult that you want ai to be fast and you want to be able to do these long climate simulations and at the same time be able to have full confidence
42:14 · in them but these are things we are working on now i don't know if this falls into weather or climate but we recently just had one of the most extreme heat waves in the history of modern climate data right um in the just right here in the uh kind of southwest united states i'm wondering where you did you i don't know if you were involved in this and regularly but do you know if you or anyone actually modeled that or predicted that correctly yeah yeah so we have i know i don't have information on this
Why the physical world is forgiving
42:44 · specific one but we've tested in our latest uh forecast net three model extreme weather events of all kinds right and that's the key like you know that's where you need probabilistic answers so having just one deterministic output and saying that this is the weather is not enough when we are looking at extreme
43:05 · events so we need careful probabilistic calibration and we show that we are able to capture those well and i think that was the surprise even in our very first attempt that we visualize certain hurricanes and storms and it was able to do well which is very surprising because you would think that rare events are
43:25 · not something ai would do well right would do well on typical events but i think this is where more broadly the lesson is the physical world may be more forgiving because you know where there are extreme events like hurricanes that have very specific physical signature yeah right so it's like extreme but in a very
43:46 · specific way so maybe you don't need as many samples because the physical world has a lot of structure and that's something we see this again and again that there is a lot of structure in so many other examples you know talk about plasma and fusion reactor you know we barely have a few thousand samples but we are able to accurately predict events like disruption very well
44:10 · and we are able to do that a million times faster than what traditional simulations were able to do to me yes all these sound very surprising but it's because i think the nature helps us a lot and it has a lot of latent space structure that's uh that i don't think traditional numerical methods are able to uncover because they are focusing more on correctness that in any scenario you should be able to solve these equations on the
44:38 · other hand with ai it's learning from data it's uncovering this structure it's uncovering how easy or kind of tractable these problems are and that's what we see in many cases yeah i've had a similar analogy so from you know my own domain is probably closer to computational biology but you know alpha fold is the obvious uh like really exciting development in the community so the solving protein structure prediction and of course all the caveats of what was actually solved well i think
45:09 · we discussed this in a in a previous episode with the bolts team encourage listeners to listen to that if they want more but one of the i think points about protein structure is that it really is constrained by physics and that's why it was in some sense one of the few big wins in the field of biology which is otherwise very complex and we've had trouble generally speaking
45:32 · machine a lot of success and it seems like problems which are solved by differential equations or i say modeled well by differential equations have a lot more room for also integrating these techniques um as a general form don't think there's a question yeah so so so getting back to my question um we have climate or weather and climate we have plasma we have um uh biology and i know that
45:57 · you you prepared for us a few visualizations can you can you just share with us um what does that look like you know what so the visualizations and then what is the thread that runs through here and i think maybe the listeners will already have a hint
46:16 · about that but i'd be really excited to see that yeah i can certainly share some of them i mean this one is just kind of showing that you know we have world at different scales right and these are examples of phenomena happening at different
Modeling across scales, from atoms to planets
46:33 · scales from atomic to protein to even planetary scales like the weather we talked about and you know we need to capture all of that that's what neural operators are designed to do and you can you know feed in data these different scales and that's really like the aspect that makes a lot of the physical world problems hard that need for fine scale you know we talked about how a lot
47:00 · of traditional computer vision video models are just designed to make things look visually good and that requires low enough resolution it's tractable regressive it's enough that you know it works out it's short enough videos but that's not how a lot
47:17 · of the physical simulation for you know industrial scale high fidelity work you really need high resolution the you know the atmosphere is one example like you depending on the resolution you observe different phenomena can be captured so you just miss
47:35 · that out if you don't have that resolution and now the question is with ai can we do this much faster than what we could with traditional simulation and this one with neural operators is kind of showing that you know if you use the standard neural network and you had a fixed number of pixels like you're seeing on this side and you zoom in it gets blurry right that's the end of it there's nothing beyond those fixed resolution that you can
48:06 · capture but the idea is with neural operators because it's a function space representation meaning you can keep zooming in you can add it the relevant details either by giving it data at higher resolution or physical constraints at higher resolution
48:24 · you can kind of bring that multi-scale phenomena together so where are the physical constraints i mean i assume physical constraints here are like local simulating um you know fluid equations or some sort of yeah so dynamic could be right so it could be of any nature the idea is now you can add like conservation laws for instance in an incompressible
48:46 · fluid you can add like material deformation like how things stretch so or it can be a full partial differential equation so that's also an interesting question we've been researching how is the curriculum of different physics like you meant you know
49:03 · some physics may be very hard to impose or add as a loss function others may be easier so you also need to kind of you know think about what what to impose you know and the intuition is here is that when i'm adding a physical constraint i'm adding it to the loss function is that more or less what's happening yeah because that's what is tractable you know making it a hard constraint is not tractable whereas adding it as a loss function
49:29 · and of course there's still the balancing of that loss with the data we have so we have to you know uh do that in the appropriate way right yeah so as i mentioned the this is the example of the weather model where here we are showing how we are able to capture like atmospheric rivers which is the phenomena we see here in california you know resultarian life storms i think we are we have one expected later this week so we'll hold you to that
50:02 · so the idea of like you know why i show this is this kind of global phenomena right these are like thousands of miles wide so you really need non-local models that capture these very large span phenomena and do that accurately and that's what our neural
50:20 · operators are able to do so so the training data for this is you were talking about this a bit before but i'm still like wondering what is this this is weather satellites or is there ground-based data is it some hybrid of the two it's it's kind of a combination of different sources so it's what we call re-analysis data so this is historical weather data that is in a way re-analyzed meaning that the sad rock satellite
The data, ECMWF, and calling Hurricane Lee early
50:46 · data is combined with essentially what the physics solvers tell you together assimilate it and so this is made available by the weather agencies and we can train on them so you're saying that they take a low resolution data set which is compiling all of the world's data set you know all of our biological data we have across the world and then they do short
51:11 · time simulations using physics-based you know classical techniques to to fill in the details and you can do that over short time spans but as you go farther it breaks down very quickly so you're amortizing that across all the everyone would have to do that and so somebody does it and then you are able to take yeah i mean this is data right that's already prepared but the idea is already this data simulation with physics kind of makes
51:40 · our model physics informed immediately so it's able to kind of you know keep that information and that's why maybe that's one reason maybe it does well on even extreme weather events yeah so this is just showing that we our model is available in the cmwf which is the weather agency like you know the european weather agency uh and so this was launched like more than two years ago but you know i think fall 2023 so you know i think ecmwf making
52:11 · these ai-based weather models available to the public to me was a very big step because that's where you know everybody could see what's happening there were several hurricanes like for instance there was hurricane lee and that's where the public could see what are these weather models doing for instance our
52:32 · forecast net was able to correctly predict that the hurricane making the landfall several days earlier compared to the standard uh weather forecasting models and so the idea that these models could be very good for extreme weather and do early prediction you know both for human lives for economic costs is a
52:54 · very big deal and so that's when the public kind of got a lot more i think buy-in and from weather scientists because of how well it was doing in these events and this is what i was talking about in an ensemble prediction both for extreme weather or if you're thinking about climate it's not just about looking at one
53:16 · trajectory right because you know unless you're somebody with a sharpie somehow saying okay the hurricane is gonna go no fun intended but you know what you really want is the probabilistic prediction meaning you know i'm gonna try different adding noise
53:33 · levels to my initial condition what the when the weather when the hurricane is forming in the caribbean i'm gonna add some noise because anyway it's noisy i don't know truly what the measurement there is and then i'm gonna look at what happens to the possible hurricane tracks and then i can come up with the probability of landfall in different regions and that's how i can do risk assessment and so this is where it gets even more expensive for traditional weather models because you have to run
54:07 · all of these ensembles and now ai weather models being so fast tens of thousands of times faster means we can now very large ensembles and this is a very big improvement in terms of what we can do for risk assessment have you gone through and done let's say looked over the historical hurricane maps and then tried to do ensemble predictions and calibrated how often your predictions are like a yeah yeah so in forecast net three paper that are you know we have
54:38 · metrics of like extreme weather events and ensemble prediction and in fact we've trained the model to do good ensemble prediction and so this is where the calibration matters for for these kind of events what was the sort of key insights or developments in forecast net three in versus two versus the first version yeah so the first version was kind of the you know the using like the Fourier neural operators but we didn't incorporate the
55:05 · spherical geometry right in this next version we said i think you know it's important that the world is a sphere because first of all otherwise distorted so you kind of are not predicting uh it in our question but if it wasn't spherical what did you do like a mercator projection or something yeah the standard like kind of the you know like and all the other weather models do the same right so
55:29 · they just kind of have the standard projection and then uh it's predict the weather and which is okay for short term prediction but when we in our goal was to have the same model also do longer term and that's when incorporating spherical geometry added this additional stability we could do longer rollouts and
55:48 · then in forecast net three the idea was it's not just about deterministic prediction we want to get ensemble predictions right so we have to train them based on this objective that we get the probabilistic predictions correct as well how long are you predicting out and how many rollouts are you doing yeah so the you know rollout is how long you predict right so each step is six hours and then you predict for how long you want you know you just have to roll out sorry how many how many examples in the ensemble df
56:22 · so and again that's our choice we can have like ensembles of different levels so we i think it's like a few tens or something like is what we are currently you know shown but you can do much larger too and that's adequate to get out how far my intuition is the longer you want to predict the more so not necessarily it's really like about again calibrating the ensembles and ensuring that they have the right spread rather than you know okay so maybe so you have tens of these um models or
56:56 · examples in your ensemble and that even with a very very long roll out that's adequate so again like you know there's as i said a lot of still outstanding questions to do very very long rollouts right because you do need to incorporate like the physical constraints
57:14 · in a way to ensure that that's something that we are actively researching now but these models that we have are able to do the longest rollouts compared to any of the other weather models that completely ignore spherical assumption and the range of other things so when you say incorporate physical laws for climate over long times i mean what does that look like because there's a lot of the local conservation which may be just broken if you take an ensemble average even though any given snapshot is respects that
Long rollouts, and why the sphere stays stable
57:48 · no the idea is to make sure you look at each ensemble member and it respecting the physics okay okay you're not assuming the ensemble how you're not deriving a like coarse grained like equivalent of a probability or something because then you would lose that you know resolution and the fit
58:05 · okay that's a little confusing to me wait so you're so it's not an average well how are you combining the ensemble you are doing the average but you're predicting each one oh you're predicting each one separately okay so each one is independently satisfies these constraints but the ensemble it does not yeah which and so that's how you ensure physical biology yeah so when you go on a sphere you operate in do use basically
58:31 · harmonics or some sort of yeah you have a yeah yeah which is actually very natural with for enough for is probably much harsher if you're doing yeah um other exactly so that's where the you know like the 40 years saves us can incorporate these geometries well and and so i think be very
58:50 · faithful to you know what the globe is yeah arguably it's much more natural than uh didn't like a marketer projection or whatever other yeah which is like green land becomes huge different story but uh but yeah but i think that this is where i think the aspect of you know
59:08 · more broadly incorporating more of geometry and information about the domain becomes a lot more important in for the physical world right so this is me again emphasizing that we need to incorporate more of the structures because one is the data is limited and the other is a lot of what we are asking is extrapolation you know to go beyond than what the strain on you
59:33 · know we're just training it to predict the next six hours and maybe do a little bit of multi-step fine tuning for auto regressive rollouts right so we are not training it to do very long like a climate because that's just too expensive but we hope magically it works well and it cannot if you just say i'm just gonna put a standard transformer or whatever else there and
59:58 · it won't work out so we added more of the domain constraints like spherical geometry we add in maybe more of the physics in certain ways and that's where it becomes more interesting algorithmically as well you know there's more involved design here so the time scale that you train on how long is that
1:00:19 · to predict for the next six hours oh so only six hours yeah and a little bit of multi-step fine tuning like i said yeah i said okay understood so which is very surprising yeah that is very surprising i would have expected it was weeks or months yeah no i've then it kind of just works well even for like now we are showing for several months that it's able to do that the number of steps is on hundreds or yeah and that your your foyer basis your it's it's many times the the the sort of base
1:00:52 · harmonic i mean this is like that's in space right so we're talking rollout is auto regressive with time nobody like in in time i'm maybe i'm misunderstanding here because it's a in foyer domain no no in time it's not that's what i'm saying it's auto regressive oh oh i understand okay so it's space it is yeah yeah interesting yeah
1:01:14 · what's the uh angular resolution at least in in this scenario in other cases we also have in time is also represented in the foyer domain and that's a question as well can we do that but in this example yes auto regressive got it um what's the angular resolution that you use for for this um in you know in the spherical version yeah so it's so all of the data that's available is like i think
1:01:41 · a quarter like 0.25 degrees so in terms of l maybe or i mean in terms of like a spherical harmonic frequency or and so this is like uh oh you mean like how many modes we utilize i think we so for that resolution and we essentially utilize i think most of only a few of them will live out i forget the details but i'm just curious like what is the actual angular resolution on the globe that you are resolving to or maybe just the the physical resolution i mean and that's what that because that's the data we get right
1:02:14 · so right now the data we get is like a quarter like uh 0.25 degrees totally okay like a 0.25 solid angle so i think it's the yeah it's like i think kind of comes out to like uh you know 700 by few thousand like resolution so but this
1:02:32 · is already standard like kind of processed yeah yeah and just uh trying to understand like how large is the basis do you need to represent this i mean that's really depends on the resolution and the idea is you know right now or whether data is just limited by this resolution but if you could you know you could like kind of do synthetic climate simulations of even higher resolution right and that's kind of the next thing on how combine these together do you think you can predict with super resolution be basically
1:03:02 · resolution lower than the data provided again like yes we can always predict them with the neural operators uh but you know you do want to incorporate more of the physical constraints to ensure that they are valid okay so at um can we talk about some of the other yes yes i know it's a lot uh so this is just showing like how you know the what i described that the on the left where the world is being assumed it's a rectangle it blows up very quickly
1:03:33 · and on the right because we assume the world was a sphere it kept rolling it out and it kept being stable so i still see a little bit of a singularity there is all right and it's still like you know so the idea is yes because it's a very long rollout and we have no guardrails of physics we are not you know kind of projecting it to the right physics right this is full extrapolation but the idea is the sphere assumption
1:04:00 · stabilize it to a much greater extent compared to the left it's a much better yeah but if you're you're in the south pole you're still not going to get exactly since the poles are the hard parts so so this is the example of the fusion reactor so this is a tokamak and we are able to model the complex plasma evolution and do this a million
Fusion: a plasma digital twin
1:04:20 · times faster than what we could do with traditional simulations and this was in a way we're creating a digital twin of the plasma right and then we can you know do further things like right now we are as a next step looking at like control but with
1:04:38 · the full valid physics like being able to prevent disruptions ideally and make fusion sustainable so are you simulating mhd equations here or sorry mignito hydro dyne n-mass equations yes okay yeah and then so for context disruption in this case is this phenomenon which plagues which plagues plasma physicists where at some point your entire plasma collects in a little tiny beam and then shoots a strong you know right to your containment vessel and so and it can damage the reactor and that's the that's a big
1:05:10 · bottleneck because then you have to kind of shut it down before that happens and then plus it's no longer possible to have a sustainable future so there's a lot of open challenges here but the idea is you know it's very expensive to go to physical experiments the more you can capture that in the digital twin but ensure physical validity the more you can even do design and
1:05:34 · other considerations in the digital realm you know we can hopefully make advances and these are the first steps towards that the goal is that if you have one of these events that you can somehow adjust the the um the magnetic field so that it contains
1:05:52 · that and and stabilizes yes and that's the next step we are doing now we are looking at like designing both the control and the simulation together are you working with this specific lab i'm just curious so this one was with the UK atomic energy agency okay and now we're also working with a few others here in the us as well so
1:06:13 · we are you know kind of getting the information from many different approaches of fusion itself so this is the tokamak we're also working with stellarators we are working with different stellarities are are tricky yes yeah but the idea is ideally you know like our goal is to be able to design them in the digital twin so can we come up with good designs that would make it maybe more practical and so that's i think also good thing as an ai person and much more like you know agnostic and not picking a winner beforehand right
1:06:46 · like i like to work with different approaches you know and see whether ai can accelerate all of them and then we can kind of you know not prematurely rule out one approach over the other so that's what ai enables us to be more kind of taking risks and exploring different approaches as opposed to in the physical world trying to build any of these you kind of have to cut a lot of the risky ones and say i'm only going to do this because this
1:07:13 · is what i would do and i'm not going to do it because it's a very difficult situation and so it's not really likely to work i noticed over your career you started out spending a lot of time on you know really theoretical foundations and mathematics of machine learning and maybe i don't know something like six eight years ago you started working really working a lot on applications and branching out in a diverse set of problems what
1:07:40 · sort of prompted that shift in your your looking at i mean so you're still working on very hard math problems as well like for example the torch lean work but um the the applications have really grown and i'm wondering what what prompted that and like um what were some of the lessons you've learned since then sure i mean to me it's like you know i feel like i've grown along with ai right so when ai was you know in this where neural
Career full circle: theory, scale, and back to principled
1:08:07 · nets were not working because there wasn't enough data and all kinds of other reasons you know then you kind of have to build the theoretical foundations and try to hope that that leads you to a place where you know you get algorithms to work right and and and and you know back then ektensor methods was with that idea that uh you know pre-deep learning we still wanted structure we have probabilistic models like late and Dirichlet
1:08:36 · allocation for topic modeling and solving those were hard but now tensor methods gave us a way to be very practical it's parallel and can be done at large scale but still has nice theoretical basis so that was where you know we're starting off and then as deep learning started taking off and we could see that it works well in practice and yes there is a little bit of
1:09:00 · maybe theoretical understanding but not a whole lot because of the way how complex it is to me theory should not be a constraint right it should be an and so that's where a lot of like the exploration was to make this work well in practice and over into amazon web services then nvidia so really like making things work at scale and really kind of getting hands dirty right was kind of like where a lot of the development is and now
1:09:29 · i see a full circle because a lot of day purely data driven approaches in a way seeing saturation right we want to ask okay either make them more hardware efficient right there's a lot of now room to kind of say can we now you know make them much more energy efficient or hardware efficient so that's one aspect but the other is areas like this where in the physical world we don't have enough data we are asking for hard extrapolation you
1:09:58 · know we want to think of doing discovery by nature it's about extrapolation so we will never have data about a new discovery right that's my definition and so there we need to again go back to thinking in principled ways and whether it's architecture design algorithm design the right loss
1:10:18 · functions so we need to be much more mindful so i see that coming a full circle because all of the things that work with deep learning let's take them but make them a bit more principled there are several other applications which seem very natural i'm wondering if you've worked on these or did i just miss some papers if i did i'm sorry um so some examples are design of like electromagnetic circuits um i think is a big one uh or maybe not
1:10:46 · a big one but i think we'll be coming up in the near future design of let's say materials um design of let's say dissipation and heat sinks or um or any sort of like fluid flow as i'm going through what differential equations i know electromagnetism i know diffusion equations um you know it mhm mhm yeah i'm wondering some of the other domains that are yeah i mean to me there is just endless possibility right so there you know as like you can just have this work on any data
1:11:19 · and we have several other examples so this was like you know being able to ask can we sequester carbon dioxide underground and model uh how carbon dioxide uh expands or you know what is the pressure build up in these reservoirs and uh you know can we kind of model how they migrate
1:11:40 · over several decades and so this one we were able to do much faster than what traditional simulations could do i mean the other aspect is being able to do all kinds of geometric shapes like you know being able to model aerodynamics in cars planes and so on and so again this is a nice example of a latent space because you can transform a car or any other shape to a donut and then model on the donut and then transform the donut back to
1:12:09 · the car you didn't turn it into a coffee cup is that the classic joke you're donating any coffee things actually so the idea of like a latent space to handle all kinds of different geometries and be able to capture the physics there in the latent space well means we can now have a model that generalizes across a lot of different geometries am i understanding that the maybe the larger vision here is that
1:12:34 · uh you can train a foundation model in the sense of being able to model many different physical phenomena with the same model and so you may fine tune or there may be some kind of prompt that you give it to have it understand the particular geometry
1:12:50 · but that you you know sort of on all these different physical problems you train and then and then you have your particular one and you're able to model that very effectively yeah i mean that's really the future right because we have foundation models for language maybe vision but not for physics so you know the idea is instead of like right now what we've seen are narrow surrogates and we're trying to broaden their scope more and more but ideally we have much broader models that
A foundation model for physics, and inverse design
1:13:22 · can work on a range of phenomena but also multi-physics so not just have like one single physics but coupled physics the real world has all of the physics kind of coming together in coupled ways so can we bring all that together so that's one aspect like you know have foundation models that can do design that can do simulation but the other aspect that's really interesting is the
1:13:46 · inverse problem right so can i now not just simulate but ask what is the best design and then these kinds of like models can like do simulation but you can even do that implicitly and come up with the best design rather than in the earlier era it was humans trying to come up with design then you go and try to simulate or go to the wind tunnel whatever physical testing and validate that but now you have ai come up with optimized designs
1:14:14 · but you have the guardrails of physics so you have models that are accurate in physics you have the confidence they work well so you're kind of able to do that as well in the same model is there reason have you seen any evidence that you talked about
1:14:32 · these like sort of multi-physics being able to transfer or that that you may be able to generalize to sort of unseen physics so i mean you know like the physics by nature if it's completely
1:14:47 · unseen it's not possible to transfer right i mean sure how many of you are saying that we're going beyond the standard model there's absolutely no data that's not possible but if you're asking about like you know for instance like you know there is the like say i've like you know shown it examples of like just how the heat propagates and the other examples of how the material like stretches and now there's coupling like because of
1:15:16 · heat there's also stretching or kind of the joint phenomena you could like now hope to fine tune with much fewer samples because it kind of individually knows this phenomena then combining them together maybe it can't do it from scratch because that's still too much to ask it's highly non-linear and coupled but it can do
1:15:38 · it with fewer examples and we've seen evidence of that in a lot of our papers that you're able to kind of essentially build up a curriculum and that's what we see again and again in many of these examples that you know the real world we can kind of control a lot of curriculum and say you know let's kind of build
1:16:00 · in like modules and put them together and that's what it now allows us to do in a systematic way here i guess the design aspect i don't know if we wanted to show very quickly so this one was like you know looking at like designing the mask for inverse lithography meaning now there's an inverse design problem and we are also able to do that for designing gates and quantum dots this is like non-linear photonics and all of this what is common is the idea that you know there's a forward model
1:16:33 · that is simulating the physics but now what we want is the inverse design like the problem that of we can optimize the best design and humans are usually not good at this right we are not good at like looking at highly non-linear phenomena and say oh somehow maybe this combination of all these gates coming together helps pull the electrons together in a quantum gate and
1:16:59 · so our collaborators were struggling to do that manually and with ai we are now able to come up with very efficient designs but also those we know actually work because we have already the simulation as part of the loop saying that they work well so i think these are examples where we see that it's not just about simulation it's about really novel designs and novel discoveries that enable us to move the needle of innovation itself each one of these examples takes a lot of domain knowledge how
1:17:34 · could somebody take your basic research if a domain expert and quickly get started applying neural operators and the other frameworks that you've developed to their problem yeah you know neural operators are an open source library it's extensively already adopted it's part of the pytorch ecosystem it's you know used by a number of not only researchers but also in companies we have a lot of documentation there so i encourage
1:18:04 · people to go there we have like you know many different architectures examples recipes so i think that's a great place to get started you recently joined the un scientific advisory board i know we're running out of time but maybe just can you quickly give a bit of the story behind this and what you hope to accomplish yeah no i'm really honored to be part of that advisory board for the un and in these tricky times with a lot of geopolitics there you know which again i'm not the expert on that but when it comes to
The UN, regulating AI for science, and a call to action
1:18:36 · you know aspects especially related to ai having scientists in the room is something that you know i think is very important i hope i could have an unbiased view and try to provide scientific evidence for any aspect right we want to think about how ai impacts globally like you know how do we ensure the benefits of
1:18:57 · ai reach everybody how do we democratize access to ai how do we ensure the unintended consequences and harmful impacts can be controlled i think these are just the beginning aspects of course the other side when it comes to weather models i'm already excited like you know there are there is a push to seeing how we can have better weather climate modeling so then
1:19:21 · our food you know like using weather for better agriculture so all these aspects are also where un has a lot of dedicated agencies and people on the ground across the world so i'm looking forward to contributing and being part of this you know looking you know looking at your career and how you you
1:19:40 · know talking what you work on it seems like you very much are a person who likes to solve concrete problems you don't like to philosophize about things which then you're also seeing i think maybe more optimistic than a lot of people in the ai space um you have a very hopeful view of the world um i think not that's not always true um yeah so i guess what are the ways that you can uniquely bring that viewpoint to the board versus maybe some you know yeah thank you i you know to me i think as i said i tried to be
1:20:13 · unbiased and as a scientist and as a scientist i think that there's a lot of beneficial aspects of ai that are sometimes missed when we think of only the harmful impacts right and and especially that is with respect to ai for science because a lot of regulatory frameworks equate ai with language models and yes language models can you know have manipulate people can have all
1:20:40 · these kinds of harmful impacts that we should know about controlling but ai for science is different so i think this one size fits all is where a lot of problems come up so we have to be mindful that there is you know ai that can change the world
1:20:58 · with new discoveries and we should enable people around the world to not only benefit from them but also be able to do research you know have access to ai that they can go in a and use them in interesting ways one question that we have been trying to ask every guest is if
1:21:17 · you could pick a bottleneck in your domain that you could magically remove what would that be and why more compute you know that's i know that's an easy one maybe a lazy one right because you know and you know of course our compute that we have is growing so much more than even a few years ago thanks to nvidia thanks to others again thanks so
1:21:45 · no no comments on that but what i mean by that is also like for research enabling more and more compute you know it's very important i know there are national labs building more supercomputers you know hoping that we can have more compute for but i think you know without that we cannot experiment we cannot innovate i think this is a part that i push a lot and you know i think i cannot emphasize that it's so critical if you had a call to action or something that you would like
1:22:19 · people to do or think about or learn about what would that be yeah so you know you can go to neural operator libraries so you kind of hands-on play with different architectures recipes you know look at use cases but also think about like you know ai for
1:22:38 · science as not just language models and agents yes that's one aspect of it but ultimately you know those are still like external rappers in a way right until we have ai that fully understands the physical world not just as symbols but as one that can simulate and design and control based on that you know
1:22:59 · there's a big piece missing so that's the other aspect that i think the people should really think about ai for the physical world in this way Anima this has been so fascinating um i i'm excited to check out neural operators myself i have some ideas in my head already i
1:23:18 · really appreciate you taking the time to sit down with thank you Arjes thank you Brandon i really enjoyed it it was really dug deep into a number of things so i appreciate you doing that thank you thank you