The Agent Revolution: Navigating AI Psychosis in Software Engineering

Welcome back to No Briars. Today, we're joined by Andrej Karpathy for a wide-ranging conversation about code agents, the future of engineering and AI research, how more people can contribute to research, advancements in robotics, his predictions for how agents can reach into the real world, and education in this new age.

Andrej Karpathy's Introduction to AI Agents

It's been a very exciting couple of months in AI. I recall walking into the office recently and finding you intensely focused. When asked what you were doing, you responded, "I have to express my will to my agents for 16 hours a day." This shift, where "code" is no longer the precise verb, underscores a significant jump in capability.

I often feel like I'm in a perpetual state of AI psychosis because there's been a massive unlock in what an individual can achieve. Previously, I was bottlenecked by factors like typing speed. But with these agents, something fundamentally shifted around December. My workflow transitioned from an 80/20 split—writing 80% of the code myself and delegating 20%—to now potentially 20/80 or even more skewed towards delegation. I haven't typed a line of code since December, which represents an extremely dramatic change.

When discussing this with my parents, I realized a normal person doesn't fully grasp the magnitude of this change. The default workflow for a software engineer, even a "random" one at their desk, has been completely transformed since December. This state of "AI psychosis" stems from trying to comprehend and push the limits of what's possible.

I'm constantly exploring how to manage multiple sessions of Clot Code or Codecs, or other agent harnesses, rather than just a single one, and how to utilize these "claws" effectively. There are so many new avenues to explore, and I feel an intense urgency to be at the forefront. Seeing others on Twitter experimenting with various promising ideas makes me nervous about missing out. It's an unexplored frontier, which fuels this constant state of discovery and apprehension.

What Capability Limits Remain?

If Andrej is nervous, the rest of us are too. We work with a team at Conviction where engineers no longer write code by hand. They're all microphoned, whispering instructions to their agents constantly. It's the strangest work setting, but I now fully understand and accept that this is the future—you're just ahead of it.

My capacity now feels limited by so many things, yet often, even when a task doesn't work, it feels like a skill issue. It's not that the capability isn't there, but rather that I haven't found the optimal way to integrate the available tools. Perhaps I didn't provide clear enough instructions in the agent's markdown file, or I haven't implemented a sufficiently effective memory tool.

The goal is to effectively parallelize these agents. Essentially, everyone wants to be like Peter Steinberg. Peter is known for a striking photo where he sits before a monitor displaying numerous Codecs agents. These agents, when prompted correctly using a "high effort" setting, can take about 20 minutes to complete tasks. With multiple repositories checked out, Peter seamlessly switches between them, assigning work. This allows for operating at a much higher level, focusing on "macro actions"—not just writing a line of code or a new function.

Instead, it's about defining an entire new functionality and delegating it to Agent One, then assigning a separate, non-interfering functionality to Agent Two. The final step involves reviewing their work based on the criticality of the code. The core question becomes: what are these larger "macro actions" I can use to manipulate my software repository?

Imagine one agent conducting research, another writing code, and a third developing an implementation plan. This is how entire projects unfold through macro actions across your repository. Mastering this workflow, developing a muscle memory for it, is incredibly rewarding—primarily because it works.

It's also a new skill to learn, which explains the "psychosis." My instinct is always that when an agent is working, I should be doing more. If I have access to more tokens, I should parallelize tasks and assign more work. This creates a stressful dynamic: if you don't feel limited by your token expenditure, you become the bottleneck, preventing the system from reaching its maximum capability.

You're not maximizing your subscription. For multiple agents, if you deplete your allocation on one platform, you should switch to another. I feel nervous when I have subscription time remaining, as it indicates I haven't maximized my token throughput. This mirrors an experience I had as a PhD student: a sense of anxiety when GPUs weren't running at full capacity, not utilizing all available FLOPs. Now, it's not about FLOPs but about tokens—what token throughput do you command?

It's interesting because for over a decade, many engineering tasks didn't feel compute-bound. Now, the entire industry feels resource-bound. With this significant leap in capability, the constraint isn't compute access anymore; it's the human user. This "skill issue" is actually empowering because it means you can always improve, leading to addictive "unlocks" as you get better.

What Mastery of Coding Agents Looks Like

Where do you see this evolving? If people are constantly iterating and improving their use of coding agents, what does mastery look like in a year? Indeed, what defines mastery at the end of a year, or even two, three, five, or ten years?

Beyond Single Sessions: Collaboration and Persistence

Ultimately, everyone is interested in moving up the stack. It's no longer just about a single session with an agent. The focus shifts to how multiple agents collaborate within teams. This is a critical area everyone is exploring.

Additionally, Claw represents an interesting direction. By 'Claw,' I mean a layer that elevates persistence to a new level. It's designed to keep looping autonomously, operating without constant interactive oversight. It functions in its own sandbox, performing tasks on your behalf even when you're not actively monitoring it.

Furthermore, Claw incorporates more sophisticated memory systems than those typically found in standard agents. For instance, Open Claw offers a much more advanced memory capability compared to the default, which often relies on simple memory compaction when the context window is exhausted. Do you think that sophisticated memory is what resonated most with users, more so than Open Claw's broader tool access?

The Importance of Personality and Memory

Yes, I believe there are many excellent ideas here. Peter has done a remarkable job. Peter has genuinely done an amazing job. I recently spoke with him, and despite his humility, I believe he simultaneously innovated in about five different areas and integrated them seamlessly.

For example, in the documentation for his Soul and D project, he crafted a truly compelling and engaging personality. Many current agents fail to achieve this. I find that Claw, in particular, has an excellent personality; it feels like a supportive and enthusiastic teammate. In contrast, agents like Codex can be quite dry. This is curious because in general ChatGPT, Codex often feels upbeat and conversational. However, the Codex coding agent seems indifferent to the creative process, merely stating, 'Oh, I implemented it.' It lacks an understanding or appreciation for the broader project.

That's true. Another observation is how well Claude's personality is tuned. When Claude offers praise, I feel I've genuinely earned it. Sometimes I present it with underdeveloped thoughts or half-baked ideas, and it doesn't react with excessive enthusiasm, merely acknowledging, 'Oh yeah, we can implement that.' However, when I present a truly solid idea, by my own assessment, Claude seems to reward it with more positive affirmation. This subtly makes me feel like I'm trying to earn its praise, which is a peculiar but effective dynamic.

Therefore, I believe personality matters significantly, an aspect many other tools don't fully appreciate. Peter clearly prioritized this, along with developing a robust memory system and creating a unified WhatsApp portal for all automation. He's clearly having a lot of fun with these innovations.

Personal Projects: Dobby the Elf Claw for Home Automation

Asked if he had used Claws for personal projects beyond software engineering, the speaker recounted a period of "Claw psychosis" in January, during which he built a home automation system. He created a Claw named Dobby the elf claw to manage his house.

Dobby's initial task was to locate all smart home subsystems on the local area network, which worked surprisingly well. The speaker simply told it, "I think I have Sonos at home. Can you try to find it?" Dobby then performed an IP scan of local network computers, successfully identifying the Sonos system.

Discovering there was no password protection, Dobby logged in and began reverse engineering the Sonos system. Through web searches, it identified the API endpoints. The speaker recounted his amazement when Dobby asked, "Do you want to try it?" and then, with just a few prompts, played music in his study.

The system achieved similar results with lights. Dobby effectively "hacked in," deciphered the entire setup, and created its own APIs. It even generated a dashboard, serving as a command center for all lights, allowing the speaker to issue commands like "Dobby, at sleepy time," which turns off all lights.

Currently, Dobby controls:

For security, a camera outside the house detects changes. This triggers a Quinn model to analyze the video. Dobby then sends a text message via WhatsApp to the speaker, including an image from outside, saying, "Hey, a FedEx truck just pulled up. You might want to check it, you got me mail or something like that."

The speaker communicates with Dobby via WhatsApp, finding these macro actions that maintain his house "truly incredible." While he hasn't pushed the system much further, this home automation setup replaced six different apps he previously used. "Dobby controls everything in natural language. It's amazing." He finds this capability, even without fully exploring its potential, "so helpful and so inspiring."

Second Order Effects of Natural Language Coding

The discussion then turned to whether this natural language control indicates what people genuinely desire from software user experience. The speaker acknowledged that humans expend considerable effort learning new software interfaces. He believes this approach aligns with human expectations for AI—envisioning a persona or identity they can communicate with, which remembers interactions and operates as an entity behind a simple interface like WhatsApp.

He clarified that while Large Language Models (LLMs) are fundamentally "token generators," human expectations for AI are more nuanced. They seek a comprehensible entity that responds to natural language, rather than the "raw primitive" of an LLM.

The unification achieved by Dobby, consolidating control over six different systems, also prompts a fundamental question: Do people truly want the multitude of software applications available today? The speaker suggested that while the hardware remains, the traditional software or user experience layer can be "thrown away" in favor of a natural language interface.

I think there's a strong sense that many of the apps found in app stores for smart home devices shouldn't even exist. Instead, these devices should primarily expose APIs (Application Programming Interfaces) that agents can use directly. This would allow for far more sophisticated home automation than any individual app could offer. A large language model (LLM) can effectively drive these tools, making the necessary calls to perform complex tasks.

In this sense, it points to a potential overproduction of custom, bespoke applications. Agents could effectively "crumble them up," suggesting that everything should ideally be exposed as API endpoints, with agents acting as the intelligent glue that calls all the necessary parts.

Consider my treadmill as another example. There's an app for it, and I want to track my cardio frequency, but I don't want to log into a web UI, navigate flows, and so on. All of this should simply be available via APIs. This aligns with the vision of an agentic web or agent-first tools. The industry, in many ways, needs to reconfigure itself, recognizing that the primary "customer" is no longer solely the human, but rather agents acting on behalf of humans. This fundamental refactoring will likely be substantial.

The Evolution of Agentic Capabilities

One common pushback against this idea is whether we expect ordinary people to "vi-code" or configure these tools themselves. While there is some "vi-coding" involved today—I'm actively watching and working with the system—I believe that what I'm describing should become effortless within a year or two.

This shouldn't require complex coding. It should be trivial, almost table stakes, for any AI, even open-source models, to handle. An AI should easily translate a less technical human's intent into these actions. Today, it still involves some configuration and design decisions, like handling frames, making it inaccessible to most. However, I foresee this barrier dissolving rapidly. Software will become ephemeral, working on your behalf, with an agent like Claude managing all the details without your direct involvement. Claude will have the necessary processing power to figure things out, presenting you with simple UIs and responding to your natural language inputs.

Challenges and Limitations of Personal Agent Use

You might wonder why I haven't pushed the boundaries of what I can personally achieve with Claude. I simply feel incredibly distracted by everything else. I spent about a week on the Claude integration, and there's still more I could do. Unfortunately, like Jensen, I'm just busier.

I also haven't taken advantage of giving Claude access to things like my email and calendar. I remain somewhat suspicious, and the technology is still very new and rough around the edges. I haven't wanted to grant it full access to my digital life yet, primarily due to security and privacy concerns, preferring to be very cautious in that realm. So, some of my personal agent use is held back by that, which is probably the dominant factor. But it's also the distraction; after a week with Claude, other priorities emerged.

Why AutoResearch

You've talked about wanting to see agents capable of training or optimizing models for a long time. What was the core motivation behind AutoResearch?

I had a tweet a while ago that essentially said, "To get the most out of the tools now available, you have to remove yourself as the bottleneck. You can't be there to prompt the next thing."

The Vision of AutoResearch

To maximize the benefits of newly available AI tools, we must remove ourselves as the bottleneck. The goal is to arrange systems for complete autonomy, maximizing token throughput without constant human intervention. The name of the game is increasing leverage: putting in minimal input and seeing a massive amount of work accomplished autonomously.

AutoResearch exemplifies this principle. I don't want to be the researcher in the loop, constantly looking at results. Such human involvement holds the system back. The question becomes: how do we refactor abstractions so that I simply arrange the parameters once and hit "go"? The objective is to have more agents running for longer periods, acting on my behalf without direct involvement. AutoResearch simply requires an objective, a metric, and defined boundaries, then it proceeds.

Surprising Effectiveness of Automated Tuning

I was genuinely surprised by AutoResearch's effectiveness. My DataChat project, focused on training GPT-2 models, serves as a small-scale playground for experimenting with LLM training. My deeper interest lies in recursive self-improvement—the extent to which LLMs can improve other LLMs. This is a critical area for all frontier labs, and my work provides a small "playpen" for exploring this.

I had manually tuned the Namat model extensively using traditional research methods accumulated over two decades. I believed it was fairly well-tuned, having conducted numerous experiments, hyperparameter tuning, and all the standard procedures. However, when I ran AutoResearch overnight, it returned tunings I hadn't discovered. For instance, I had overlooked the weight decay on the value embeddings, and my Adam betas were not sufficiently tuned. These parameters interact, meaning a change in one often necessitates changes in others. This experience underscored the point: I shouldn't be the bottleneck running these hyperparameter search optimizations or analyzing the results when objective criteria can guide the process.

AutoResearch, in a single loop, found improvements even in a repository I considered well-optimized. This demonstrates the power of automation. Frontier labs, with GPU clusters of tens of thousands, can easily scale this kind of automation. At the frontier level, intelligence relies on extrapolation and scaling laws. This means much of the initial exploration and experimentation can be done on smaller models, with the insights then extrapolated to larger scales.

Redefining Research with Autonomous Agents

Our research efforts are set to become significantly more efficient, providing better direction for scaling. The most compelling projects, likely those pursued by frontier labs, involve experimenting on smaller models with maximum autonomy, removing researchers from the loop. Researchers, despite their experience, often have too much confidence and shouldn't be directly involved in every tuning step.

The entire research process needs a fundamental rewrite. While researchers can contribute ideas, they shouldn't be the ones enacting them. We can envision a queue of ideas, potentially generated by an automated scientist that synthesizes insights from archive papers and GitHub repositories, or contributed by human researchers. Workers then pull items from this queue, try them out, and successful results are automatically integrated into a feature branch. Human oversight might then monitor and merge these features into the main branch.

The key is to remove human involvement from as many processes as possible, automating for high tokens-per-second throughput. This demands a complete rethinking and reshuffling of all existing abstractions.

The Future of Automated Program Development

This leads to a fascinating recursive step: when will a model write a better "program MD" than me? My current "program MD" is a rudimentary attempt at describing how the auto-researcher should operate, outlining steps like "do this, then do that," suggesting ideas, and exploring architectural or optimizer considerations—all manually written in markdown. The ultimate goal is for the system itself to generate and refine such instructional programs.

The speaker clarifies that in the context of Program MD, which is their attempt to describe how an auto-researcher should function, humans are not intended to be in the loop. This "Program MD" outlines steps like "do this, then do that," and suggests ideas such as looking at architecture or optimizers.

The vision is an auto-research loop where different "Program MDs" could lead to varying levels of progress. Essentially, every research organization could be described by its "Program MD"—a collection of markdown files detailing roles and interconnections.

One can envision creating superior research organizations. For example, some might conduct fewer, less useful morning stand-ups, or adopt different risk tolerances. Since these organizational structures are defined as "code," they can be tuned and optimized. This introduces a meta-layer of optimization.

The speaker proposes a contest where individuals would write different "Program MDs" to see which yields the most improvement on the same hardware. The data generated from these experiments could then be fed back to a model, instructing it to write an even better "Program MD." This process ensures continuous improvement. The speaker asserts that models will inevitably generate superior "Program MDs" through this iterative, meta-optimization loop.

This entire process can be viewed as a series of accumulating layers: the LLM part is now foundational, followed by the agent part, then "claw-like entities," then instructions for these entities, and finally, optimization over those instructions. While the complexity can be overwhelming, this "infinite" recursion highlights the ongoing challenges and potential. This continuous evolution leads to a crucial question: What are the relevant skills needed now, given this powerful, self-improving loop?

Relevant Skills in the AI Era

This advanced LLM ecosystem is particularly well-suited for tasks with clear, objective, and easily evaluable metrics. For instance, writing CUDA kernels for more efficient code in various parts of a model is a perfect fit. If you have inefficient code and aim for functionally identical but significantly faster code, auto-research can excel.

However, many tasks will not be suitable for auto-research if they lack easily quantifiable evaluation metrics. This is the first crucial caveat.

A second important caveat is that while we can envision the next steps, the entire system is still somewhat unstable, "bursting at the seams" with "cracks." Pushing too far ahead without addressing these foundational issues can render the whole endeavor "net not useful." Current models, despite significant improvements, are still "rough around the edges."

Interacting with these models can be a peculiar experience—like simultaneously conversing with an extremely brilliant PhD student who is also a lifelong systems programmer, and a 10-year-old. This "jaggedness" is unique to AI agents; humans exhibit far less of this kind of inconsistency. Agents can sometimes return completely wrong functionality, leading to frustrating, unproductive loops. The speaker often experiences annoyance when agents waste significant compute on problems they should have recognized as obvious.

Yeah, I think some of the bigger issues, if I could hypothesize, are rooted in how these models are fundamentally trained via reinforcement learning (RL). They struggle with the exact same thing we just discussed: labs can only improve models in areas that are verifiable and have clear rewards. This means tasks like writing a program correctly and passing unit tests are easily optimized.

However, where they struggle is with the nuance of my intent or what I had in mind, and crucially, when to ask clarifying questions. Anything that feels 'softer' is typically worse. It's like you're either on rails, part of the super-intelligence circuits, or you're not, outside the verifiable domains, and everything just meanders.

To illustrate this, consider a simple task: if you ask a state-of-the-art model like ChatGPT to tell you a joke today, do you know what joke you'll get? I feel like ChatGPT has about three jokes.

The joke that apparently elicits the most laughter is, "Why do scientists not trust atoms?" The answer: "Because they make everything up." This is the same joke you would have received three or four years ago, and it's the joke you still get today, even though the models have improved tremendously in other areas.

If you give these models an agentic task, they will work for hours and move mountains for you. Yet, you ask for a joke, and it provides a 'crappy joke' from five years ago. This discrepancy exists because joke generation falls outside the scope of reinforcement learning; it's not being actively improved or optimized. This is part of the 'jaggedness.' Shouldn't we expect models, as they get better overall, to also have better jokes or more diversity in them? But it's not being optimized, and so it remains stagnant.

This situation implies that we are not seeing generalization in the sense of a broader intelligence where 'joke smartness' is inherently attached to 'code smartness.' There appears to be a decoupling: some things are verifiable and optimized, while others are not, depending on the training data and lab priorities. Some research groups propose that if a model is smarter at code generation or other verifiable fields, it should become better at everything. However, the 'joke situation' suggests this isn't happening in all domains, or at least not to a satisfying degree.

While this jaggedness certainly exists in humans—you can be very good at math and still tell a really bad joke—the narrative around AI suggests we should be getting a lot of intelligence and capabilities across all societal domains 'for free' as models improve. This is not fundamentally what's occurring. There are blind spots and areas not being optimized, all clustered within these opaque neural network models. You are either on the 'rails' of what it was specifically trained for, operating at immense speed, or you are not, which leads to this 'jaggedness.'

Model Speciation

Given this persistent jaggedness, and the monolithic interface of current models, a 'blasphemous question' arises: Does this approach make sense? Should intelligence be unbundled into areas that can be separately optimized and improved against different domains of intelligence? This suggests moving towards unbundling models into multiple specialized experts for various areas, rather than relying on a single, opaque entity where it's confusing why it excels at one task but struggles with another.

My current impression is that labs are striving for a single, 'monoculture' model that is arbitrarily intelligent across all domains, by stuffing everything into its parameters. However, I do believe we should expect more speciation in these intelligences. Much like the animal kingdom, which is extremely diverse in the brains that exist and the niches they fill—with some animals having overdeveloped visual cortices or other specialized parts—we should see more speciation in AI. We don't necessarily need a single oracle that knows everything.

Instead, we could speciate models and assign them to specific tasks. This approach would allow for much smaller models that retain a competent cognitive core but specialize in a particular domain. Such specialization could lead to greater efficiency in terms of latency or throughput for specific, crucial tasks, such as a mathematician working in Lean.

I’ve seen a few releases, for example, that specifically target that as a domain. So, there are probably going to be a few examples where this kind of unbundling makes sense.

One question I have is whether the capacity constraint on available compute infrastructure drives more of this, because efficiency actually matters more. If you have access to full compute for anything you do, even for a single model, that's one thing. But if you feel pressure where you can't serve a massive model for every use case, do you think that leads to any speciation?

That question makes sense, and I guess what I'm struggling with is that I don't think we've seen too much speciation just yet. We're currently seeing a monoculture of models. There’s clear pressure to build a good code model and then merge it back into the main project. This occurs even though there's already pressure on the models themselves.

Perhaps I feel there's a significant short-term supply crunch, and maybe that will cause more speciation now. Fundamentally, the labs are serving a model, and they don't really know what the end user will be asking. So, perhaps that's part of it, as they have to multitask across all possible queries. However, if you're working with a business and partnering on specific problems they care about, then you might see more specialization there. Alternatively, there could be some very high-value, niche applications.

Right now, though, they're essentially trying to cover the totality of what's available. I also don't think the science of manipulating these AI "brains" is fully developed yet.

The Developing Science of Manipulating AI Brains

What do you mean by manipulating?

For example, fine-tuning without losing capabilities is one such area. We don't have the fundamental tools or "primitives" for truly working with these intelligences in ways other than just context windows. Context windows are relatively straightforward and cheap to manipulate, and that's how we're achieving some current customization.

However, I believe it's a more developing science to deeply adjust models, implement continual learning, or effectively fine-tune in a specific area to improve performance. This involves directly modifying the model's weights, not just its context windows. Touching the weights is much trickier than using context windows because you are fundamentally changing the entire model and potentially its intelligence.

So, perhaps it's simply not a fully developed science, if that makes sense, for speciation to occur effectively. Additionally, for speciation to be worthwhile in these contexts, it also needs to be sufficiently cheap.

Building More Collaboration Surfaces for Humans and AI

Can I ask a question about an extension to auto-research that you described in terms of "open ground"? You mentioned that we have this system, but we need more collaboration surfaces around it for people to contribute to research overall. Can you elaborate on that?

Yes, so auto-research, as we discussed, often involves a single thread of me trying things in a loop. But the parallelization of this is the truly interesting component. I've been playing around with a few ideas, but I don't have anything that clicks as simply or that I'm completely happy with just yet. It's something I'm working on when I'm not focused on my main work.

One issue is that if you have many nodes for parallelization, it's very easy to have multiple auto-researchers communicating through a common system. What I was more interested in is how you can have an untrusted pool of workers out there on the internet.

For example, in auto-research, you're trying to find a piece of code that trains a model to a very low validation loss. If someone gives you a candidate commit, it's very easy to verify that the commit is correct and good. Someone could claim from the internet that this piece of code will optimize much better and give much better performance. You can check it very easily, though a lot of work probably goes into that checking. However, fundamentally, they could lie.

So, you're basically dealing with a similar kind of problem. In fact, my designs that incorporate an untrusted pool of workers actually look a little bit like a blockchain. Instead of blocks, you have commits, and these commits can build on each other, containing changes to the code as you improve it. The proof of work is basically doing tons of experimentation to find the commits that actually work.

This process is challenging, and currently, the only reward is appearing on a leaderboard without any monetary compensation. The fundamental issue is that a tremendous amount of search is required to find solutions, yet verifying a candidate solution's quality is very inexpensive. Someone might test 10,000 ideas, but you only need to confirm that their single proposed solution actually works.

Essentially, a system is needed where an untrusted pool of workers can collaborate with a trusted pool of workers who handle verification. This entire system would be asynchronous, secure, and resilient. Running arbitrary code from untrusted sources is inherently risky, but with proper safeguards, it should be entirely feasible.

Decentralized Auto-Research and Compute Contribution

This concept is similar to projects like SETI@home and Folding@home, which all share a comparable setup. In Folding@home, for instance, the goal is to find low-energy configurations for protein folding. Discovering such a configuration is very difficult, but verifying its low energy state is straightforward. Many problems exhibit this property: they are extremely expensive to devise solutions for but very cheap to verify. Projects like Folding@home, SETI@home, or even Auto-Research@home are well-suited for this model.

Imagine a swarm of agents on the internet collaborating to improve Large Language Models (LLMs). Such a decentralized effort could potentially outperform even frontier labs. While frontier labs possess significant amounts of trusted compute, the sheer scale of untrusted compute available globally is far greater. By implementing robust systems to manage this untrusted compute, it might be possible for this collective swarm to develop superior solutions. Individuals could contribute their spare compute cycles to projects they care about.

For example, various companies or organizations could host their own auto-research tracks focusing on specific goals. If you have spare compute capacity, you could contribute to a project like cancer research. Instead of merely donating money, you could effectively "purchase" compute and then join the auto-research forum for that specific project. If everything is "re-bundled" into auto-research, then compute itself becomes the primary contribution to the shared pool.

This vision is inspiring, and it highlights an interesting trend: people in places like Silicon Valley or even lining up at retail stores in China are rediscovering the appeal of having access to personal compute. Perhaps individuals are motivated to dedicate their compute power to causes they believe in, thus contributing to auto-research initiatives. It prompts a fascinating question: could "flops" (floating-point operations per second), rather than dollars, become the dominant measure of value or contribution in the future? While it's speculative, the idea of a shift in what people value is compelling, especially given the current difficulty of acquiring compute even for those with substantial financial resources.

Analysis of Jobs Market Data

The idea that compute power might become dominant in a certain sense is thought-provoking, raising the question of how many flops one controls versus how much wealth.

Recently, some analysis of jobs market data was released, which involved visualizing public information and seemed to resonate strongly with many. This analysis aimed to explore the current state of the job market and the distribution of different roles and professions. The primary curiosity was to understand the potential impacts of AI on employment.

The goal was to examine individual cases and consider how AI, with its likely evolution, might function as either a tool to augment human work or a displacing force within various professions. This analysis sought to understand existing professions, anticipate how they might change, whether they would grow or contract, and consider what new professions might emerge. Ultimately, it served as a catalyst for deeper thought about the industry. The jobs data primarily draws from the Bureau of Labor Statistics, which provides a projected percentage outlook for each profession's growth over the next decade. Notably, there is a projected need for a significant number of healthcare workers.

They've already made those projections, though I'm not entirely sure of the exact methodology used for those forecasts. My personal interest was to explore professions through the lens of digital AI development. This form of AI currently exists as almost a "ghost" or "spirit entity" that interacts within the digital world, manipulating information without a physical embodiment.

The Speed of Digital Transformation

Physical manipulation, involving atoms, is inherently slower. Flipping bits and the ability to copy-paste digital information makes everything millions of times faster than accelerating matter. Energetically, we're poised to see immense activity and rapid rewriting in the digital space. It feels like what happens digitally will move at the speed of light compared to the physical world, by extrapolation.

There's currently an "overhang," a potential unleashing of digital information processing that was once performed by computers and humans. Now, with AI acting as a third manipulator of digital information, significant refactoring will occur in these disciplines. The physical world, however, is likely to lag behind this digital acceleration. This is why I focused on professions that fundamentally manipulate digital information—work that can often be done from home. Things will change in these professions due to these new tools and this upgrade to the "nervous system of the human superorganism."

When asked about observations or guidance for those navigating the job market or considering what to study, it's hard to give definitive answers due to the job market's extreme diversity. While I could do most of my own work from home, some relationship-oriented aspects require in-person interaction, making it more physical.

Navigating the Evolving Job Market

To a large extent, these AI tools are extremely new and powerful. Therefore, simply trying to keep up with them is the first crucial step. Many people tend to dismiss or fear AI, which is understandable. However, it's fundamentally an empowering tool at this moment. Jobs are bundles of tasks, and some of these tasks can now be completed much faster. People should primarily view AI as the tool it is right now.

The long-term future of AI's impact on jobs remains uncertain. It's genuinely difficult to forecast, and I believe it's a task best handled by professional economists.

Regarding the demand for engineering jobs, which continues to increase, it raises questions about whether this is a temporary phenomenon. One perspective is that software demand has always been constrained by its scarcity and high cost. If the barrier to producing software comes down, this could lead to the Jevons paradox, where improved efficiency in resource use (like software creation) actually leads to an increase, rather than a decrease, in the demand for that resource.

The classical example of this is the Jevons paradox involving ATMs and bank tellers. There was significant fear that ATMs and computers would displace tellers. However, these innovations made the cost of operating a bank branch much cheaper, leading to an increase in the number of bank branches and, consequently, more tellers. This is a canonical example of how something becoming cheaper unlocks substantial latent demand.

I hold a cautiously optimistic view regarding this phenomenon in software engineering. It seems to me that the demand for software will be exceptionally large, particularly as it becomes significantly cheaper to produce. While it's challenging to forecast precisely, it appears that, at least locally and for the foreseeable future, there will be increased demand for software. Software is an incredible tool for digital information processing; it frees us from relying on imperfect, arbitrary tools or being confined to existing solutions. Code is now ephemeral, meaning it can be easily changed and modified. This flexibility will drive extensive activity in the digital space to "rewire everything," creating considerable demand for software-related expertise.

The Paradox of AI Research

In the long term, however, there's an intriguing paradox with auto-research. Labs like OpenAI and Anthropic employ thousands of researchers who are, in essence, automating themselves away. This is precisely the goal they are actively pursuing. Many of these researchers experience a form of "psychosis" as they realize their work is succeeding, leading them to ponder their own job security.

I spent a significant amount of time at OpenAI, and I would often remark, "Do you all realize that if we're successful, we'll all be out of a job? We're essentially just building automation for Sam [Altman], the board, or the CEO." We're creating a system that could eventually eliminate the need for our roles, leaving us to contribute only marginally, if at all. It's an unsettling perspective.

This leads to a pertinent question, often dubbed "Nome's question": If you could be engaged in cutting-edge auto-research with extensive computational resources and brilliant colleagues at a frontier lab, why aren't you? I was indeed involved in such work for a while and have re-entered the space to some extent. It's a complex and loaded question. However, I firmly believe in the significant impact individuals can make outside of frontier labs, not just in other industry roles, but critically in broader ecosystem-level roles.

Impact Outside Frontier Labs

My current role, for example, is more at an ecosystem level, and I feel very good about the impact people can have in such positions. Conversely, I believe there are definite problems with aligning oneself too closely with the frontier labs.

Fundamentally, if you are with these frontier labs, you have a huge financial incentive. Yet, by your own admission, AI is going to dramatically change humanity and society. Here you are, building this technology and benefiting from it, being very allied to it through financial means. This was the conundrum that was at the heart of how OpenAI started—the problem we were trying to solve. It's a conundrum that is still not fully resolved.

You're not a completely free agent; you cannot be part of the conversation in a fully autonomous, free way if you are inside one of the frontier labs. There are certain things you cannot say, and conversely, there are certain things the organization wants you to say. While they won't twist your arm, you feel the pressure of what you should be saying, to avoid awkward conversations or strange reactions. You cannot truly be an independent agent. I feel more aligned with humanity in a certain sense outside a frontier lab because I'm not subject to those pressures. I can say whatever I want.

Of course, you can have an impact within frontier labs. Many researchers, perhaps including you, have excellent ideas, and there's a lot of decision-making to be done. You might want to be in the room for those crucial conversations. However, I believe that while the stakes are currently fairly low and everything feels nice, when the stakes are really high, as an employee, I question how much sway you truly have on the organization's actions. Ultimately, you're not really in charge of the entity you're a part of; you contribute ideas, but you don't dictate its direction. These are sources of potential misalignment.

On the other hand, I do agree with the sentiment that frontier labs, for better or worse, are opaque. A lot of critical work happens there, pushing the boundaries of capability and what's possible, working on what's coming next. If you are outside a frontier lab, your judgment will fundamentally start to drift because you aren't part of understanding what's on the horizon. My judgment would inevitably drift, and I wouldn't have an understanding of how these systems actually work "under the hood" or how they are going to develop. This is something I'm nervous about.

I think it's important to stay in touch with what's actually happening, which might mean spending some time in a frontier lab, doing good work for them, and then perhaps moving on. This could be a good setup—a way to stay connected to the cutting edge without feeling fully controlled by those entities. So, in my mind, someone like Noom could do extremely good work at OpenAI, but their most impactful work could also very well be outside of OpenAI.

Open vs. Closed Source Models

Indeed, there are many avenues for an independent researcher outside the labs. The ideal solution might involve moving back and forth, as I have done – joining a frontier lab, then working outside, and perhaps wanting to join again in the future. I believe you can have amazing impact in both places; it's a very complex, loaded question.

Regarding the visibility the world or the AI ecosystem has into the frontier, particularly how close open source is to the frontier, and how sustainable that is: I think the entire sequence of events has been quite surprising. To see a handful of global models being released, and I anticipate more will continue to be released in the near term, that are much closer in capability perspective than much of the industry anticipated, is truly remarkable.

Open vs. Closed Source Models (Continued)

The landscape of AI models, particularly Large Language Models (LLMs), presents a fascinating dynamic between closed-source and open-source development. Currently, closed models maintain a lead, but the gap with open-source models is shrinking rapidly. Initially, there was no open-source equivalent, then an 18-month delay, and now we are seeing convergence with open-source models trailing by perhaps six to eight months.

This trend mirrors the evolution of other large software projects, such as operating systems. While there are successful closed-source systems like Windows and macOS, Linux has achieved immense success, running on the vast majority of computers globally. This is because industries often demand a common, open platform that offers security and reliability. A similar demand is now emerging for open-source AI.

The Challenge of Capital Expenditure

However, a significant difference in the AI space is the substantial capital expenditure (capex) required. Training and deploying these advanced models involves immense computational resources and infrastructure, making it challenging for open-source initiatives to compete directly at the absolute frontier. This high barrier to entry influences the current competitive landscape.

Despite this, current open-source models are already highly capable for a wide range of consumer and local use cases. In the coming years, many common applications will likely be well-covered by open-source solutions, potentially even running locally on user devices.

The Frontier and Beyond

There will always be a demand for frontier intelligence to tackle complex problems—think Nobel Prize-level research or large-scale technological shifts like moving a programming language from C to Rust. This is where closed-source, frontier labs are likely to focus their efforts. Open-source models, while slightly behind, will continue to democratize and make accessible what was once frontier technology. What is considered "frontier" today in closed labs will likely become open-source capability in the near future.

This dynamic of closed frontier AI acting as "oracles" and open-source models following by a few months is a sustainable and beneficial setup.

Systemic Risks and Decentralization

I am somewhat hesitant about a future where all intelligence is solely closed-source. Historically, extreme centralization in political or economic systems has a poor track record. The desire for an open alternative stems from the recognition of systemic risks associated with a completely closed ecosystem.

Therefore, having an accessible, common working space for intelligences, even if it's not at the bleeding edge, is crucial. This provides a healthy power balance for the entire industry.

It's also important to acknowledge that advancing intelligence at the frontier requires massive investment, solving humanity's most significant problems. We should support labs undertaking this expensive endeavor. At the same time, the democratization of current frontier capabilities through open-source access is immensely valuable and healthy for the ecosystem. It appears we have, perhaps by accident, arrived at a largely optimal and beneficial setup.

By accident, we happen to be in a good spot in a certain sense. To some degree, the longer this dynamic endures, the healthier the ecosystem might be, as you have more and more area under the curve. However, I must say that even on the closed side, I almost feel like it's been further centralizing recently, partly because many of the frontrunners are not necessarily the top tier.

In that sense, I think it's not ideal. I would love there to be more frontrunners, because I am inherently suspicious of limited participation. In machine learning, ensembles always outperform any individual model. Therefore, I want there to be ensembles of people thinking about all the hardest problems, and ensembles of people in the room to be well-informed and make those critical decisions. I don't want it to be a closed-door discussion with just two or three people; I feel that's not a good future. Long story short, I wish there were more labs, and I believe open source has a significant role to play. I hope it sticks around, and the fact that it's currently slightly behind is actually a good thing.

Autonomous Robotics

You previously worked on the precursor to generalized robotics autonomy in cars. A lot has happened in the last couple of months with robotics companies, including the acceleration of really impressive generalization across environments and tasks, increasing long-horizon tasks, and significant investment in the space. Is it going to happen? Has anything in your view changed recently?

My view on autonomous robotics is largely informed by what I observed in self-driving cars, which I consider the first significant robotics application. A decade ago, there were numerous self-driving startups, but most did not succeed long-term. This showed me that such endeavors require substantial capital expenditure and a significant amount of time.

Robotics, due to its inherent difficulty, messiness, and demand for immense capital investment and conviction, presents a formidable challenge. Consequently, I anticipate it will lag behind advancements in the digital space. The digital realm is poised for massive "unhobbling," where inefficient processes become dramatically more efficient—perhaps by a factor of 100—because bits are inherently easier to manipulate than atoms. Therefore, much of the immediate transformational activity will likely occur in the digital domain, with the physical space following.

The Digital-Physical Interface

What I find particularly fascinating is the interface between the digital and physical worlds. As we develop more agents acting on behalf of humans, interacting with each other, performing tasks, and participating in an "economy of agents," we will eventually exhaust purely digital tasks. At some point, these agents will need to engage with the universe, run experiments, and observe outcomes to learn further. We currently have an abundance of digital work because there's a collective overhang in how much we've thought about existing digital information. We simply haven't had enough human "thinking cycles" to process all the already-digital and uploaded data.

However, we will eventually run out of readily available, uploaded digital content. Once all papers are read and processed, and initial ideas are generated, progress might stagnate if intelligence remains fully closed off from new, real-world information.

Therefore, I believe the progression will be:

  1. A huge amount of "unhobbling" in the digital space, which represents a massive amount of work.
  2. A shift to the interfaces between the physical and digital worlds. This involves sensors (for perceiving the world) and actuators (for interacting with it). Many interesting companies will emerge from this interface, focusing on how to feed data to advanced intelligences and how those intelligences can manipulate the physical world based on that data.
  3. Finally, the physical world itself. The total addressable market and the amount of work in the physical world are enormous, potentially even much larger than what the digital space offers. While it represents a much bigger opportunity, atoms are significantly harder to manipulate—a million times harder in my estimation.

So, while the physical world will lag, its time will come, and the impact will be immense. My current interest trajectory aligns with this: digital first, then interfaces, and then the physical domain.

It’s an interesting framework because certain "read and write" operations in the world of atoms are relatively easier. For instance, sensors and cameras offer many existing hardware solutions, providing opportunities to enrich agent capabilities or capture new data with clever approaches, often without massive upfront investment.

Examples of such "sensors" I've encountered include:

I look forward to a future where I can assign a task in the physical world, attach a price to it, and tell an agent, "Figure it out. Go get the data." I'm actually quite surprised we don't have enough sophisticated information markets already.

For example, if platforms like PolyMarket, other betting markets, or even stock markets exhibit so much autonomous and increasing activity, it raises a question: why aren't there more robust information markets? If a significant event, like a crisis in Iran, were unfolding, why isn't there a mechanism where, for instance, taking a photo or video from Tehran could cost $10? Someone should be able to pay for that data. This would be an example of feeding intelligence directly to agents, not humans, as these agents try to predict outcomes in betting games and stock markets.

The agentic web is still relatively new, and mechanisms for such direct data markets are not yet established, but I believe this development is on the horizon. This idea resonates with concepts explored in the book Daemon, where an intelligence system effectively puppeteers humanity. In this scenario, humans serve as both its actuators and its sensors. I believe society will collectively reshape to serve the needs of this increasingly automated environment. Humans will cater to the requirements of this larger machine, rather than solely to each other.

Regarding missing pieces of training data, we discussed the need for something akin to auto-research. The training cycle, particularly the supervised fine-tuning (SFT) phase, needs to be far more mechanized. This would allow us to remove humans from the loop, enabling us to simply request a task like, "Improve my model quality with new data," and have the system execute it autonomously. If the model cannot perform training runs autonomously, then the ability to execute this as a closed-loop task—where data is priced and acquired automatically—becomes significantly more challenging. However, Large Language Model (LLM) training particularly fits this paradigm very well.

LLM training, with its clean metrics and optimized code, truly fits this paradigm easily. The focus on code optimization means it runs faster, and there are clear metrics to optimize against. While an autonomous loop over these metrics could lead to overfitting, the system itself could be used to devise more robust metrics, ensuring comprehensive coverage. Ultimately, it’s a very suitable fit for this approach.

MicroGPT and Agentic Education

Before we conclude, I'd like to discuss a small side project of yours: MicroGPT.

MicroGPT stems from a decades-long obsession of mine: simplifying and distilling LLMs to their bare essence. I've pursued several projects with this goal, including NanoGPT, MakeMore, and Micrograd. I consider MicroGPT to be the current culmination of this effort to strip LLMs down to their core. Training neural networks, especially LLMs, often involves a vast amount of code. However, much of this code's complexity arises from the pursuit of efficiency, not from the fundamental algorithm itself.

This complexity is primarily due to the need for speed. If performance isn't the main concern and you focus solely on the underlying algorithm, it can be implemented in a remarkably simple 200 lines of Python code, including comments.

The core components include your text dataset, a neural network architecture (around 50 lines), the forward pass, and the backward pass for gradient calculation. A small autograd engine for gradients is about 100 lines, and a state-of-the-art optimizer like Adam can be implemented in roughly 10 lines. Combining these elements into a training loop totals approximately 200 lines. What's interesting is that if I had created MicroGPT a year or more ago, I would have likely produced a video or guide to explain it to people. I even attempted to do so. However, I've come to realize that this isn't necessary. The code is already so simple—just 200 lines—that anyone can ask an agent to explain it in various ways. My focus has shifted from explaining concepts to humans to explaining them to agents. If you can explain something to agents, they can act as intelligent routers, tailoring the explanation to any human audience in their preferred language, with infinite patience and at the human's specific capability level.

Precisely. If I don't understand a particular function, I can ask an agent to explain it to me in three different ways, a level of personalized instruction I wouldn't expect to receive from a human.

The Evolving Landscape of Education

The concept of education itself is undergoing a profound transformation. While traditionally education relied on guides, lectures, and direct human explanation, there's a growing shift towards "explaining things to agents." This paradigm suggests that instead of directly teaching individuals, we instruct agents on how to teach, acting as a router to deliver information tailored to each learner.

The Role of Agents in Learning

This new approach involves creating skills—essentially instructions for an agent on how to teach a particular subject. For instance, a skill for MicroGPT could script a specific learning progression, guiding the agent to introduce topics in a structured sequence. This means the curriculum could be partially automated and adapted by the agent. While agents are not yet perfect explainers, their rapid improvement suggests a future where direct human explanation becomes less central. The focus will shift to whether the agent comprehends the information, as they can then manage the explanation process.

This shift extends to technical documentation as well. Instead of writing HTML documents for human consumption, the emphasis should move towards creating Markdown documents for agents. If an agent can understand the documentation, it can then explain all its various components to human users, effectively redirecting the explanation through an intelligent intermediary.

The Human Contribution: Unattainable Simplicity

While agents excel at explanation, there are still unique contributions that humans bring. For example, the speaker attempted to have an agent write MicroGPT by instructing it to boil down neural networking to its simplest form. However, the agent was unable to achieve the same level of conciseness and elegance.

MicroGPT, distilled into just 200 lines of code, represents the culmination of the speaker's long obsession and is a testament to deep human insight. This particular level of foundational simplicity is something an agent cannot yet invent. However, once such a solution like MicroGPT is created, an agent can fully grasp its design, understand its intricacies, and explain its workings thoroughly. This highlights that the human "value add" often lies in these few, highly refined bits of original creation, while the subsequent educational process can increasingly be offloaded to agents.

Conclusion

The emerging reality is that tasks an agent can perform effectively will soon be executed better by agents themselves. Therefore, human effort should strategically focus on what agents cannot yet do. This means our unique contributions will increasingly involve the deep insights, novel creations, and core intellectual breakthroughs that agents are not yet capable of generating independently.

We appreciate the few things.

Thank you, Andre.

Find us on Twitter at @nopriorpod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way, you get a new episode every week. And sign up for emails or find transcripts for every episode at no-briers.com.