The Future of AI and Data: A Conversation with Matei Zaharia

The convergence of advanced AI models and readily available data is poised to revolutionize traditional software development. As artificial intelligence models demonstrate increasingly sophisticated reasoning capabilities, the paradigm is shifting towards leveraging these powerful tools by simply ensuring the right data is in place. This approach promises to unlock unprecedented innovation, but its success hinges on the foundational availability and accessibility of that data.

Introduction to Databricks and the Data AI Summit

Matei Zaharia, co-founder of Databricks, joins us to discuss the company's recent innovations and the evolving landscape of data and AI. He reflects on the humble beginnings of the Databricks Data AI Summit, which started as a small meet-up at Berkeley with just 50 attendees focused on teaching Spark. Today, the summit has grown into a global phenomenon, attracting over 100,000 participants worldwide, with 30,000 attending in person. Zaharia also praises Ali Ghodsi, CEO of Databricks, for his exceptional leadership and presentation skills, noting that Ghodsi's growth as a CEO has been remarkable, driven by a deep commitment to understanding and mastering every facet of the business.

Databricks has recently launched a suite of new initiatives, including Omnigents, El Tap, Dream Engine, Genie, Customer League, and the acquisition of Panther, alongside advancements like Open Sharing and the Unity AI Gateway. While many of these align with expected industry trends, Omnigents and El Tap stand out as particularly unique and differentiated.

Omnigent: Building the Agent Infrastructure Layer

The development of Omnigent was driven by several converging factors. Internally, Databricks' developer infrastructure team created Isaac, a wrapper around coding agents like Cloud Code and CodeX, enabling their use across various environments. Advanced engineers began building complex agent workflows and custom UIs on top of this infrastructure. Simultaneously, the research team, co-led by Zaharia, developed agents like Genie, a data science agent, and numerous internal and customer-facing agents. A common challenge emerged: the need to frequently switch models and manage agent sessions, history, and search capabilities for effective collaboration.

Zaharia recognized that the problems faced in building coding agents and custom agents were fundamentally the same. The goal was to create a platform that facilitates the delivery and control of agents, ensuring security and portability across different models and environments. This led to the development of Omnigent, which prototypes and builds upon these core challenges.

This architectural approach draws parallels to the evolution of operating systems and network protocols, emphasizing the need for standardized interfaces and interoperability. The concept of "Open Sharing," where companies can share real-time views of their data tables with partners, exemplifies this need for structured data access, moving beyond ad-hoc communication methods like emails and spreadsheets.

The inspiration for Omnigent also stemmed from personal experiences. Zaharia recounts a period of intense coding where he was constantly tethered to his laptop, even while driving, to keep agent sessions running. This frustration highlighted the need for a more robust and persistent environment, leading to the development of cloud sandboxes that don't shut down and can be provisioned quickly for both agentic sessions and general development. The ability to interact with a shell, list files, and tail logs within these sandboxes was a crucial feature, reminiscent of accessing a mainframe. The need for a seamless markdown rendering experience within the agent environment was also identified.

Furthermore, the collaborative aspect was paramount. Engineers building powerful individual agent setups realized the limitations of solo development. Omnigent aims to provide a server-based solution for secure, collaborative agent development and deployment, addressing the security concerns that often arise when agents need to access sensitive company data.

Agent Clouds, Common APIs, and Open Source Strategy

Omnigent is envisioned as an "Agent Cloud," a platform with a runner component and a server component, featuring a uniform API. This open-source platform allows users to deploy agents on their own machines or leverage hosted collaborative agents. The open-source approach is strategic, fostering a network effect where a broad community can contribute integrations and extensions. This mirrors the success of Spark, which benefited immensely from an open ecosystem of libraries and data source connectors.

Databricks' decision to open-source Omnigent is rooted in the belief that certain foundational layers benefit most from widespread collaboration. By providing a common API and framework, they aim to enable developers to build and customize agents more easily. This contrasts with proprietary solutions that might offer specific advantages but lack the broad ecosystem support that open-source initiatives can cultivate.

The platform has already seen significant community engagement, with numerous pull requests and integrations emerging shortly after its release. These include support for Kubernetes deployments and various cloud sandbox integrations, allowing agents to run in secure, isolated environments.

The Modern AI Stack and Unified Data Management

The emergence of Omnigent and similar initiatives suggests a potential shift towards a "modern AI stack," analogous to the "modern data stack" that revolutionized data management. The modern data stack decomposed data workflows into distinct layers for ingestion, transformation, warehousing, and visualization. However, this often led to fragmentation and the need to integrate multiple vendors. Over time, the industry has moved towards consolidation, with platforms offering more integrated capabilities.

Similarly, the complexity of orchestrating multiple agent frameworks to achieve simple AI tasks is becoming apparent. Omnigent's core contribution is a common API that abstracts away the underlying agent harnesses, providing a unified interface for sending messages, files, and receiving streaming text or tool calls. This abstraction layer, which maps various models and SDKs to a single interface, is a significant value proposition, reducing the maintenance burden for developers when underlying model APIs change.

Databricks Scale and Internal AI Workflows

Databricks operates at an immense scale. The company launches approximately 50 to 60 million virtual machines daily across three major clouds, processing exabytes of data. This vast operational capacity provides a unique vantage point for understanding and optimizing AI workloads.

Internally, Databricks leverages its own platform for AI development. The AI team has access to extensive resources, including significant token capacity and infrastructure, enabling them to work with large internal codebases and experiment with new AI tools. This internal adoption and testing are crucial for refining the platform and identifying areas for improvement.

Agent Security, Governance, and Spend Controls

Security, governance, and cost management are critical concerns in the current AI landscape. Omnigent addresses these by introducing contextual policies, which go beyond simple allow/disallow rules. These policies track the state of an agent session, allowing for more nuanced decision-making based on the agent's actions. For instance, an agent might be permitted to install new packages, but if it installs a recently released, potentially compromised package, the policy can restrict further actions.

This stateful approach enhances both security and usability. The platform also incorporates libraries that parse low-level events into high-level actions, enabling policies to be written based on more meaningful metrics. This is where acquisitions like Panther, which focuses on event processing, play a role.

Furthermore, Omnigent tracks session spending, allowing users to set budget caps for agent activities. This feature enables granular control over AI costs, preventing unexpected overruns and providing transparency into resource consumption. The integration of these security and cost controls reflects Databricks' extensive experience in data governance, particularly with its Unity Catalog.

LTAP: The Dream Engine for Unified Databases

The concept of LTAP (Low-Latency Transactional Analytics Processing) represents Databricks' vision for a unified database engine. Historically, databases have been bifurcated into OLTP (Online Transaction Processing) for transactional workloads and OLAP (Online Analytical Processing) for analytical queries. This separation often leads to complex data pipelines, including Change Data Capture (CDC), which are notoriously brittle and prone to failure.

The industry has long sought a single database system that can efficiently handle both transactional and analytical workloads – the "holy grail" of database engineering. However, previous attempts have often resulted in compromises, leading to suboptimal performance in both areas or the creation of proprietary ecosystems.

LTAP aims to achieve this unification by focusing on a single storage layer. By writing data in a column-oriented format, such as Parquet, directly to an open data lake, analytics can access data with minimal latency, eliminating the need for complex replication and CDC pipelines. This approach allows for real-time reasoning on data, making it immediately available for analysis and AI applications.

The breakthrough for LTAP came from an unexpected observation: Databricks' storage fleet had idle CPUs that could be utilized for transcoding data from row-oriented formats (ideal for OLTP) to column-oriented formats (ideal for analytics). This process not only enables unified data access but also improves compression, leading to faster writes to object stores.

Databricks' Culture of Fast Prototyping and Incremental Evolution

Databricks fosters a culture of rapid prototyping and innovation. The development of LTAP, for instance, was driven by an engineer who independently prototyped the Parquet storage solution, demonstrating the company's commitment to empowering its engineers to pursue bold ideas. This approach, combined with a focus on hiring top talent and maintaining a coherent product strategy, allows Databricks to deliver complex solutions efficiently.

The company emphasizes an incremental approach to product development, prioritizing unification and adding capabilities one at a time. This strategy, exemplified by the evolution of Spark and Delta Lake, ensures that new features are well-integrated and deliver immediate value. The tight feedback loop with target customers is crucial, ensuring that products are built to address real-world needs.

The Dream Engine Vision: Rewriting the Database Stack

The "Dream Engine" project represents a fundamental rewrite of database engines, acknowledging that many existing analytical databases are a decade old and have accumulated technical debt through incremental additions. The goal is to design a new engine from the ground up, leveraging a decade of operational experience and billions in revenue.

This ambitious undertaking faces the "second system effect," where ambitious second projects can fail due to overreach. However, Databricks' engineering team, comprised of experienced database engineers, is well-equipped to tackle this challenge. They are employing a novel approach by building a "factory" for database engines. This factory uses machine learning models trained on trillions of data points from past query traces to predict the performance of different algorithms and data structures for specific workloads. This allows for the selection of the most optimal algorithms and data structures at both implementation and runtime, ensuring high performance across diverse use cases.

Vector Databases, Query Engines, and the Future of Data

The conversation touches upon the evolving landscape of specialized databases, including vector databases and transactional accounting databases like TigerBeetle. The consensus is that many of these specialized categories may eventually converge into more general-purpose data platforms. The emphasis is shifting from collapsing query layers to unifying storage layers, allowing for greater flexibility and interoperability.

The idea of a single query language is also discussed, with the conclusion that for AI agents, the ability to access data efficiently is more critical than a universal query language. Agents can adeptly handle different SQL dialects, provided the data is accessible.

Databricks vs. Snowflake: Openness and AI Focus

A key differentiator between Databricks and Snowflake is Databricks' commitment to open standards and its early focus on AI and machine learning. While both companies adopted a separation of storage and compute architecture, Databricks has consistently championed open formats like Parquet and Delta Lake. This open approach fosters a broader ecosystem and prevents vendor lock-in, a significant concern for enterprises.

Databricks' platform was built with machine learning use cases in mind from its inception, positioning it to capitalize on the AI revolution. This contrasts with Snowflake's initial focus on serving smaller, curated datasets for business users. Databricks' strength in large-scale batch processing and data ingestion, combined with its open data formats, has enabled it to evolve more effectively into a comprehensive data and AI platform.

The Role of Context and Specialized Models

The concept of "context as the new oil" is explored, highlighting how domain-specific data, combined with advanced AI capabilities, can provide a significant competitive advantage. Databricks' approach to AI development focuses on making AI models useful by enabling them to query and understand company-specific data. This includes developing specialized models for tasks like document parsing and creating agents like Genie, a virtual data scientist.

While Databricks is developing its own models, such as DBRX, the emphasis is on creating specialized models that outperform general-purpose models for specific high-volume use cases. This includes fine-tuning models for tasks like document understanding and developing advisor models for coding agents. The ease of model customization is expected to increase over time, driven by advancements in RL fine-tuning and synthetic data generation.

The Data + Agents Paradigm: Rewriting Software

The core thesis remains that with data in the right place, AI models and agents can unlock significant value. This paradigm shift is expected to lead to the rewriting of many traditional software applications. By leveraging agents on top of well-organized data, organizations can achieve remarkable results. This approach is being applied to various domains, including security and customer data platforms, where Databricks is launching new products designed to simplify workflows and enhance decision-making.

Key Takeaways