Under the LF AI & Data Foundation, Delta Lake aims to accelerate innovation and enhance data reliability and performance on a global scale.
We are excited to announce Delta Lake has joined the LF AI & Data Foundation as an incubation project. Databricks, the founder of Delta Lake, serves as a Premier Member and Board Member of LF AI & Data demonstrating its commitment to open source AI/ML solutions as well as growing and supporting the project’s community.
As an LF AI & Data-hosted project, Delta Lake will benefit from a vibrant community of over 100K developers across all hosted projects, a shared commitment to open source principles, and access to a wide variety of resources in support of the project and its community.
This transition is expected to foster greater collaboration between Delta Lake and other open source AI and Data ecosystem projects, and accelerate the creation of cutting-edge tools and frameworks for AI and Data analytics.
The integration with LF AI & Data underscores the importance of open source in driving technological progress. It provides Delta Lake with a broader platform to reach more developers and organizations, facilitating the development of robust, scalable data solutions.
This partnership, along with the Linux Foundation’s commitment to enabling a thriving open source AI ecosystem, promises to bring new advancements in data lake technology, enabling more efficient and effective data management practices. Delta Lake’s alignment with LF AI & Data marks a pivotal moment in the open source community, paving the way for advanced open source AI technologies hosted by neutral organizations and managed via an open governance model, further accelerating innovations in AI and data.
LF AI & Data, an umbrella foundation of the Linux Foundation, is dedicated to fostering open source innovation in AI and data. It was established to support and sustain an open source AI and data ecosystem, facilitating the development of AI and data products and services with open-source technologies. LF AI & Data promotes collaboration in a neutral environment with open governance, accelerating the harmonization of technical projects across the AI and data communities.
Delta Lake is at the forefront of optimizing storage for lake houses, setting itself apart from existing data lake solutions. By introducing a scalable metadata layer to data stored in Parquet format, it significantly improves efficiency and scalability. This innovation allows users to utilize any engine on a single copy of data, simplifying operations and enhancing interoperability across various platforms.
With more than 10,000 companies using Delta Lake (including > 60% of Fortune 500) and 20 million downloads each month, its impact is both profound and widespread. The community around Delta Lake is robust and continually expanding, providing support to a multitude of other open source data systems including delta-rs, delta-go, Trino, PrestoDB, Google BigQuery, DuckDB, Polars, Apache Druid, and many more.
“Delta Lake not only enriches the LF AI & Data portfolio of technical projects but also strengthens our mission to build transparent, community-driven technology solutions, under the auspices of a neutral organization. We are excited about its progression under an open governance model, which is pivotal in attracting developers, organizations, and enhancing collaboration across the open source AI and Data ecosystem.”- Ibrahim Haddad, Executive Director of LF AI & Data Foundation
The LF AI & Data Foundation is dedicated to offering a framework that supports long-term stewardship by a community deeply invested in Delta Lake’s success. This transition to LF AI & Data marks a significant milestone, paving the way for greater achievements in the world of data and AI.
To learn more about the Delta Lake project, visit delta.io or check out the Delta Lake roadmap.
