As we reach the midpoint of 2024, we want to take a moment to reflect on the remarkable progress and growth the LF AI & Data Foundation has experienced. In just six months, we have welcomed numerous new members, introduced several innovative projects, and launched exciting initiatives that promise to shape the future of AI and data technologies. Special thanks to our Governing Board Members and all the members of LF AI & Data and project communities, without whom this work would not have been possible.
A Message from Ibrahim Haddad PH.D., Executive Director of LF AI & Data
As we celebrate the first half of 2024, it’s clear that the LF AI & Data Foundation is on an incredible trajectory. Our growing membership and the addition of new projects are a testament to the value we bring to the AI and data community. Looking ahead, we anticipate even more robust growth and innovation as we continue to support our members and projects. GenAI has become a focal point of both excitement and scrutiny. Generative AI and large language models (LLMs) will reshape how we live, work and do business.
In the The State of Generative AI and Open Source Innovation in Enterprises report published in January 2024, A global survey conducted by LF AI & Data in collaboration with Linux Foundation Research delves into GenAI’s integration into businesses and its openness. What we found is that there is growing adoption of open source applications and GenAI profoundly affects operations, especially in product development and enhancement. This highlights the shift toward open-source solutions and their advantages in collaboration, transparency, and integration. The survey results also validates the crucial need for neutral governance frameworks to guide responsible and ethical innovation in the GenAI domain.
The Role of Open Source
Businesses are concerned about the openness of GenAI technologies. “Around two-thirds of respondents are extremely or moderately concerned, reflecting the importance of transparency and control.”
Survey respondents generally favor open source. “This highlights the benefits of open source technologies, including transparency, reproducibility, diverse data access, and ease of integration. Security concerns do not deter open source GenAI adoption.”
Neutral governance is key to GenAI development. “It fosters responsible growth, ensures widespread benefits, and aligns with societal values, maintaining integrity and sustainability.”
For the remainder of 2024, the Foundation’s goal is to maintain its position as a leading advocate for open source in AI and data through promoting thought leadership and industry awareness. We aim to build on these pillars by:
- New Members and Projects: Strengthen and grow the LF AI & Data community to support an expanding ecosystem, offer more opportunities for collaboration across open source projects to foster an inclusive environment for innovation.
- Maintain Transparent and Neutral Governance: Ensure that governance remains transparent and neutral, which is essential for the responsible growth and integrity of GenAI advancements.
- Promote Open Source Benefits: Advocate for open source in AI and data to help organizations reduce technical debt and avoid vendor lock-in, while highlighting the advantages of transparency, reproducibility, and ease of integration.
- Drive Thought Leadership and Industry Awareness: Continue to promote thought leadership and industry awareness to solidify LF AI & Data’s position as a leading advocate in the field, ensuring organizations are informed about the benefits and best practices of open source GenAI technologies.
Additionally, we relaunched the Outreach Committee to enhance our community engagement efforts. This committee is crucial in maximizing the outreach and visibility of LF AI & Data, ensuring our initiatives resonate throughout the industry.
2024 New Members
The value of open source is becoming increasingly evident to organizations worldwide, driven by legislative trends and growing concerns about software transparency. Open source allows companies to build on robust, community-driven foundations, shaping the future of technology.
In the first half of 2024, we welcomed Databricks, Microsoft, Neural Magic as Premier members and several other General and Associate members to our membership roster, furthering our mission:
Project Growth & Velocity
LF AI & Data project growth metrics over the past 12 months reflect development efficiency, community engagement, and momentum toward achieving goals.
LF AI & Data projects bring unique strengths and potential, and we are committed to supporting their momentum within our ecosystem. Here are some highlights from our projects in the first half of 2024 include:
Graduated Projects
 |
DocArray, a Python library that facilitates the representation, transmission, storage, and retrieval of multimodal data graduated from an incubation to graduate level. Learn more. |
|---|---|
 |
Marquez is an innovative open source metadata service designed for the collection, aggregation, and visualization of a data ecosystem’s metadata an incubation graduated from an incubation to graduate level. Learn more. |
New Projects
 |
Delta Lake, an open-source storage framework that enables building a format agnostic Lakehouse architecture with compute engines including Spark and is an incubation-stage project. Learn more. |
 |
IREE, a versatile and adaptable suite of open source tools designed to improve the deployment of ML model representations across various platforms, was announced as a new sandbox project for machine learning deployment. Learn more. |
 |
Monocle is redefining how developers monitor generative AI (GenAI) applications, addressing the growing needs within the dynamic GenAI ecosystem and is a sandbox project. Learn more. |
 |
The mission of the Open Platform for Enterprise AI (OPEA) Project is to develop an ecosystem orchestration framework to efficiently integrate performant GenAI technologies and workflows leading to quicker GenAI adoption and business value. Learn more. |
 |
vLLM is a high-throughput, memory-efficient inference and serving engine for Large Language Models (LLMs). Learn more. |
 |
Unity Catalog, is the industry’s only unified and open governance solution for data and AI, built into the Databricks Data Intelligence Platform. Learn more. |
Generative AI Commons
Launched in September 2023, Generative AI Commons has been at the forefront of advancing the ethical use and democratization of generative AI technologies. Under the elected leadership of Anni Lai of Futurewei as Chair and Arnaud Le Hors of IBM’s Open Technologies division as Vice Chair, the initiative has made impactful advancements in the last six months. The initiative operates through five specialized workstreams:
- Applications
- Models & Data
- Education & Outreach
- Responsible AI
- Frameworks
Noteworthy achievements include the introduction of several tools and publications designed to enhance AI community resources:
- The Model Openness Framework (MoF): The Model Openness Framework (MOF) is a comprehensive framework for objectively evaluating and classifying the completeness and openness of machine learning models. It does so by assessing which components of the model development lifecycle are publicly released and under what licenses.
- The Model Openness Tool (MoT): To implement the MoF, Generative AI Commons created the Model Openness Tool (MOT). This tool evaluates each criterion from the MOF and generates a score based on how well each item is met. The MOT provides a practical, user-friendly way to apply the MOF framework to your model and produce a clear, self-service score. The MOT presents users with 16 questions about their model. Users need to provide detailed responses for each question. Based on these inputs, the tool calculates a score, classifying the model’s openness on a scale of 1, 2, or 3.
- Generative AI Commons Glossary: A living document evolving through community contributions to stay current with field developments. The LF AI & Data Outreach Survey on open source and generative AI captured insights from software developers, data scientists, researchers, and students, shaping initiatives to meet technological needs.
- LF AI & Data Outreach Survey: This survey captured insights from software developers, data scientists, researchers, and students, to help shape Generative AI Commons initiatives to meet technological needs.
Building on these advancements, a notable collaboration for Generative AI Commons this year has been with the Department of Commerce’s AI Safety Institute Consortium (AISIC). By joining this initiative, LF AI & Data aligns with over 200 key AI stakeholders to support the development of trustworthy AI technologies. This partnership is crucial as it prepares the U.S. to manage risks associated with advanced AI models and technologies.
The Generative AI Commons has been actively expanding its educational offerings to enhance understanding and foster responsible usage of AI technologies. Below is a summary of our recent publications and webinars aimed at broadening knowledge within the AI community:
Publications:
- White Paper: “Conversational AI Technology and Remote Patient Monitoring (RPM)” – This paper delves into the use of AI in healthcare, particularly in monitoring patient health remotely, offering insights into practical applications and benefits AI” – A session dedicated to discussing the significance of transparency and openness in the development and use of AI technologies.
- “Demystifying Gen AI” – This webinar aims to clarify the complexities and misconceptions surrounding generative AI, making the technology more accessible to a wider audience.
- “Strategic AI Adoption: Leadership Insights for the Modern Enterprise – Trustmark Webinar” – A detailed exploration of how businesses can strategically implement AI, with expert insights on leadership and enterprise management in the context of AI adoption.
Webinars:
Social Media Highlights
The growth and success of LF AI & Data Foundation are not just reflected in our expanding membership and new projects but also in our vibrant social media presence and connection to the AI & Data communities.
Our LinkedIn audience has grown by over 26% in 2024. This significant increase in followers demonstrates the growing interest and trust in LF AI & Data as a source of valuable information and insights in the AI and data space.
The number of shares on LinkedIn has increased by over 66%. This boost in shares indicates that our content is not only reaching more people but is also resonating with them. More shares mean that our posts are deemed valuable enough to be shared within personal and professional networks, amplifying our reach even further.
One of the most remarkable metrics is the 215% increase in impressions over the last six months. This means our strategic approach to content creation and distribution is highly effective, ensuring that our messages reach a broader audience with minimal additional effort. In addition, we saw an increase of interactions from our content increase by over 110%.
AI_Dev Conferences- The conference for Open Source AI
AI_dev Paris
The Linux Foundation and LF AI & Data AI launched a new open source AI conference at the end of 2023 co-locating with Casandra. The AI_dev conference is positioned as a hub for developers exploring open source generative AI and machine learning. Central to this event is our belief in open source as the driving force behind AI innovation. By bringing together top developers globally, we aim to spark discussions, encourage collaborations, and influence the future of open source AI. Highlights include:
AI_dev North America
Later this summer and fall, LF AI & Data will host two additional events in Hong Kong and Tokyo, solidifying its role as the leader for developers exploring the intricate realms of open source AI and ML. We’d like to invite your organizations to participate in these AI conferences as leaders in open source Generative AI.
Looking Ahead
Looking back on the first half of 2024, LF AI & Data has achieved remarkable growth and success. With expanding membership, groundbreaking projects, and a vibrant social media presence, our future looks promising. These achievements reflect the dedication of our community and underscore the impact of open source in AI and data.
But this is just the beginning. The second half of 2024 promises even more opportunities for innovation, collaboration, and growth. We are committed to supporting our members and projects, fostering an inclusive and dynamic community, and leading the charge in open source AI and data technologies.
We invite you to join us on this exciting journey. Whether you have a model, project, or innovative idea, there are numerous ways to get involved with LF AI & Data. Together, we can drive the future of open source AI and data technologies, creating a more transparent, collaborative, and innovative world.
LF AI & Data Resources
- Learn about membership opportunities
- Explore the interactive landscape
- Check out our technical projects
- Join us at upcoming events
- Read the latest announcements on the blog
- Subscribe to the mailing lists
- Follow us on Twitter or LinkedIn
