LF AI & Data Foundation—the organization building an ecosystem to sustain open source innovation in artificial intelligence (AI) and data open source projects, today announced the graduation of the Marquez project.
Marquez is an innovative open source metadata service designed for the collection, aggregation, and visualization of a data ecosystem’s metadata. It efficiently maintains data provenance, offering a comprehensive view of how datasets are consumed and produced, and provides global visibility into job runtime and frequency of dataset access. Beyond centralizing dataset lifecycle management, Marquez is a beacon for data lineage queries across datasets, associating dependencies between jobs and datasets they interact with. As a modular system, it stands out as a scalable, extensible, and platform-agnostic solution for metadata management, emphasizing lifecycle management and timely processing of datasets.
Milestones and Achievements:
- An engaged community of 293 total contributors.
- Earned the Core Infrastructure Best Practices Badge with a perfect score on Silver and an impressive 98% on Gold.
- Demonstrated commitment with 177 commits in the past three months and a total of 2.62K commits over three years.
- Established governance and procedural frameworks, including a Technical Steering Committee (TSC), Contributor guide, Charter, Code of Conduct, and a Code quality statement.
- An exponential rise in popularity, from 180 GitHub stars in December 2019 to over 1.4K stars today.
Key Partnerships:
Marquez proudly collaborates with several leading organizations and projects:
- OpenLineage: The project that Marquez initially prototyped.
- Decathlon, Asana, Northwestern Mutual, Astronomer, Egeria: These partnerships symbolize Marquez’s commitment to integrating with industry leaders.
- Anonymous Partnerships: Marquez’s influence is further underscored by its collaborations with a “large tech company” and a “large bank.”
Growth Stats over the last 3 years:
As Marquez moves from the Incubation stage to the Graduation stage, it marks an important phase in its journey within the LF AI & Data Foundation. This transition is a testament to Marquez’s consistent growth, governance, and its role in the community. We recognize Marquez’s contributions and look forward to its continued success!
LF AI & Data Resources
- Learn about membership opportunities
- Explore the interactive landscape
- Check out our technical projects
- Join us at upcoming events
- Read the latest announcements on the blog
- Subscribe to the mailing lists
- Follow us on Twitter or LinkedIn
- Access other resources on LF AI & Data’s GitHub or Wiki
