LF AI & Data Foundation—the organization building an ecosystem to sustain open source innovation in artificial intelligence (AI), machine learning (ML), deep learning (DL), and data open source projects, is announcing Flyte as its latest Incubation Project.
Flyte is a production-grade, declarative, structured, and highly scalable cloud-native workflow orchestration platform. It allows users to describe their ML/Data pipelines using Python, Java, or in the future, other languages. It manages the data flow, parallelization, scaling, and orchestration of these pipelines. Flyte was released and open sourced by Lyft.
“The LF AI & Data Foundation is focused on growing and supporting an ecosystem of open source AI and Data projects. Today’s announcement of Flyte joining the Foundation in incubation represents significant progress toward achieving our vision,” said Dr. Ibrahim Haddad, Executive Director of LF AI & Data. “This project has proven highly effective in creating concurrent, scalable, and maintainable workflows for machine learning and data processing. We look forward to working to further grow the Flyte community and encourage adoption of this exciting project, and enable collaboration with the various other projects hosted in LF AI & Data and the open source community.”
The feature-set and extensibility of Flyte makes it extremely useful for Machine Learning pipelines like Feature Engineering, Adhoc and periodic Model Training, Model evaluation, ML monitoring, ETL, and general data processing. Flyte has extensions for Pandas, Tensorflow, Pytorch, Spark and integrates with Hosted cloud services such as AWS Sagemaker, Athena, etc.
Flyte is built to streamline the journey from idea to production for Data Scientists and ML Engineers. Its mission is to break down silos between infrastructure, data, ML, and product teams.
Ketan Umare, the project lead for Flyte, said: “We want Flyte to become the conduit for collaborating across myriad open source projects. To achieve this, it is essential that Flyte be perceived as a truly open platform. This is sorely missing in the current open source landscape. Today contributing Flyte to The LF AI and Data Foundation is a giant step towards our goal of simplifying the life of all hardworking ML and Data Engineers.”
LF AI & Data supports projects via a wide range of services, and the first step is joining as an Incubation Project. LF AI & Data will support the neutral open governance for Flyte to help foster the growth of the project. Learn more about Flyte on their GitHub and be sure to join the Flyte-Announce and Flyte-Technical-Discuss mail lists to join the community and stay connected on the latest updates.
A warm welcome to Flyte! We look forward to the project’s continued growth and success as part of the LF AI & Data Foundation. To learn about how to host an open source project with us, visit the LF AI & Data website.
Flyte Key Links
LF AI & Data Resources
- Learn about membership opportunities
- Explore the interactive landscape
- Check out our technical projects
- Join us at upcoming events
- Read the latest announcements on the blog
- Subscribe to the mailing lists
- Follow us on Twitter or LinkedIn
- Access other resources on LF AI & Data’s GitHub or Wiki
