Guest Author(s): Dr. Ofer Hermoni, Director of Product Strategy in Amdocs’ CTO office,
Use and adoption of Machine Learning (ML) and Deep Learning (DL) technologies is exploding with the availability of dozens of such open source libraries, frameworks and platforms, not to mention all the proprietary solutions. While there are many applications and tools out there, the integration between them can be complicated, can pose additional challenges especially in relation to long term sustainability, and may present a barrier for and adoption as part of a commercial product/service.
To help developers and data scientists make sense of the diversity of projects, the LF AI landscape (Figure 1) was originally published in December 2018 and has been continuously updated ever since. The LF AI landscape is an interactive tool that shows both how fragmented the space as well as the wide range of projects in each technology category.
Most open source AI projects started as proprietary efforts and are the result of years of investment and talent acquisition. At different points in time, the founding company (or companies) of such an effort decide to open source projects as a consequence of wanting to build an ecosystem around it and to collaborate with others on constructing a platform. The end result of that phenomenon is a large ecosystem of open source projects.
The important question from an adoption perspective is which open source project to adopt and how to integrate it with other open source solutions (libraries, frameworks, etc.) and internal proprietary stacks.
Goal: Better Integration among Projects and Tools
One of the goals of LF AI Foundation is building integration among LF AI projects and generally available and open source solutions so users can easily take advantage of a wide array of options and further the adoption of open source for AI solutions. This effort to improve integration and collaboration is aimed at helping bring everyone up to the same level of understanding of common deployments of ML workflow. Few companies are willing or able to provide this. This filtering and analysis is uniquely suited to a foundation like the LF AI Foundation, since we can look across specialties and provide help and guidance.
In his talk “How Linux Foundation is Changing the (Machine-Learning) World,” Ofer Hermoni, Ph.D., Director of Product Strategy, CTO Office, Amdocs, and the Chairperson, of LF AI Technical Advisory Council, highlights one of the key goals of the LF AI:
“Harmonization, Interoperability – Increase efforts to harmonize open source projects and reduce fragmentation; increase the interoperability among projects”
This has led to the LF AI Technical Advisory Committee (TAC) pushing to clarify the current landscape. First, what is a typical workflow? What projects are already available under the LF AI umbrella that can implement parts of that workflow? Finally, what open source projects are out there that help fill the gaps and provide good alternatives? This way, users can quickly start to understand the larger picture (landscape) and have a great understanding of not just available open source components in the AI/ML/DL space but also how to integrate them together in implementing an end-to-end ML workflow. At the same time, LF AI can better evaluate where integration is already strong and where there are gaps that can be opportunities to collaborate and fill following the open source approach for the benefit of the broader open source AI community.
The reference ML workflow produced by TAC is summed up in three main layers.
We started with reviewing existing published flows. We then built on them and extended them to create an entire workflow that covers the lifetime of ML integration across three major phases, starting with data preparation including data governance, moving through model creation, including ethics management, and then moving toward solution rollout including security management.
Second, the identification of existing LF AI hosted projects and where they fit in the ML workflow.
And, third, the ML workflow highlighting other open source projects and where they fit in, such as TensorFlow, Keras, PyTorch, Kubeflow and many more.
The figures are a great way to quickly grasp the entire process and identify the scope of the applications and tools that are needed, and is especially helpful in identifying integration opportunities across these different projects. The result is a better understanding of the connections or lack of connections, and a path to create these connections or integration points.
Who should use this?
We would like to hear from as many developers and data scientists as possible, since we are just getting started. There are certainly more connections and gaps to be identified. Integration work takes time. It’s been built up over the past year. This activity is open not only to LF AI members, but to the entire community, and many companies already participate in the discussions.
How Does My Project Get Involved?
The ML Workflow effort is open for participation and we are soliciting feedback to improve our reference workflow. There are various ways in which you can participate and get involved:
- Subscribe to the LF AI Technical Advisory Council mailing list
- Join the LF AI Technical Advisory Council bi-weekly open calls
- Participate in the ML Workflow effort directly with the team working on it
Meet the LF AI Team in San Diego (August 20, 2019)
LF AI is hosting an open meeting in San Diego on August 20th with the goal to discuss the ongoing projects, explore new collaboration opportunities, and provide face-to-face feedback and updates on various Foundation ongoing technical efforts. We welcome you to join, get to meet our members, projects, and staff, and explore ways to get involved in our efforts.
For more information please visit: https://lfai.foundation/event/lf-ai-meetings-in-san-diego/.
About the author
As the Director of Product Strategy in Amdocs’ CTO office, Dr. Ofer Hermoni is responsible for leading all of Amdocs’ activities in the Machine-Learning open-source community, including defining Amdocs’ product strategy in the area of AI/machine learning. In addition, he is the Chairperson of the LF AI Foundation Technical Advisory Council and a member of the LF AI Foundation Governing Board. Ofer is also an active contributor to the Acumos AI project, and a member of the Acumos AI Technical Steering Committee.
