All Posts By

dbrown

Horovod

LF Deep Learning Welcomes Horovod Distributed Training Framework as Newest Project

By Press Release

Contributed by Uber, Horovod makes distributed deep learning fast and easy to use

SEATTLE – KubeCon + CloudNativeCon North America – December 13, 2018 – The LF Deep Learning Foundation, a community umbrella project of The Linux Foundation that supports and sustains open source innovation in artificial intelligence, machine learning, and deep learning, announces the Horovod project, started by Uber, as its newest project. Horovod, a distributed training framework for TensorFlow, Keras and PyTorch, improves speed, scale and resource allocation in machine learning training activities.

“The LF Deep Learning Foundation is focused on building an ecosystem of AI, deep learning and machine learning projects. Today’s announcement of Uber’s contribution of the Horovod project represents significant progress toward achieving this vision,” said Ibrahim Haddad, Linux Foundation Director of Research. “This project has proven highly effective in training machine learning models quickly and efficiently, and we look forward to working to further grow the Horovod community and encourage adoption of this exciting project.”

Horovod makes it easy to take a single-GPU TensorFlow program and successfully train it on many GPUs faster. Horovod also achieved significantly improved GPU resource usage figures. The project uses advanced algorithms and leverages features of high-performance networks to provide data scientists, researchers and AI developers with tooling to scale their deep learning models with ease and high performance. In benchmarking Horovod against standard distributed TensorFlow, Uber has observed large improvements in its ability to scale, with Horovod coming in roughly twice as fast.

Real-world activities Uber has used Horovod to support include self-driving vehicles, fraud detection, and trip forecasting. It is also being used by Alibaba, Amazon and NVIDIA. Contributors to the project outside Uber include Amazon, IBM, Intel and NVIDIA.

“Uber built Horovod to make deep learning model training faster and more intuitive for AI researchers across industries,” said Alex Sergeev, Horovod Project Lead. “In this spirit, we are honored to contribute Horovod to the deep learning community as the LF Deep Learning Foundation’s newest project. As Horovod continues to mature in its functionalities and applications, this collaboration will enable us to further scale its impact in the open source ecosystem for the advancement of AI.”

Horovod joins existing LF Deep Learning projects: Acumos AI, a platform and open source AI framework; Angel, a high-performance distributed machine learning platform based on Parameter Server; and EDL, an Elastic Deep Learning framework designed to help cloud service providers to build cluster cloud services using deep learning frameworks. Horovod complements these existing projects and future collaboration is anticipated between them.

Horovod Background

Contributed to the LF Deep Learning Foundation by Uber, the project currently has 175 commits from 26 committers, and is licensed under Apache-2.0.

Horovod, which has secured a Linux Foundation Core Infrastructure Initiative Best Practices Badge, is also included in deep learning distributions including AWS Deep Learning AMI, Azure Data Science VM, Databricks Runtime, GCP Deep Learning VM, IBM FfDL, IBM Watson Studio and NVIDIA GPU Cloud. More information on Horovod can be found on the Uber Engineering blog and in this Q&A with Horovod creator, Alex Sergeev.

Following recent news of Uber joining the Linux Foundation as a Gold member, Uber continues to deepen its contributions to open source technology. Another hallmark open source technology from Uber, Jaeger, is a Cloud Native Computing Foundation project.

Organizations and developers interested in contributing projects and learning more about LF Deep Learning Foundation, can go to www.deeplearningfoundation.org.

About LF Deep Learning

The LF Deep Learning Foundation, a Linux Foundation project, accelerates and sustains the growth of artificial intelligence, machine learning and deep learning open source projects. The initiative’s Acumos AI Project is a platform and open source framework that makes it easy to build, share and deploy AI models. Backed by many of the world’s largest technology leaders, LF Deep Learning is a neutral space for harmonization and ecosystem engagement to advance AI, DL and ML innovation. To get involved with the LF Deep Learning Foundation, please visit https://www.deeplearningfoundation.org.

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and industry adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

# # #

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.

Introducing the Interactive Deep Learning Landscape

By Blog

The artificial intelligence (AI), deep learning (DL) and machine learning (ML) space is changing rapidly, with new projects and companies launching, existing ones growing, expanding and consolidating. More companies are also releasing their internal AI, ML, DL efforts under open source licenses to leverage the power of collaborative development, benefit from the innovation multiplier effect of open source, and provide faster, more agile development and accelerated time to market.

To make sense of it all and keep up to date on an ongoing basis, the LF Deep Learning Foundation has created an interactive Deep Learning Landscape, based on the Cloud Native Landscape pioneered by CNCF. This landscape is intended as a map to explore open source AI, ML, DL projects. It also showcases the member companies of the LF Deep Learning Foundation who contribute contribute heavily to open source AI, ML and DL and bring in their own projects to be housed at the Foundation.

This tool allows viewers to filter, obtain detailed information on a specific project or technology, and easily share via stateful URLs. It is intended to help developers, end users and others navigate the complex AI, DL and ML landscape.

All data is also available in a GitHub repo, and anyone may update or add to the landscape by submitting a pull request on GitHub.

We encourage you to spend some time with this tool, learn more about the current AI, DL and ML space, and begin contributing to it.

LF Deep Learning Delivers First Acumos AI Release Making it Easier to Deploy and Share Artificial Intelligence Models

By Press Release

First Acumos AI release enables deployment of AI applications in private/public cloud environments, and enhances the platform user interface experience

SHANGHAI (KUBECON + CLOUDNATIVECON CHINA) – November 14, 2018 – The LF Deep Learning Foundation, a project of The Linux Foundation that supports open source innovation in artificial intelligence (AI), machine learning (ML), and deep learning (DL), today announced the availability of its first software release of the Acumos AI Project – Athena.

Acumos AI is a platform and open source framework that makes it easy to build, share and deploy AI applications. Acumos AI standardizes the infrastructure stack and components required to run an out-of-the-box general AI environment. This frees data scientists and model trainers to focus on their core competencies and accelerate innovation.

“The Acumos Athena release represents a significant step forward in making AI models more accessible for builders of AI applications and models along with users and trainers of those models and applications,” said Scott Nicholas, senior director of strategic planning at The Linux Foundation. “This furthers the goal of LF Deep Learning and the Acumos project of accelerating overall AI innovation.”

Major highlights of the Athena release include:

  • One-click deployment of the platform utilizing Docker or Kubernetes;
  • The ability to deploy models into a public or private cloud infrastructure or in a Kubernetes environment on users’ own hardware including servers and virtual machines;
  • A design studio, which is a graphical interface for chaining together multiple models, data translation tools, filters and output adapters into a full end-to-end solution;
  • Use of a security token to allow simple onboarding of models from an external toolkit directly to an Acumos AI repository;
  • Decoupling of microservices generation from the model onboarding process to easily repurpose models for different environments and hardware; and
  • An advanced user portal with the ability to personalize marketplace view by theme, data on model authorship as well as to share models privately or publicly and user experience upgrades.

All of these features are designed to make it quick and easy to get started deploying and sharing Acumos AI applications.

Full release notes can be accessed at https://wiki.acumos.org/display/REL/Athena+Release.

“LF Deep Learning members, including Amdocs, AT&T, Orange, Tech Mahindra and others, are contributing to the evolution of the platform to ease the onboarding and the deployment of AI models,” said LF Deep Learning Outreach Committee Chair Jamil Chawki. “The Acumos AI Marketplace, a catalog of community-contributed AI models with common APIs that can be shared securely across multiple systems, remains open and accessible to anyone who wishes to download or contribute models and applications.”

“The LF Deep Learning Foundation is focused on building an ecosystem of AI, deep learning and machine learning projects, and today’s announcement represents a significant milestone toward achieving this vision,” said LF Deep Learning Technical Advisory Council Chair Ofer Hermoni of Amdocs.

“We’re already inspired and energized by the progress of the Acumos AI Project since its initial launch earlier this year,” said Mazin Gilbert, Vice President of Advanced Technology and Systems at AT&T and Governing Board Chair of LF Deep Learning. “Athena is the next step in harmonizing the AI community, furthering adoption and accelerating innovation.”

What’s Next for Acumos AI

The developer community for Acumos AI is already working on the next release, which will be available in mid-2019, introducing convenient model training as well as data extraction pipelines to make models more flexible. Additionally, the next release will include updates to assist closed-source model developers, including secure and reliable licensing components to provide execution control and performance feedback across the community.

Organizations interested in contributing projects and more information about LF Deep Learning Foundation, can go to www.deeplearningfoundation.org.

Supporting Quotes

“Amdocs is proud to be an active member of The Linux Foundation, and in particular, a Founding Member of the Acumos AI project. The Acumos AI Athena release is a big milestone for open source AI and its role in driving intelligence, automation and machine learning in the communication and media industries.  Acumos AI is positioned to become the de-facto marketplace for machine learning models, both for open source as well as peer to peer (company to company) models, delivering faster time to innovation.” – Anthony Goonetilleke, Group President of Amdocs Technology

“As one of the founding members of LF Deep Learning Foundation, Huawei is excited to see this growing AI developer ecosystem continue to help the Acumos AI platform mature, and reach the first major technical milestone – the Acumos Athena release! The Acumos project offers significant value to Huawei SoftCOM AI strategy – building autonomous networks with the goal of automation, self-optimization, and self-healing to help operators significantly improve network utilization and maintenance efficiency. Huawei is proud to be part of this project and will continue to work with the Acumos developer community to further enhance the Acumos AI platform and unleash the power that AI can bring to our operators.” – Xiaoli Jiang, GM, Cloud Open Source Development Team, Huawei

“Orange has been actively involved in Acumos since April 2018 through a Project Team Leader for the model onboarding module to manage and drive evolutions in the onboarding capabilities. This involvement shows the willingness of Orange to take part and promote the AI ecosystem in the telecom domain. Orange also considered the coherency of integrating Acumos in the continuity of all the works performed in the network automation LFN/ONAP project. Acumos is seen as a common platform that can bridge existing AI technologies and new ones through its openness. It can also favor cross business AI-based developments through its federative approach and thanks to its marketplace.” – François Jezequel, Head of ITsation, Procurement and Operators, Orange

“As a key contributor to the Acumos Athena release we are excited to foster a collaborative ecosystem through the Acumos AI Marketplace. It is also a testimony to our COPA framework (Co-Create using Open Source to create Platforms and bring in Automation and AI). With companies and customers increasingly adopting Acumos, industrialization of AI through a common standard and marketplace like Acumos is bound to gain more traction.” – Dr. Satish Pai Sr. Vice President, Americas Communications, Media and Entertainment, Tech Mahindra

About LF Deep Learning

The LF Deep Learning Foundation, a Linux Foundation project, accelerates and sustains the growth of artificial intelligence, machine learning and deep learning open source projects. Backed by many of the world’s largest technology leaders, LF Deep Learning is a neutral space for harmonization and ecosystem engagement to advance AI, DL and ML innovation. To get involved with the LF Deep Learning Foundation, please visit https://www.deeplearningfoundation.org.

About The Linux Foundation

The Linux Foundation is the organization of choice for the world’s top developers and companies to build ecosystems that accelerate open technology development and industry adoption. Together with the worldwide open source community, it is solving the hardest technology problems by creating the largest shared technology investment in history. Founded in 2000, The Linux Foundation today provides tools, training and events to scale any open source project, which together deliver an economic impact not achievable by any one company. More information can be found at www.linuxfoundation.org.

# # #

The Linux Foundation has registered trademarks and uses trademarks. For a list of trademarks of The Linux Foundation, please see our trademark usage page: https://www.linuxfoundation.org/trademark-usage. Linux is a registered trademark of Linus Torvalds.