Skip to main content
All Posts By


LF AI Day – Paris Edition, Recap

By Blog
Nicolas Demassieux (SVP, Orange Labs Research) in his opening speech 

The LF AI Day – Paris, was held September 16 in Paris-Châtillons, France, at Orange Gardens, 44 Avenue de la République. It was a fantastic day with presentations and a panel discussion from well-known organizations like Orange, NTT, Nokia, Ericsson, IBM, LF AI Foundation, and more.

LF AI Days are regional, one-day events hosted and organized by local members with support from LF AI and its hosted projects. These events are open to all for participation and have no cost to attend. More LF AI events information here.

Dr. Margriet Groenendijk from IBM discussing Trusted AI 


Nicolas Demassieux, SVP, Orange Labs Research, spoke of 3 challenges:

  1. The need to control “AI economics,” meaning ROI optimization and risk management when introducing AI models
  2. The need to speed up development of end-to-end AI tools and interoperability with enterprise data lakes
  3. The need to set up guidelines for trusted and fair AI

Masakatsu Fujiwara, Project Manager, NTT Network Technology Laboratories, talked about NTT view’s that the future of network management will be based on AI-driven autonomous maintenance loops.

Anwar Aftab, Director, Inventive Science, AT&T Labs, discussed how the future for network AI will autonomous, contextual and predictive networks that drive new experiences at higher velocities.

Philippe Carré, Senior Specialist Open Source, Nokia Bell-Labs & CTO, covered how Nokia’s three priorities for AI operations are Security, Fault Management, and Configuration management.

The Startups Panel Discussion, entitled “Barriers for AI development,” covered several types of barriers and including participation from François Tillerot, Intrapreneur-CMO, Orange AI Marketplace, Rahul Chakkara, Co-Founder, Manas AI, Laurent Depersin, Research & Innovation Home Lab Director, Interdigital, Marion Carré, CEO Ask Mona, and Sana Ben Jemaa, Project Manager Radio & AI, Orange Labs Networks

  • Why to introduce AI – Difficulty to describe an AI use case and translate business benefit, not clear ROI.
    • Supporting customers in their strategy to introduce AI and build ROI can tackle this issue
  • Project Technical or HR (skills) issues – Multiple environment/tools constrain End to End Solution, lack of talent/skills in companies, difficulty to deploy and scale.
    • Open source solutions, in particular LF AI and Acumos AI project will facilitate mutualized approaches and multi-skills collaboration to workaround
  • Readiness (technical or mindset) – Data supply chain not ready, lack of trust
    • Trustful AI approaches and better awareness in AI capabilities (also avoiding “overselling” AI) are potential solutions to tackle this.

The list of presentations made available:

Startup Panel discussing barriers for AI development

Come join us next time, and join the open source AI conversation!

Announcing the First Ever LF AI Summit

By Blog

The LF AI Foundation is proud to announce our first LF AI Summit. This is an important step forward in our support for open source AI development around the world. The LF AI Summit will facilitate presentations, discussions and networking among an incredible group of leading AI specialists and organizations like AT&T, Amazon, Capital One, Google, IBM, and more. The 3-day event will bring together individuals and organizations to share information and best-practices and make to help make important decisions in areas of privacy, ethics, training and much more.

The LF AI Summit will be co-located with the Open Source Summit EU, being held in Lyon, France, October 28-30. Attendees register for the Open Source Summit EU. There is no extra fee to attend the LF AI Summit.

Explore LF AI Summit Agenda

The LF AI Summit takes place over three days and is devoted to a wide array of open source AI topics. There are 19 presentations scheduled which cover AI privacy, ethics, training pipeline, model versioning and interworking with network automation and edge cloud technologies.

The LF AI Foundation itself will have a dedicated booth with demos of AI Marketplace, Acumos, plus LF AI hosted projects including Horovod, Angel, Pyro, and EDL. Please come by the booth and ask any questions. Find out how Acumos and the LF AI projects are expanding AI beyond specialists to all groups within companies. And find out how you can host your own project in LF AI!

Join us by registering to attend the Open Source Summit EU – Register Now! 

Please note, the LF AI Summit is co-located with the Open Source Summit EU. Please register for the Open Source Summit EU. There is no extra fee to attend the LF AI Summit.

Updated LF AI Project Proposal Process and Lifecycle Document

By Blog

Download the new and improved ​“LF AI Projects: Process and Lifecycle​” (PDF) to learn the process of submitting your open source project for hosting under LF AI Foundation!

The LF AI Foundation is releasing today an updated version of its Project Process and Lifecycle document, a little over a year since we had our first version in August of 2018. Over the past year, we have welcomed four new projects to the Foundation (Angel, EDL, Horovod and Pyro) and went through the on boarding process with each one of them. As a result of these experiences and various incoming feedback, we realized there are many improvements opportunities to be made. We took action and the outcome is an improved document that better describes the various stages of the projects, explains how a project transitions from one stage to another, and a detailed description of the various ways we support the project.

If you are interested in hosting your open source AI/ML/DL project in the LF AI, please review the document and email us via We’re eager to help and discuss with you such possibilities.

For further reading, please visit these pages:

LF AI Day – Shanghai: Full Day Deep Dive into Open Source AI

By Blog

Register Now! 

Huawei, Tencent and the LF AI Foundation are pleased to announce LF AI Day – Shanghai is being held in beautiful Shanghai on September 17 at the Huawei Institute of Research and Development. LF AI Days are regional, one-day events hosted and organized by local members with support from LF AI and our hosted projects. Speakers from leading operators and AI industry with a focus on open source strategies for machine learning and deep leaning.

These events are open to all for participation and have no cost to attend.

Agenda is available from:

For questions, please contact

To view LF AI Days happening is other geographical regions, please visit the LF AI Events page. 

Register Now! 

Dataquest: Tech Mahindra launches GAiA 2.0 to help enterprises adopt artificial intelligence faster

By In The News

Tech Mahindra announced the release of GAiA 2.0, the latest version of its Enterprise Artificial Intelligence (AI) and Machine Learning (ML) lifecycle management platform GAiA, powered by Acumos.

GAiA 2.0 will enable comprehensive AI and ML driven platform capabilities and services to be deployed across mainstream, optimizing enterprise operations in real time across industry verticals. It offers an enriched marketplace of models and numerous features to empower enterprises across industry verticals to build, manage, share and rapidly deploy AI and ML driven services and applications addressing critical business problems.

Read more at Dataquest.

Introducing FATE 1.0: Milestone Version Introduces New Features, Stability and Performance Enhancements

By Blog

This is a guest blog post by the FATE community, a Linux Foundation member with commonality of interests with LF AI

The FATE community is excited to announce the availability of FATE 1.0. We are striving to improve the development of federated learning technologies to achieve more powerful functions and applications. We consider FATE 1.0 to be a milestone version which empowers the FATE community with more powerful tools and a significantly improved developer experience.

FATE (Federated AI Technology Enabler) is a federated learning framework that fosters collaboration across companies and institutes to perform AI model training and inference in accordance with user privacy, data confidentiality and government regulations.

FATE recently joined the Linux Foundation with several organizations supporting the project including 4Paradigm, CETC Big Data Research Institute, Clustar, JD Intelligent Cities Research, Squirrel AI Learning, Tencent and WeBank.

What’s new in the FATE 1.0 release

  • FATEBoard, a visual tool for federated learning modeling for end-users
  • FATEFlow, an end-to-end pipeline platform for federated learning
  • Performance updates for all algorithm modules
  • Mature features of online federated inference

FATE 1.0 benefits and features

  • FATEBoard visualizes the federated learning process
    • Greatly improving the federated modeling experience, FATEBoard allows end-users to explore and understand models easily and effectively
    • FATEBoard supports visualization in the status changes of training, model graphs, logs tracking, and much more, which makes federated learning modeling easier to understand, debug and optimize
    • Click here for more information
  • FATEFlow builds highly flexible, high performance federated learning pipeline production service
    • FATEFlow supports model life-cycle management functions, which implement state management of pipelines and the collaborative scheduling of operation, and automatically tracks data, models, metrics, and logs generated in the task to facilitate analysis of users
    • Learn more or get started here
  • Performance Updates provide high flexibility, high stability and high performance for federated learning
    • FATE 1.0 supports use of DSL to describe federated modeling workflow
    • FATE 1.0 introduces a new Homomorphic Encryption algorithm based on Affline Transforms
    • FATE 1.0 also supports the Nesterov Momentum SGD Optimizer, which makes the federated learning algorithm converge quickly

Getting Started

FATE supports three deployment modes: Standalone in Docker, Standalone Compiled, and Cluster Compiled. The Cluster in Docker mode is expected to come with the next release. Stay tuned by joining the Fate-FedAI mailing-list, or you can also visit the FATE README

Suggestions or Contributions

Join us in our community via regular meetings, or our mailing-list and give us your feedback.

Anyone interested in federated learning is welcome to contribute code and submit Issues or Pull Requests. Please refer to the FATE project contribution guide first.

About the FATE Project

FATE is an open-source project initiated by WeBank’s AI Group to provide a secure computing framework for building the federated AI ecosystem. It implements a secure computation version of various machine learning algorithms, including logistic regression, tree-based algorithms, deep learning and transfer learning. For developers who need more than out-of-box algorithms, FATE provides a framework to implement new machine learning algorithms in a secure MPC architecture. Learn more about the project at

ZDNet: IBM joins Linux Foundation AI to promote open source trusted AI workflows

By In The News

AI is advancing rapidly within the enterprise — by Gartner’s count, more than half of organizations already have at least one AI deployment in operation, and they’re planning to substantially accelerate their AI adoption within the next few years. At the same time, the organizations building and deploying these tools have yet to grapple with the flaws and shortcomings of AI– whether the models deployed are fair, ethical, secure, or even explainable.  

Before the world is overrun with flawed AI systems, IBM is aiming to rev up the development of open-source trusted AI workflows. As part of that effort, the company is joining the Linux Foundation AI (LF AI) as a General Member. 

Read more at ZDNet.

IBM Joins LF AI Foundation

By Blog

San Diego – Open Source Summit North America – Aug 21, 2019 – The LF AI Foundation, the organization building an open AI community to drive open source innovation in artificial intelligence (AI), machine learning (ML) and deep learning (DL), today announced IBM is joining LF AI as a General Member. IBM, a global leader in delivering AI solutions and an acknowledged leader in bringing AI into widespread use with commercial solutions like Watson, has been working closely on an informal basis with the LF AI Foundation participating in events worldwide and contributing to open source trusted AI workflows with projects like the LF AI Foundation Technical Advisory Committee’s ML Workflow project.

“IBM is a world leader in AI. They provide leadership not only technically, but also from an ethical and trusted AI standpoint. IBM will help spread AI with clear guidelines on ethics, fairness, robustness, and explainability that benefit all participants of the open source AI ecosystem. This is a big step forward in strengthening the reach of AI and helping data scientists and developers worldwide,” said Dr. Ibrahim Haddad, Executive Director of the LF AI Foundation. “We’re excited to have IBM join the LF AI Foundation to promote and shape the future of trusted AI workflows, and foster synergy and collaboration across multiple Linux Foundation umbrella foundations.”

IBM is a Platinum Member of the Linux Foundation, and, in addition to joining in the LF AI Foundation, is also a member of the Open Mainframe Project, the Cloud Native Computing Foundation, the OpenJS Foundation, the Hyperledger Foundation, the R Consortium, LF Edge LF Energy, LF ONAP, ODPi,  and many others. IBM is strongly committed to the development and advancement of the open source ecosystem. IBM’s data and AI product offerings are built on open source and are strengthened by the accelerated pace and energy of community development. With their membership in LF AI, they are committing to support open source AI technologies and to collaborate with the global community in a vendor-neutral environment for advancing the open source AI platform. 

“IBM has a long history of contributions to open source foundations and community projects,” said IBM VP Todd Moore, Open Technologies and Developer Ecosystem.“ The time is right for IBM to join LF AI as a General Member to work closely with existing members and the broader community to lay the foundations for trusted AI workflows together. IBM Research is a leader in trusted AI and ethical AI guidelines, and IBM’s Data and AI offerings are built on open source components.”

“We are very excited to have IBM join the LF AI Foundation. They are a key piece in the continued growth of the LF AI Foundation. We have been working closely together on areas like our ML Workflow effort and exploring possible collaboration with other industry initiatives and other Linux Foundation hosted projects. IBM has already been active suggesting additions to the LF AI landscape, including projects in the area of trusted AI.  This announcement expends an already strong relationship,” said Dr. Ofer Hermoni, Director of Product Strategy at Amdocs and Chair of the LF AI Technical Advisory Counsel. “IBM is well-known for their leadership in open source AI ethics, and we welcome their strong contributions in these areas as part of the LF AI Foundation.”

“AT&T is excited to welcome IBM and its AI expertise to the LF AI community. It is encouraging to see industry leaders commit to open innovation with ethics as a strong foundation,” said  Mazin Gilbert, Vice President at AT&T Labs.

“IBM is a welcome addition to the LF AI Foundation membership as a leader in the development of open source AI through their Center for Open Source Data and AI Technology” said Dr. Jamil Chawki, Chairman of the LF AI Foundation Outreach Committee. “We look forward to working closely with IBM and helping chart the future of AI together, including the area of trusted AI workflows.”

“As one of the founding members of LF AI, Tech Mahindra is excited to see the AI ecosystem growing. We extend our warm welcome to IBM for joining LF AI  and bringing in the expertise in Ethical AI which will play a crucial role in ensuring that we build AI right,” said Dr. Satish Pai, Sr. Vice President, Americas Communications, Media and Entertainment, Tech Mahindra. “Tech Mahindra looks forward to collaborate and create synergies with IBM across LF AI hosted projects including Acumos.”

“IBM is a pioneer in the field of AI. Tencent sends its warmest congratulations on joining the LF AI Foundation,” said Dr. Han Xiao, Engineering Lead, Tencent AI Lab. “IBM’s continued success using and developing AI applications will help strengthen the message of the LF AI Foundation, and together with all the members we can build an open and collaborative AI ecosystem.”

About IBM Data and AI

For more information go to and

About LF AI Foundation

The LF AI Foundation, a Linux Foundation project, accelerates and sustains the growth of Artificial Intelligence (AI), Machine Learning (ML) and Deep Learning (DL) open source projects. Backed by many of the world’s largest technology leaders, LF AI is a neutral space for harmonization and ecosystem engagement to advance AI, ML and DL innovation. To get involved with the LF AI Foundation, please visit

Horovod Updates

By Blog

Version 0.17.0 of Horovod, the distributed deep learning framework, has been released. With the new release, Horovod extends and improves support for machine learning platforms and libraries. The release also contains a new run tool, performance improvements, and minor bug fixes.


Running Horovod training directly using Open MPI gives a lot of flexibility and allows fine-grained control over options and settings. The flexibility comes with the challenge of providing a significant number of parameters and values, even for simple operations. Missing or wrong parameters or values will prevent Horovod from running successfully.

With this release, the command-line utility horovodrun is introduced. The utility horovodrun is an Open MPI-based wrapper for running Horovod scripts, without the complexity of running Open MPI commands. The horovodrun utility automatically detects and sets parameters and allows the user to show the used MPI command if desired. 


Let’s say we have a Horovod script, and we want to run it on one machine using four GPUs, the horovodrun command would be:

horovodrun -np 4 -H localhost:4 python

The flag -np specifies the number of processes, and the -H flag specifies the host. If more machines are used, the hosts can be listed separated by commas.

The equivalent Open MPI command would be:

mpirun -np 4 \
    -H localhost:4 \
    -bind-to none -map-by slot \
    -mca pml ob1 -mca btl ^openib \

Apache MXNet 

Apache MXNet is a high performant deep learning framework used for building, training, and deploying deep neural networks and supports distributed training.

Apache MXNet 1.4.1 and 1.5.0 are the releases officially supporting Horovod. Previously, the MXNet 1.4.0 release supported Horovod on certain OS, and users had to run the master branch version of MXNet to have Horovod support. In addition, the DistributedTrainer object is now introduced to better support Gluon APIs and enable Automatic Mixed Precision (AMP) in MXNet.

MPI-less Horovod alpha

MPI is used extensively in the supercomputing community for high-performance parallel computing, but can be difficult to install and configure for the first time.  This change introduces support for Facebook’s Gloo as an alternative to running Horovod with MPI. Gloo comes included with Horovod, and allows users to run Horovod without requiring MPI to be installed.  

For environments that have support both MPI and Gloo, users can choose their preferred library at runtime with a single flag to horovodrun:

$ horovodrun –gloo -np 2 python

Gloo support is still early in its development, and more features are coming soon, most notably: fault tolerance. Stay tuned!

TensorFlow 2.0 support

TensorFlow 2.0 introduces some significant changes, not only when it comes to new features, but also when it comes to the API. The changes include removing redundant APIs and making the API more consistent, with a focus on improving the integration experience. Horovod supports the new TensorFlow 2.0 features and APIs in the latest release.

Intel MLSL

Intel Machine Learning Scaling Library (MLSL) offers a set of communication features which can provide performance benefits for distributed performance, such as asynchronous progress compute/communication overlap, message prioritization, support for data/model/hybrid parallelism, and employment of multiple background processes for communication.

Horovod supports different communication backends, such as MPI, NCCL, and DDL, and with the latest release, Horovod also supports Intel MLSL. Using MLSL as the communication backend improves both the scalability of communication-bound workloads and the compute/communication ratio. 


Horovod version 0.16.3 contains performance improvements for existing features with the most noteworthy being updates for PyTorch and large clusters. 

PyTorch performance

PyTorch is an open source, Python-based framework built for easy and efficient deep learning. PyTorch can utilize GPUs to accelerate tensor computation, and provides great flexibility and speed. 

In the new release of Horovod, performance has been improved for gradient clipping, which is a method used for preventing instabilities caused by gradients with excessively large values. 

Large cluster performance

Performance for ultra-large clusters is improved in Horovod 0.16.3. One example of an ultra-large cluster, which takes advantage of this improvement, is Oak Ridge National Laboratory’s Summit supercomputer. Summit has more than 27,000 GPUs and was built to provide computing power to solve large-scale deep learning tasks for which great complexity and high fidelity is required.  

In Horovod, network communication is used in two distinct ways. First and foremost, network communication is used to carry out the collective operations to allreduce/allgather/broadcast tensors across workers during training. To drive these operations, network communication is also used for coordination/control to determine tensor readiness across all workers, and subsequently, what collective operations to carry out. With large cases on these systems spanning many hundreds to thousands of GPUs, the coordination/control logic alone can become a severe limiter to obtaining good parallel efficiency. To alleviate this bottleneck, NVIDIA contributed an improvement to the coordination/control implementation in Horovod to reduce the network communication usage for this phase of operation. In the improved implementation, a caching mechanism is introduced to store tensor metadata that was, in the original implementation, redundantly communicated across the network at each training step. With this change, coordination/control requires as little as a single bit per tensor communicated across the network per training step, instead of several bytes of serialized metadata per tensor.

RDMA support in the provided docker containers

Starting with Horovod version 0.16.4, RDMA support is available with Docker containers, which increases Horovod’s performance.

Previously it was necessary to build your own Docker image with the appropriate libraries to, such as MOFED, to run Horovod with RDMA. That is no longer necessary, as the provided containers now support RDMA. 

If you have Mellanox NICs, we recommend that you mount your Mellanox devices (/dev/infiniband) in the container and enable the IPC_LOCK capability for memory registration:

$ nvidia-docker run -it –network=host -v /mnt/share/ssh:/root/.ssh –cap-add=IPC_LOCK –device=/dev/infiniband horovod:latest
root@c278c88dd552:/examples# …

You need to specify these additional configuration options on primary and secondary workers.

Curious about how Horovod can make your model training faster and more scalable? Check out these new updates and try out the framework for yourself. Be sure to join the Horovod Announce and Horovod Technical-Discuss mailing list.

Angel 3.0 Available Now – Major Milestone in Providing Full ML Stack

By Blog

Angel 3.0 is now available via Angel offers a full stack machine learning platform designed for sparse data and huge model scenarios, built on a high-performance Parameter Server (PS). Angel is used by Tencent and more the 100 companies in products or internally to their organizations. It boasts 4200+ stars on GitHub, 7 sub-projects, 1100+ forks, and 2000+ commits.

Angel joined the LF AI Foundation in August 2018 as an incubation project from Tencent, a Premier member of the Foundation. 

Angel 3.0 Features

Angel 3.0 adds Auto Feature Engineering, New or Enhanced Computation Engines including Angel native, Spark on Angel (SONA) and PyTorch on Angel (PyTONA). Angel 3.0 therefore allows users to switch to Angel from Spark or PyTorch smoothly with nearly zero cost. 

A detailed white paper on Angel 3.0 authored by Fitz Wang, Ph.D., Senior Researcher, Tencent, and Angel’s maintainer and core developer, introduces the new features of Angel 3.0. It shows what distinguishes Angel from existing machine learning platforms such as TensorFlow, PyTorch, MxNet, PaddlePaddle and Spark. It is available here:

LF AI Foundation Projects 

LF AI is an LF umbrella foundation that was founded in March 2018 to support and sustain collaboration and open source innovation in AI, machine learning and deep learning. It offers a neutral environment to its hosted open source projects and support them with a number of services to help the projects gain wider adoption. Current projects include Acumos AI, Angel, Elastic Deep Learning (EDL), Horovod, and Pyro. For more information please on these projects, please visit:

The LF AI Foundation supports open source AI developers and organizations around the world. We are constantly looking to host and support additional projects. People interested in hosting  their projects under the LF AI Foundation are encouraged to email us at Details on proposing projects for hosting in LF AI are available via

Meet Angel’s Developers at OSS NA

Angel core maintainers and other developers are presenting on August 20 at the LF AI Meetings in San Diego, co-located at the Open Source Summit NA, and will also be at the LF AI booth Aug 21-23 to show demos and answer questions.

For more information, including both schedules and more, please see:

LF AI Meetings, San Diego – How to Register 

LF AI Meetings, San Diego – Agenda

LF AI Booth #43 – Developer Schedule at Open Source Summit