All Posts By

Carly Driggers

Adding EfficientNet-Lite4 and SSD-MobileNetV1 to the ONNX Model Zoo

By Blog

Guest Author: Shirley Su, University of Southern California (USC) freshman majoring in Computer Science. Her passion is to utilize technology for social good: whether it is creating usable applications or advancing research, she wants to be a positive contributor for the community.

The Coronavirus pandemic was a major impediment for job opportunities as it prompted several internships to be cancelled this year. While disappointed, I still wanted to have a productive summer and further my experience in the computer science field. I decided to reach out to my former Microsoft AI Platform intern co-workers from last summer. We were eager to contact our former mentor and continue our projects on ONNX – this time, serving as open-source contributors.

As an exploration of the recent advancements in computer vision, I was eager to research new machine learning models and contribute them to the ONNX Model Zoo. The ONNX Model Zoo provides many state-of-the-art pre-trained models that come with instructions for integrating into applications. I investigated the tensorflow-onnx GitHub repo, which detailed the conversion of both the EfficientNet-Lite4, an image classification model, and the SSD-MobileNetV1, an object detection model. These are popular computer vision models and I wanted to add them to the ONNX Model Zoo so others could more readily use them.

I began the conversion process by initially copying and then running the Jupyter Notebook Script from the GitHub repo. This process included setting up environmental variables and downloading the pre-trained model. After saving the model, I ran the script to convert the model from TensorFlow into the ONNX format. I then ran inference on the saved model using ONNX Runtime to view and validate results. I also uploaded the models to Netronan open source viewer that allows users to visualize the neural networks inside ONNX models. It provides information on the model’s operator set, the ONNX version, and input and output tensor shapes and types. I included that information in the template for the new model’s README as part of the instructions on how to pass in input data and how to interpret the output.

Comments from the ONNX community on GitHub were especially helpful in pointing out mistakes and helping me resolve the issues in the model folders. In both my EfficientNet-Lite4 and SSD-MobileNetV1 model, my file with the sample inputs and outputs were incorrect. To revise and fix the code, I converted the NumPy array to a TensorProto and saved the TensorProto as a serialized protobuf file.  Moreover, there was an error in the postprocessing code for the SSD-MobileNetV1, which incorrectly outputted the array of object detection predictions. I realized that while the model produced detection classes from the inference, the most accurate class label was not being outputted. To fix this issue, I changed how the results were looped over to include the most accurate class label.

My pull requests were merged and now everyone can use the EfficientNet-Lite4 and SSD-MobileNetV1 models I contributed to the ONNX Model Zoo.

Trials and Tribulations

Another model that interested me was the Visual Question Answering (VQA) model, in which users input an image and a question about the image and the model outputs an answer. I used a VQA GitHub that had the necessary files and an open-source license.

However, I ran into several issues during the process. The most time-consuming and tedious task was downloading 64 GB worth of MS COCO data onto my computerall without a fast internet connection or a powerful machine. This process took several hours and my computer crashed. Realizing that this attempt was futile, I began to look into Microsoft’s Azure Virtual Machines, which had the necessary memory and space needed. Using the virtual machine expedited the task and shorted the download time from approximately 10 hours to just 1 hour.

After I had successfully downloaded and preprocessed the data, the next obstacle was exporting the model to ONNX. When I passed in the standard model arguments, I was getting issues with PyTorch’s ONNX export call. Since the model was written with a much older version of PyTorch, I suspected the model needed updates to make it compatible with TorchScript and ONNX export.

I hope others in the open-source community will continue working on making this model accessible and contribute it to the model zoo.

Takeaways

Throughout this project, I gained more expertise working with Git and became more comfortable making pull requests and becoming an open-source contributor. I was able to contribute 2 important computer vision models to the ONNX model zoo. And while I was unable to contribute the VQA model, that project allowed me to gain hands-on experience with working with Azure virtual machinesa tool that is crucial when handling large amounts of data. I also became more comfortable reading others’ Python code and learning how to efficiently debug errors.

This summer, our team was especially grateful for the help we received from our mentors Vinitra Swamy and Natalie Kershaw. They have given us opportunities that allow us to make meaningful contributions to the ONNX Model Zoo. From dedicating time for weekly meetings with us to helping us debug strenuous errors, their guidance has been immensely helpful in our technical and professional development.

Artificial Intelligence (AI) Enables a Personalized Customer Experience at Scale

By Blog

Guest Authors: Utpal Mangla: VP & Senior Partner; Global Leader: IBM’s Telco Media Entertainment Industry Center of Competency at IBM (utpal.mangla@ca.us.ibm) AND Luca Marchi: Artificial Intelligence Lead – Center of Competence – Telco, Media andEntertainment at IBM (lmarchi@us.ibm.com)

As a client, how many times did you say: “This company really gets me?”. And when that happens, how did it influence your loyalty and the likelihood to do more business with them? Probably a LOT!!!

That’s exactly why customer experience is a KEY competitive advantage for companies in every sector and specifically for those in the telecommunication, media and entertainment industry. Providing an excellent and frictionless customer experience is the best way to reduce churn, improve NPS and increase ARPU through up-sell activities

On the other end, telecommunication, media & entertainment industries have been challenged by digitally native ‘Over The Top’ (OTT) entrants that design their organizations around the concept of superior customer experience. Matching their level of client focus and exceeding expectations is the only way to resist their attacks.

Using AI to understand your customers

AI has proven to be a key enabler in better understanding customers. AI enables companies to better relate with their client in 3 impactful and powerful ways:

  • 1) Derive insights from Big Data: In order to understand your customers, you need data about them. Luckily, we live in the era of big data, where user-generated data, internal and external data, public and private data abound. AI has the ability to ingest, understand and provide actionable insights based on those data. Also, AI models can be optimized and tuned automatically, improving after each interaction.
  • 2) Leverage Context: When making decisions, AI takes into account the context. The user experience is not only based on who the customer is but also on what the customer is going through at this very moment.
  • 3) Scale a personalized interaction: Ideal way to provide a customized experience would be to assign an attendant to each customer, like in a high-end restaurant. That is clearly not feasible, but AI comes to help. Through natural language understanding technology, companies can scale high touch customer support to provide the same level of expertise and engagement to all their clients

Customer experience starts with the Network

When it comes to telco, there is no personalized customer experience that can make up for network failures. We expect services to be up ALL the time. If not, we are ready to switch provider. Expectation of network availability has become even more pertinent in the current COVID environment. Implicit expectation is that networks are up and running 24/7

Telecommunication companies are acutely aware of these expectations. That’s why they are infusing artificial intelligence in their network operations

In fact, machine learning models can leverage network data to proactively get ahead of anomalous network activity and degradations; thereby increasing speed and accuracy of the detection, prevention, and repair of network issues. Artificial intelligence supports network engineers in root cause analysis, repair recommendation and covering the entire problem to resolution process.

AI can also be used to link network and services. Predictive insights proactively monitor and manage end-to-end service quality thus, enabling operators to prioritize actions based on impact to services and customer experience. 

Shaping Digital Re-invention and Personalized Engagement

Making the network available 24/7 is a great first step to improving personal experience, but it is not sufficient. Clients engagement with telco and media companies comprises different touchpoints, most of them digital, and a great experience has to be delivered at every single touchpoint. Hence, digital processes need to be re-designed around the customer experience like many new entrants have done.

Artificial intelligence is a key contributor to digital customer engagement.

In fact, AI enables business platforms to gather and leverage all relevant data emerging from customer interactions, providing a unique and complete view of the customer. This information is then used to support a personalized experience, that means a personal touch in the interaction, the anticipation of the customer needs, the resolutions of problems that have yet to be communicated and the offer of personalized packages.

This goes beyond the simple natural language interaction we discussed before. Most of the time, using virtual agents to provide customer support is the first step in the direction of a comprehensive redesign of the customer experience that needs to be proactive and not reactive and leverage all available information about the customer with the support of data platforms.

Recommended by a friend

Do you know why our friends are very good at recommending us the best TV show to follow or the next restaurant we should pay a visit to? Because they know us very well. Media companies need to achieve that same level of customer knowledge, intimacy & trust to keeping them engaged. More engagement means more screen time, more screen time leads to greater subscription and/or advertising revenues.

Artificial intelligence can turn media companies into a “friend” that recommends the best content. Media platforms already have access to a lot of user data (user profile, content history, payment ability) and they can integrate it with other sources like social media. Artificial intelligence leverages this data to recommend the best content and keep the viewer engaged and satisfied.  

On the other side of the ecosystem, advertisers need to maximize the return on their advertising investment. Media companies can now offer them the ability to show targeted advertising to on digital platforms, for example streaming services, matching the ads with the most receptive viewers, based on the information they gathered about them.

Personalization at scale

A personalized customer experience has a strong value and requires a high touch relationship, making it difficult to scale. Artificial intelligence applied to digital customer engagement has the ability to understand customers leveraging a large amount of data, engage with every customer in a consistent and personalized way and improve the organizations decision-making process over time. As a consequence, customers are pleased, satisfied and more loyal. 

LF AI & Data Resources

Formtek: Artificial Intelligence: Open Source and Open Data Team Up

By In The News

Late last year, two AI groups jointed forces to create an open source clearinghouse for projects related to AI, Machine Learning (ML), data, and analysis. LF AI Foundation and Open Data Platform Initiative (ODPi), previously separate Linux Foundation projects, merged to become the LF AI and Data Foundation.

The focus of LF AI had been to address issues like fragmentation and governance of open-source AI projects. ODPi’s focus was on data analytics. The group had created a project for managing the exchange of metadata between products and databases.

Read more at Formtek.