How to make anomaly detection more accessible

What is anomaly detection?

Anomaly detection is a process of identifying unexpected items or events in data sets. From a statistical view, assuming there is a distribution of events, anomalies are considered as unlikely events with respect to a defined threshold. There is also a more event-oriented view of anomalies, as there are anomalous events that have some unexpected, and typically negative effects on the processes we are interested in.

Anomaly detection can be used to achieve improvements in important areas, such as:

• Reducing turnover time for application errors
• Ensuring high solution performance
• Identifying security threats before they happen

This applies to many businesses, and it is significant in the area of telecommunications. As we expand into 5G and Industry 4.0, there is a growing need to manage and monitor a large amount of data coming from more complex systems. An operator cannot extract insights from the fluctuations of thousands of different performance indicators, and anomaly detection with such interconnected processes becomes impossible to visualize with the naked eye, even for domain experts. A typical instance of an anomaly would be a sudden spike in base station load, or an increase in call drops in a network.

Owing to the recent developments in machine learning and AI, there is a growing need to democratize anomaly detection frameworks for a wide range of users. Businesses will need their people to become more data savvy in different aspects of their work, ranging from domain experts to business analysts to software engineers.

One project from Ericsson’s Global Artificial Intelligence Accelerator (GAIA) is aiming to make AI tools more accessible for anomaly detection: Introducing E-ADF, an in-house anomaly detection framework for quick and easy prototyping.

Motivation behind E-ADF

The idea behind E-ADF developed from multiple anomaly detection projects that were coming into the GAIA pipeline, many of which would need to start from scratch. Collaboration between data scientists working on those projects resulted in the idea to leverage reusable components from the code and build an anomaly detection framework. E-ADF is an Ericsson inner source project – inner source being the development of proprietary software using best practices from large-scale open source projects.

The project benefitted from contributions from the best anomaly detection practices in GAIA and Ericsson Research. The initial goal to create a reusable asset for data scientists evolved into a goal to create an easy to use AI platform for data scientists and non-data scientists alike. 

How does it work?

E-ADF facilitates faster prototyping for anomaly detection use cases, offering its library of algorithms for anomaly detection and time series, with functionalities like visualizations, treatments and diagnostics.

Fig 1. E-ADF Framework

The main features of E-ADF include:

• Interactive visualizers to understand the results of the features applied on the data.
• Detector explainer that provides explanations for data points were flagged anomalous (root-cause analysis).
• Ensemble model to combine the results of multiple algorithms.
• Diagnostics to identify if the time series data is stationary.
• Treatments to perform differencing, detrending, de-seasonalizing.
• The tagging tool to visually look at the data and add or remove anomalies.
• MULTIAD to compare the results of multiple algorithms.
• Pipeline to combine multiple steps like diagnostics, treatments, and forecasting.

This set of features will allow a user to prototype their anomaly detection use cases effectively all within the same framework.

The features in E-ADF are continuously expanding, with each release introducing new functionalities to continue to improve the quality of anomaly detection modeling. Included in E-ADF are algorithms to handle both univariate and multivariate data, features like rolling window and segmentation, a detector explainer to help find the root cause for certain anomalies, and pipelines.

Fig 2. Functionalities in E-ADF

There are existing open-source anomaly detection libraries being used today, with some of the most popular ones being PYOD and Prophet. The primary differentiator between E-ADF and already existing anomaly detection frameworks is its combination of anomaly detection algorithms as well as its extensive set of features. This helps in end-to-end prototyping of the anomaly detection use case from the preprocessed data to the anomaly detection results.

To illustrate how E-ADF works, the following diagram best describes a typical flow of anomaly detection use case modeling.

Fig 3. Typical Workflow of Anomaly Detection

In the following example we will go through the anomaly detection flow using the imputer, MULTIAD and Detector Explainer from E-ADF. _Time in Travel is the feature under consideration.  On input data without time series patterns like trend/seasonality, the first step is to perform missing values imputation using KNN Imputer and pass the data to multiple anomaly detection algorithms using MULTIAD. 

MULTIAD ensembling is one of the most distinguishing features of E-ADF.  This feature allows a user to compare the results from multiple algorithms at once and get a single, interactive visualization where they can compare different algorithms to choose the best one for the data they are investigating.

Three different anomaly detection algorithms from E-ADF are chosen: Isolation Forest, ZScore, and M­ultivariate Gaussian to apply to the treated univariate dataset. From the interactive visualization, we can observe that Multivariate Gaussian is identifying anomalies only on the spikes, but the Isolation Forest and ZScore are capturing both spikes and dips as anomalies.

Fig 4. Anomalies detected using Multivariate Gaussian

Fig 5. Anomalies detected using Isolation Forest

Fig 6. Anomalies detected using ZScore

On the anomalies detected, we use the surrogate explainer and interpret the results using the explainer tree to identify the root cause on how the anomalies are classified.

Fig 7. Explainer tree

These types of comparisons can be made easily by using the MULTIAD feature that can save time when choosing the best algorithm for a given use case.

What’s next?

The current users of E-ADF can use the established framework for their anomaly detection use cases. However, to expand the user base and make E-ADF more accessible for non-data scientists, the E-ADF development team is tackling four main areas:

• Creating a user interface.
• Growing the library of anomaly detection functions.
• Increasing the number of use cases in the E-ADF portfolio.
• Making E-ADF available on the Ericsson data science platform.

While E-ADF is in its early stages of making anomaly detection accessible outside the data science community, it is mature in its ability to aid in anomaly detection projects. With the increasing amount of data in the telecommunications domain, the need to identify anomalies increases with it. By making its anomaly detection capabilities accessible to a wider user base, E-ADF will help mitigate the growing need to autonomously manage and monitor data.

Acknowledgements

In writing this post, I would like to thank N Venkatachalam, Gautham Krishna Gudur, Krishna Sangeeth, A K Raghavan, Ebenezer Isaac, R Raaghul, Sharmili S, and Adithya K A for their work in the creation and continued development of E-ADF. Thank you Simone Vincenzi and Zeljka Lemaster for proof-reading and revision suggestions.

Discover more

Read Nikita Butakov’s blog post How to build robust anomaly detectors with machine learning.