Anomaly Detection for ARM Radiometers using Machine Learning Algorithms

 
Poster PDF

Authors

Laurie Gregory — Brookhaven National Laboratory
Jeffery Thomas Mitchell — Brookhaven National Laboratory
Lynn L. Ma — Brookhaven National Laboratory
Richard Wagener — Brookhaven National Laboratory
Laura Dian Riihimaki — CIRES | NOAA ESRL GML

Category

ARM infrastructure

Description

This plot shows the results of a machine-learning-based anomaly detection algorithm analyzing more than 2 years of data from a CIMEL sun photometer. Displayed are several known problems with this instrument that were identified by the algorithm.
We present an overview of machine learning anomaly detection algorithms developed for the the Cimel Sunphotometer (CSPHOT) and the Multifilter Rotating Shadowband Radiometer (MFRSR). These machine learning algorithms have been developed to increase efficiency and accuracy of data quality review. Generally, problems are detected by reviewing data daily and checking for data anomalies. Previously, many problems have been found by reviewing data by eye. However, there is much variation in the data set due to weather and function of the instrument. So there are problems that may go unrecognized until there is a clear trend or gross malfunction. A machine learning algorithm was developed to bring together multiple features in the data and detect these trends with more efficiency and accuracy than other methods. A Python script is being developed to train a multivariate regression model based on data taken when the instrument is functioning normally. This model is then used to quickly detect anomalies in the data set over long periods and to help with early detection of problems developing in the near term. One run for the CSPHOT instrument took only 30 seconds to identify all known problems over a 2-year period. Algorithm development has been guided by and verified using tools, plots, diagnostics, and data quality reports provided by the Data Quality Office. We have found that problems have been identified with high accuracy.