ARM Data Quality Office: Fusion of Automated and Manual QA/QC Techniques
Poster PDF
Authors
Adam Theisen — Argonne National Laboratory
Randy A. Peppler — University of Oklahoma
Kenneth Kehoe — ARM Data Quality Office - University of Oklahoma - CIWRO
Justin Monroe — University of Oklahoma
Category
ARM infrastructure
Description
The Atmospheric Radiation Measurement (ARM) Climate Research Facility collects data from over 350 instruments on a daily basis. These data range from simple surface meteorological observations to scanning polarimetric radar data. The ARM Data Quality (DQ) Office is tasked with ensuring that data downloaded by end-users are of the highest quality.
The DQ Office employs a mixture of automated processing and problem identification techniques along with manual inspection of the data by undergraduate students to determine if there are problems with the instruments. If problems in the data are found, the appropriate personnel are informed, the instrument is repaired, and the data are flagged with a data quality report.
As ARM has grown to collect more and more data, so too have the DQ Office processes and tools. Currently, the DQ Office runs approximately 10,000 processes a day to test and visualize the latest data coming in. This results in the daily creation or updating of over 5,000 images and over 400 daily summaries of QC tests that are used in the analysis of the data. New methods are always being tested in order to increase the efficiency of the process. A recent upgrade to use multiprocessing techniques has greatly enhance DQ Office capabilities and will pave the way for new processes to review the ~25 years worth of climate data that ARM has collected since its inception in the hopes of further improving the data set.
