Automated use of data quality information currently stored in ARM Data Quality Reports
Authors
Sean T. Moore — Orbital ATK Inc.
Giri Prakash — Oak Ridge National Laboratory
Kenneth Kehoe — ARM Data Quality Office - University of Oklahoma - CIWRO
Raymond A. McCord — retired
Randy A. Peppler — University of Oklahoma
Category
Infrastructure & Outreach
Description
The ARM Climate Research Facility strives to provide datastreams of quality suitable for scientific research. The Data Quality Office, Instrument Team, Site Scientists, and others within ARM regularly review and assess ARM's datastreams for problems. Any issues discovered and confirmed as problems are summarized in Data Quality Reports (DQRs) and delivered as text files to users ordering ARM data. While valuable, it is impractical to use such reports with the large amount of data processed by most users. To improve the usefulness of these reports, we are developing methods to simplify application of the DQR data quality status to affected data.
The primary simplification will be to filter bad or suspect data before actually delivering it to the user. A custom netCDF file will be produced with data affected by a DQR marked as “missing.” Users will be able to decide if this should be done, and to what degree, during the ordering process. Simplified reordering procedures will ensure users can easily acquire and maintain data sets that incorporate the latest data quality information.
We will also provide a mechanism for users to query on-demand the latest known issues affecting a measurement or derived quantity. This mechanism, implemented as a web service, can be incorporated into data processing codes in order to identify and eliminate problem data as needed. Detailed documentation and code samples will be provided to help users utilize this service.
Feedback and suggestions for other methods to improve DQR dissemination are welcome.