Breakout Summary Report

ARM/ASR User and PI Meeting

10 - 13 June 2019

Session Title: CM Data Consolidator Wizard Demo
Session Date: 12 June 2019
Session Time: 1:30 PM - 3:30 PM
Number of Attendees: 20
Summary Authors: Krista Gaustad, Carina Lansing

Breakout Description

Do you want to spend less time preparing data for your analysis and more time doing the analysis? Come to our session to see how the new ARM data consolidation wizard can help you take inputs from different continuous ARM time-centric data sets and combine them to produce a single output product with just the measurements you need and on a time scale of your choosing. The wizard walks users through selecting the input data sources to be consolidated and creating an output product for a specific location and time range. The user can save this information in the wizard to access it in the future to run against different locations and time periods.

Main Discussion

The ‘data_consolidator’ is an application that performs coordinate dimension transformations and mappings from retrieved variables to output variables for any process defined in the PCM. It allows users to consolidate data from diverse datastreams without the need to create or compile any source code. The ARM data consolidator application can now be accessed via a wizard integrated into ARM’s Data Integration (ADI) graphical interface. The wizard interface provides simplified workflows for the most common data integration use cases and simplifies process setup by defining reasonable default values for the many parameters that ADI allows users to control. It also allows users to run the ADI data consolidator for a given process directly from the PCM interface, view plots of the results, and download integrated data sets and plots. Using the wizard, scientists and PIs can immediately organize data from multiple ARM data sources into a single data product that has the sampling interval required for their analysis and meets ARM DOD standards. This capability decreases algorithm development costs and increases the usability of ARM data by the broader scientific community.

A high-level overview of the ADI framework and the Data Consolidator Wizard’s capabilities and features were presented as background. Participants were then invited to participate in a hands-on demonstration of a typical data consolidation use-case via a temporary username and password. The exercise included:

• Entering the input- and output-related information into the wizard.
  
• Running the data consolidator application from the graphical interface.
  
• Plotting the results.
  
• Accessing the data.
  
• Evaluating results by reviewing process logs, debug information, and internal data structures.
  
• Reviewing how to request the necessary ARM account to access and use the wizard.
  
• Reviewing the wizard’s advanced capabilities using integrated online help.
  
• Using the ADI Data Object Design (DOD) interface to identify input data sources.

Key Findings

All participants were able to understand and navigate intuitively through the wizard and understood the options in selecting variables to retrieve and setting up the common coordinate grid. They were appreciative that the data consolidation integrated the retrieved data’s QC into the transformed product and that they could apply additional QC to transformed measurements by setting limits on transformation metrics captured by ADI. The most enthusiastic participants were atmospheric science graduate students and ARM PIs. Several asked if the wizard would be integrated into the Data Center ordering interface.

Issues

The wizard leverages the existing Data Quality Office (DQO) libraries for creating plots, as these libraries have been well tested and have many advanced features. Leveraging these existing capabilities is much more efficient than creating new applications. We would like the plots to also include output QC, but we found that the latest DQO libraries are not completely operational with RedHat7 and are not actively supported.

Needs

N/A

Decisions

Based on feedback from participants in the demonstration and from Laura Riihimaki who has been an alpha tester, it was concluded that there is enough interest from PIs and the ARM scientific user community to pursue examining the possibility of integrating the ADI wizard into the ARM Data Center data ordering interface.

Future Plans

Future capabilities of the wizard that will be supported next FY include:

• Plotting with output QC.
  
• Supporting hourly, monthly, and yearly output in addition to daily.
  
• Rolling up QC tests into one test representing all ‘bad’ conditions and one representing all ‘indeterminate’ conditions.

Action Items

The following near-term improvements were identified during the demonstration either as explicit requests or to resolve confusion encountered. These will be available on the beta interface (link available via the i.arm.gov PCM tile) by the end of July:

• Make it easier for users to open an existing process in ‘wizard’ mode.
  
• Add automatic support for retrieving alt, lat, and lon variables needed for plotting.
  
• Delete all corresponding logs, plots, data, etc. on the file system when a process is deleted.
  
• Support creating plots via the dq_inspector tool in order to include output QC. (Completion by end of July is dependent on DQO library updates being made in a timely manner.)

The following action items will be done before the end of FY19:

• Continue discussions with key ADC staff that were initiated at the ASR meeting on the value of integrating the ADI wizard into the existing ARM data ordering interface.
  
• Discuss VAP plotting requirements with DQO and dq_inspector software owners to ensure that support for VAP plotting is maintained.