Surviving the deluge (part II): planning the ARM “Big Data” system for infrastructure and research processing of terabyte scale data volumes

Poster PDF

Description

In 2010, ARM will be installing many new radar and lidar instruments for 3D atmospheric measurements. These instruments generate very large volumes of data (>10 GB and 100s data files per day) for basic data products. These data sizes will increase the overall data collection rate for ARM by a minimum of 4x. Processing tasks requiring more than a few months of these data result in Terabyte scale problems. To meet the expected data deluge, the ARM infrastructure will be implementing at the Archive a new data system specifically designed for very large-scale data tasks. Part of the historical experiences and trends in managing and delivering ARM data is relevant to this new installation. However, the Big Data system will have many additional requirements. Accordingly, this poster will have two main goals: (1) To outline current thinking on the problem (hardware, software, and policies) and (2) to engage users in the design process for a better understanding of the issues. The poster will be organized around the following themes: (1) What will the Big Data system look like? (2) What tasks will the new system be required to support? (3) Will the system support visualization, data extraction, and analysis? (4) What are the user expectations? In this context, key points from the recent survey conducted by ARM management about the ARM Computing Environment will be presented. For example; what kind of analysis facility will this system need to provide to researchers and infrastructure? (5) Key policy and operational issues and alternatives will also be presented. For example, will the system need to operate in batch mode? If so, how will tasks be defined and submitted for processing? During the STM meeting the presenters of this poster will actively seek additional feedback on these issues from ASR Science Team members.