RAINEX Data Management

1
RAINEX Data Management
RAINEX Planning Meeting, Miami, FL

• UCAR Joint Office for Science Support
• José Meitín
• Jim Moore
• Dick Dirks

2
RAINEX Planning Meeting, Miami, FL
JOSS Data Management

• Early involvement in project planning
• Involvement with PIs to develop data management
  strategy
• Consistent implementation of strategy for
  lifetime of project and beyond
• Reliable and efficient archive and distribution
  system
• Easy and efficient access to datasets by broader
  community

UCAR Office Of ProgramsJoint Office for Science
Support
3
RAINEX Planning Meeting, Miami, FL
Data Management Considerations

• Develop a consistent Data Policy
• Consider Data Types Formats and Documentation
  (metadata)
• Data Collection strategies (operations support,
  supplementary data)
• Real-time Requirements ( data calibration,
  intercomparison, and needs for operational
  decision making)
• Data Quality Control and Processing
  responsibilities
• Data Archive structure, functions and
  responsibilities
• Data Distribution protocols (user interface,
  metadata inventories, browse products)
• Collaboration with other programs and data
  centers (interoperability)

4
RAINEX Planning Meeting, Miami, FL
5
RAINEX Planning Meeting, Miami, FL
JOSS Data Services

• Data Questionnaire
• Data Management Plan
• Project Web Pages
• Real-time Operations and Data Ingest
• Field Data Catalog
• Quality control and dataset development
• Interactive Data Archive and Distribution
• Special media products (e.g. CD-Roms)

UCAR Office Of ProgramsJoint Office for Science
Support
6
RAINEX Planning Meeting, Miami, FL
JOSS Data Services Data Questionnaire
UCAR Office Of ProgramsJoint Office for Science
Support
7
RAINEX Planning Meeting, Miami, FL
JOSS Data Services

• Project Web Pages

8
RAINEX Planning Meeting, Miami, FL
JOSS Data Services

• Real-time Operations Support and Data Ingest
• Satellite Ground Station
• Forecast Products
• Numerical Model Analyses
• Operational Data Streams

UCAR Office Of ProgramsJoint Office for Science
Support
9
RAINEX Planning Meeting, Miami, FL
GTS METAR Observation Locations 6-10 November
2004
10
RAINEX Planning Meeting, Miami, FL
GTS Ship and Buoy Observation Locations 6-10
November 2004
11
RAINEX Planning Meeting, Miami, FL
Coastal and Marine Observation Locations
NDBC Buoy o TCOON (TX) CMAN TABS
(TX) o NWLON LUMCON (LA) o PORTS o WAVCIS
(LA) COMPS (FL)
12
RAINEX Planning Meeting, Miami, FL
Land Observation Locations
ASOS o LAIS (LA) AWOS TX ET (TX) AWOS
modem o FAWN (FL) o GAEMN (GA) TNRCC (TX) GFC
(GA) o ALMESO (AL)
13
RAINEX Planning Meeting, Miami, FL

• JOSS Data Services
• Field Data Catalog (WWW based)
• Daily Mission Reports
• Operations Summary
• Status Reports
• Data Analysis Products
• Authoring Tools

UCAR Office Of ProgramsJoint Office for Science
Support
14
RAINEX Planning Meeting, Miami, FL

• JOSS Data Services
• Quality control and dataset development
• Processing of research-quality, high resolution
  data
• Processing of standard operational data streams
• Compatible formats
• Composite datasets and data integration

UCAR Office Of ProgramsJoint Office for Science
Support
15
RAINEX Planning Meeting, Miami, FL
Composite Data Sets at UCAR/JOSS
A composite dataset is a collection (over some
time period and region) of similar data (e.g.
surface meteorological) from a variety of
sources, put into a common format, and passed
through a uniform quality control. Why does
UCAR/JOSS develop composites? - Provides data
in a uniform format with QC. - Allows
determination of network/site problems. -
Useful for model applications. - Prevents
duplication of effort.
16
RAINEX Planning Meeting, Miami, FL
KVII
CVT
Visual
IHOP_2002 Hourly Surface
Composite Development
ARM
HRLY
Visual
CVT
LTER
Visual
CVT
AWOS
HRLY
Visual
CVT
Gross Limit Checks
ASOS
HRLY
Visual
CVT
ABLE
HRLY
Visual
CVT
HPCN
Visual
Horizontal QC
CVT
MERGE
SCAN
Visual
CVT
NMSU
Visual
CVT
Examine Statistics
GWMD
Visual
CVT
MADIS
Visual
CVT
Ameriflux
Visual
CVT
Visual
NCAR (4)
HRLY
Visual
CVT
OKMESO
HRLY
Visual
CVT
Hourly Surface Composite
WTXMESO
HRLY
Visual
CVT
Others (7)
Visual
CVT
17
RAINEX Planning Meeting, Miami, FL
Aircraft Flight Level Datasets
NCAR / RAF NetCDF
NOAA / AOC standard format
JOSS RAINEX Archive
JOSS convert type 5 to NetCDF
Integrated RAINEX NetCDF
18
RAINEX Planning Meeting, Miami, FL
JOSS DATA MANAGEMENT TOOLS

• Field Catalog
• In-field tool to ingest and display operational
  and preliminary research data and project
  documentation for making real-time decisions and
  evaluating project progress
• Features
• Daily Mission Reports
• Operations Summary
• Facility Status Reports
• Data Analysis Products
• Authoring Tools
• Web-based access

• Interactive Data Archive/Access
• System
• Primary means for all project scientists and
  researchers to browse and retrieve data from any
  JOSS-supported projects
• Features
• Long-term field project data archival and
  distribution
• Interactive data browsing, subsetting, and format
  conversion
• Remote Archive Access
• Value-added datasets
• Data documentation

19
RAINEX Planning Meeting, Miami, FL

• JOSS Data Services
• http//www.joss.ucar.edu/codi
  ac/
• Interactive Data Archive and Distribution
• Ease of Data Access (web based)
• Data available via FTP, magnetic tape, CD-Rom
• Metadata descriptions
• Data Browse, Subsetting, Format Translation
  Capabilities
• Quality Control
• Distributed Data Archives
• Customized data ordering and reformat

UCAR Office Of ProgramsJoint Office for Science
Support
20
RAINEX Planning Meeting, Miami, FL

• RAINEX Data Management Issues
• Data Policy
• Data Management Plan
• Real Time Data Needs (model, satellite)
• Distributed Archive Responsibilities
• Special Data Processing (surface, upper-air,
  airborne radar aircraft flight-level)

21
RAINEX Planning Meeting, Miami, FL
http//www.joss.ucar.edu/
meitin_at_ucar.edu jmoore_at_ucar.edu dirks2_at_ucar.edu
UCAR Office Of ProgramsJoint Office for Science
Support
22
Acronyms
PORTS Physical Oceanographic Real-Time
System NWLON National Water Level Observation
Network TCOON Texas Coastal Ocean Observation
Network (TAMU Corpus) TABS Texas Automated
Buoy System (TAMU) LUMCON Louisiana
Universities Marine Consortium WAVCIS Wave
Current Surge Information System (LSU) COMPS
Coastal Ocean Monitoring and Prediction System
(USF) GAEMN Georgia Automated Environmental
Monitoring Network (U of Georgia) GFC Georgia
Forestry Commission ALMESO Alabama Mesonet
(Auburn U) LAIS Louisiana Agriclimatic
Information System (LSU) TX ET Texas
Evapotranspiration Network (TAMU) FAWN Florida
Automated Weather Network (U of Florida) TNRCC
Texas Natural Resources Conservation Commission
23
(No Transcript)