Preserving Cloud Information

1
Preserving Cloud Information

• Bruce R. BarkstromJohn J. Bates
• NCDC

2
Outline

• Fundamental Preservation Commandments
• Questions
• Variability Quantification
• Error Analysis and Physics
• Costs
• What Can We Do Now?

3
Four Commandments for Preserving Information

• Thou shalt not be forced to preserve information
  before it is ready
• Thou shalt not lose information if possible
• Thou shalt not cost more than necessary
• Thou must make data accessible and valuable
• To current users
• To future users

4
When is Data Ready for Preservation?

• When we have a good model of the underlying
  natural variability and expected climate
  change of the fields being measured
• Not just mean and standard deviation current
  applications need description of extreme events
• Need regional time variations
• When we have a physical basis for estimating
  errors and their impact on climate change
  detectability
• Need more than just measurement statistics
• Must include probability distribution of possible
  biases

5
Quantification of Field Variability

• The variability Turing test
• Can you generate an ensemble of computer
  generated fields with statistics that is
  indistinguishable from those of the real field?
• The climate Turing test
• Can you generate a model of trends whose
  statistics are indistinguishable from those of
  the expected climate changes?

6
Current State

• Measurement Requirements for Climate usually
  stated as global values of means and standard
  deviations
• Corresponding statistics can be generated by
  appropriate white noise
• Is this adequate?
• Probably not clouds variations are more complex
  than a global mean and simple latitudinal
  variations
• Can we come up with a common basis for stating
  variability across Earth science?
• Regional?
• Regional with moving systems?

7
No Preservation Without Understandable Error
Assessments

• Error assessments for climate data records are
  difficult
• Need physical basis for estimating uncertainties,
  not just internally consistent measurement
  statistics
• Error assessments must be tied to algorithm code
  data editing is as important as coefficients or
  outlines of algorithms
• Errors are not believable if entire data
  production process is not publicly understandable

8
Current State

• Algorithm Theoretical Basis Documents do not
  necessarily represent the as-built algorithms
  with their data editing
• EOS data production systems are overwhelmingly
  complex
• May need new documentation tools to provide
  understanding 100,000 lines of code is not
  readable in a Sunday afternoon
• As Science Teams disperse, community knowledge
  will be lost unless we take steps to prevent it
• May need to develop data scholars

9
Action Items

1. Can this workshop produce an understandable,
   quantitative description of cloud variability
   and of expected cloud property changes?
2. Is it possible to develop a community-accepted
   standard checklist of errors for cloud properties?

10
Sample Error Checklist

• Are the as-built instrument drawings available?
• Is the ground calibration data available?
• Is there a computational math model of the
  instrument that includes all of the physics of
  the measurement?
• How was the gain determined?
• How was the spectral response determined?
• How was the Point Spread Function measured?

11
Models for Preservation Funding

• The Cemetery Model
• Pay when the body is deposited live off the
  interest
• The Advanced Cemetery Model
• Pay for the previous bodies, as well as the one
  youre depositing make sure to add new bodies
  (the Cemetery as Pyramid)
• The Cemetery as Theme Park
• Make the cemetery interesting to visit charge
  admission
• The Public Broadcasting Approach
• Beg for support annually and ask for volunteers

12
Actions That Can Reduce Preservation Costs and
Risk

• Arrange a Submission Agreement (data will) with
  your designated archive
• Gather required original documents and make sure
  your archive can accept them
• Drawings
• Calibration plans and procedures
• Science Team minutes
• Source Code
• Arrange peer review of documentation

13
Summary

• Our data will not survive without careful thought
  to ensure
• Physical insight into the measured variables and
  the measurement process
• Adequate public access to the measurement process
• Cost-effective archival
• Archives know less than you do about your data
  if you dont act to preserve that information,
  archives cant preserve it!