Clarus Quality Checking Algorithm Enhancements

1
ClarusQuality Checking Algorithm Enhancements
Clarus ICC5 Meeting Session 2

• Andy Stern
• Consulting Meteorologist (Noblis)
• FHWA Road Weather Management Team

2
In the beginning

• There were 16 proposed QCh algorithms

• Sea Level Pressure Test
• Optimal Interpolation Spatial Test
• Potential Temperature Test
• Theoretical Solar Radiation Test
• Precipitation Amount Test
• Wind Direction Test
• Soil Moisture Change Test
• Soil Moisture Freeze Test

• Manual Change QCh Flag
• Sensor Range Test
• Climate Test
• Barnes Spatial Test
• Step Test
• Persistence Test
• Like Instrument Test
• Dew Point Temperature Test

3
Clarus QCh Algorithms

• 9 of the 16 proposed QCh algorithms were
  implemented

• Sea Level Pressure Test
• Optimal Interpolation Spatial Test
• Potential Temperature Test
• Theoretical Solar Radiation Test
• Precipitation Amount Test
• Wind Direction Test
• Soil Moisture Change Test
• Soil Moisture Freeze Test

• Manual Change QCh Flag
• Sensor Range Test
• Climate Test
• Barnes Spatial Test
• Step Test
• Persistence Test
• Like Instrument Test
• Dew Point Temperature Test

4
QCh Task Force

• Task Force held in Boulder 5/31-6/1 2006
• 22 attended, with a diverse mix of backgrounds
• 3 Federal Govt
• 2 Canadian Govt
• 4 DOTs (State local)
• 7 Academia/National Labs
• 6 Private sector

5
QCh Task Force Sessions

• Current Algorithms Summary of algorithms used
  in Clarus and other popular data ingest and
  dissemination systems (Mesowest, MADIS)
• QCh Gaps Gaps identified include pavement and
  subsurface algorithms, algorithms using
  integrated ancillary data sets (e.g., radar)
• QCh Summaries understanding the needs of the
  end users

6
QCh Algorithms the Next Generation

• FHWA compiled a plan for a next generation of
  Clarus QCh algorithms
• Components of the plan include
• Improve existing algorithms
• Current persistence test only checks observations
  that are 60 minutes apart. This should be
  modified to use available data at higher
  frequencies as they become available

7
Improve Existing Algorithms (cont)
1 of 2

• Improve the sea-level pressure test. Current
  technique is optimized for use in the plains.
  Improve the technique for stations in mountainous
  terrain
• The current climate range test requires that
  nearly all observations have site specific
  climate range values prepared. Each parameter
  should be evaluated to see if they need to be
  tested. This is both an optimization, metadata
  and workload issue

8
Improve Existing Algorithms (cont)
2 of 2

• QCh algorithms currently use ASOS data. Evaluate
  if additional surface/marine data sets should be
  incorporated for use in checks
• Much of the adaptable and derived parameters used
  in QCh algorithms are not made available to end
  users. Without these data, it may be impossible
  to determine the exact reason for a failure flag.
  Explore which parameters should be made available
  in QCh summary output

9
Evaluating New Algorithms
1 of 3

• New algorithms may benefit Clarus using a more
  varied supplemental data set
• Spatial quality checking Clarus currently uses
  a Barnes Spatial Analysis (BSA) scheme for
  neighbor checking. The BSA may benefit from
  adding coastal near shore marine observations.
• Neighbor Testing determine if a Barnes Spatial
  Analysis (BSA) should be run versus an Optimal
  Interpolation (OI) scheme. BSA works well in flat
  areas. OI may work better in mountainous areas.
  Should both be run?

10
Evaluating New Algorithms (cont)
2 of 3

• Precipitation estimation at ESS sites using
  Doppler Weather Radar data. Determine best
  methodology to use for mosaicked products,
  mountainous terrain, derived precipitation
  amounts, etc.
• Recommend new ways of performing pavement and
  subsurface QCh tests (e.g., satellite,
  model-forecasted insolation, use of Long Term
  Pavement Preservation (LTPP) data, NCEPs Real
  Time Mesoscale Analysis, etc)

11
Evaluating New Algorithms (cont)
3 of 3

• Create an overall observation confidence value
  utilizing all QCh flags to create one value to
  describe the overall quality of the observation.

12
Deliverables

• Documentation of all quality checking algorithms
  and data quality flags will be created to be used
  by both technical engineers and laymen
• Results will be reported in a scientific journal
  or conference preprint