Forecast Revision Goals

1
Forecast Revision Goals

• Improve Reliability, Fault Tolerance, Recovery
• Measure and Improve Quality
• Change Management, Configuration Management,
  Standards, Documentation
• Performance
• Flexibility
• System Monitoring, Maintenance
• Facilitate Collaboration with CORIE Researchers

2
Towards Reliable Forecasts

• Forecast monitoring team
• Arun, Ethan, Paul
• Science, systems, software
• Team members cross-train in specialty
• Oncall rotation
• Monitoring and Alerting
• Big Brother
• Oversee Change Management

3
Managing Change

• Change and Configuration Management
• Development, production environments
• Deploy products from development to production
• Version control using CVS
• Standards
• Perl, C coding standards
• CORIE.pm
• libelio.a
• Documentation

4
Oncall

• 24/7
• Weekly rotation
• Respond to alerts received via E-mail, pagers and
  resolve problems whatever it takes.
• Oncall procedures page

5
Monitoring and Alerting

• In control and processing scripts
• Problems with model forcings
• Run fails to complete
• Processing problems
• Big Brother
• Monitors network connectivity, ping
• Network protocols e.g. HTTP, SSH
• Disk, CPU
• Specific processes e.g. master_process.pl

6
Measure and Improve Quality

• Error analysis
• 3 and 7 day error analysis (model data
  comparisons using database)
• Summarized values (averaged over all stations) to
  quantify forecast skill
• Comparisons with external forcings (river, wind
  (TBD))
• Comparisons (TBD)
• between forecasts
• With near term hindcast
• With field exercises
• Comparisons with verified data

7
Forecast Systems and Data Flow
8
Databases

• Postgresql
• Amb105 production DB server
• Amb104 backup production DB server
• Amb36 development DB server
• Ease of access via Perl DBI
• Automatic archiving of external data
• Telemetry (parallel with process on amb24)
• Verified data (TBD)
• Performance issues

9
Forecasts

• Reference (AKA Production)
• Experimental
• Development
• Near term hindcast

10
Reference Forecast

• Runs every day
• Controlled, infrequent changes
• Failure rate minimal, most stable forecast
• Atmospheric forcings from etaosu
• Hosted on amb1018

11
Experimental Forecast

• Runs like production mode
• Changes managed but more frequently allowed than
  reference
• Failure rate can be higher
• Failed forecasts need to be updated
• Atmospheric forcings from eta only
• Hosted on amb1017

12
Development Forecast

• Does not run in production mode
• Minimal results stored (3 days)
• Test changes to be incorporated in ref/exp
  forecasts, e.g. model forcings
• Development environment for new products and
  scripts
• Hosted on amb1019

13
Hindcasts

• Runs once a week for past week
• Parameter files based on previously set database
  (currently database06)
• Runs based on week number
• River forcings from relational database
• Atmospheric forcings from locally stored NOAA
  archive
• Hosted on amb1020

14
Forecast Forcings

• River forcings amb1020 daily
• 745,1045,1345,1645 getforcings.pl (to DB)
• Atmospheric forcings amb103 daily
• 0005 get_eta.csh (to NFS)
• 0010 get_gfs_air.csh
• Atmospheric forcings amb104 daily
• 0200 get_avn.csh (to NFS)
• 0400 get_mrf.csh
• 0930 get_osu.csh

15
Forecast execution

• On each forecast system daily
• 0010 simlink.pl on local directory
• 0010 simlink.pl on NFS directory
• 0900 do_error_analysis.pl (processing)
• 1100 place_hdf_files_new.csh
• 1125 prep.pl
• 1135 checkinputs.pl
• 1200 start.pl

16
Forecast processing

• Master process, runs continuously as a daemon.
  Executes on local disk looping over
• do_isolines.pl
• do_ll_isolines.pl
• do_transects.pl
• do_hab_isolines.pl
• do_plumevol.pl
• do_intrusionlength.pl
• extract_station_ADP.pl (from DB)
• extract_station_CTD.pl (from DB)
• do_stationextraction.pl
• do_stationplots.pl
• rsync to NFS

17
Hindcast Processing

• Uses same scripts as forecasts
• Remove differences between hindcast and forecast
  processing (2 vs 7 days)
• Some plot parameter file differences

18
Develop and Deploy

• Checkout module from CVS
• Modify, add codes on a local copy
• CVS commit
• Deploy to development environment
• Deploy to experimental environment
• Deploy to reference environment
• Development web page

19
Going Forward

• Improve monitoring in processing codes
• Failover for forcings, climatology
• Revise relational databases (per Bill H.)
• Tune BB threshholds and start paging
• Review current products
• Document procedures and products
• Migrate to new grid, quadrangles
• Forecast/forecast forecast/hindcast comparisons
  using verified data
• Comparisons with external forcings