Intersite Coordination Performance and Enhancements

1
Intersite CoordinationPerformance and
Enhancements

• Mark Mathewson

2
Overview

• Items to help reduce processing requirements.
• Configuration Items
• Forecaster Methodology
• What is planned for future revisions?
• RPP ISC Focus Group

3
ISC Internals
4
Spatial Mosaic
5
Temporal Mosaic
6
ISC Processing

• Consumes a lot of resources
• Shouldnt be on the lx1 box
• Px processor plans
• Volume of traffic determines resources needed.
• Forecaster can control traffic volume
• Focal Points can also regulate volume through
  configuration

7
Performance Characteristics

• Number of Grids
• Hourly grids, vs. longer grids
• Number of weather elements
• Number of Gridpoints per Grid
• 5km resolution typically 25k points
• 2.5 km resolution typically 100k points
• AWIPS H/W, S/W sized for 25k points

8
Performance Characteristics

• Three Processing Steps
• Creation of netCDF file for transmission
• MHS Sending over WAN
• Receive processing of netCDF file, remapping
  projection, and mosaic operations.
• Full set of grids 100 million gridpoints,
  handled multiple times. Can cause delays of 20
  minutes.

9
Methods to send ISC Data

• Capabilities
• Choice of SITE
• Choice of time range
• Choice of weather element
• Choice of OFFICIAL or FCST database

10
Methods to send ISC Data

• Dialogs
• Send Intersite Grids dialog
• Publish dialog
• Save dialog
• MB3 popup menu over grid block in GM
• Send on Auto Save (IFPS13)

11
Send ISC Grids Dialog
12
Publish Dialog
13
Save Dialog
14
Are you duplicating ISC sends?

• Options have been added per field request.
• Options used together can cause duplicate grids
  to be sent and received.
• Forecast Methodology important in determining the
  right method to use.
• Future s/w changes can alleviate problems.

15
Sending Guidelines

• Dont simply send all of your grids all the time.
  For example,
• Send when save
• Send Intersite Coordination Grids dialog
• Publish dialog
• Triplicate grid sends dont help performance.

16
Sending Guidelines

• Send grids as you do your work basically a few
  at a time.
• Sending when Save is a good idea
• Auto Save and Sending
• Configure your system to
• Preset the dialogs to only include certain times
  or elements

17
Sending Guidelines

• Send single grids to single site when working out
  a coordination issue
• Quick delivery/processing of grid - lt 1 minute

18
Configurability (hidden)

• Server-Based (IFPS13)
• MAX_ISC_GRIDS_PER_TRANSACTION
• to optimize use of WAN
• ISC_NEVER_RECEIVE_PARMS
• to toss incoming data based on weather element
  name
• GFE-Based (IFPS12)
• ISC_neverSendParms
• List of weather element names to NEVER transmit
• ISC_sendLimitBeforeCurrentTime,
  ISC_sendLimitAfterCurrentTime
• Limit transmissions based on current time
  (0-7days)

19
Configurability (GUI)

• SendISCTimes
• List of time periods on SendISC and ISC
  Discrepancy Dialogs
• SendISCGridsWhenSave, SendISCGridsWhenAutoSave,
  SendISCGridsWhenPublish
• Controls default button position on dialog
• Controls auto save automatic sends

20
Configurability (GUI)

• ISC_Sites
• List of sites to which ISC data may be sent
• SendISCGridDialogInitialWEGroup
• Pre-selected set of weather element on SendISC
  Dialog

21
ISC Enhancements
22
ISC Focus Group Purpose

• Help define ISC capabilities
• Suggest changes to ISC
• Help prioritize changes
• Test prototype software with changes prior to
  AWIPS deployment
• First meeting was last Thursday.
• Ideas presented here are under discussion.

23
Discrepancy Algorithm

• Highlights Grid Manager if discrepancy thresholds
  are exceeded
• All it takes is ONE grid point to exceed the
  threshold to highlight the grid in the grid
  manager.

24
Discrepancy Algorithm

• Color code Grid Manager
• RED YELLOW GREEN
• Based on points that exceed threshold
• Based on mean absoloute error
• Based on ???
• Add Discrepancy Tools for
• Vector Direction, Wx, and Discrete Fields

25
Difference Grid

• Show_ISC_Grid smart tool

83
56
Fcst ISC grid TDiff
Original Fcst/ISC Grids
Difference Grid
26
Difference Grid

• All points outside the calculated area are set to
  0.
• Could expand that with the ISC-Fcst algorithm
• All points are set to 0, if the threshold is not
  exceeded anywhere
• Even if there is some discrepancies.

27
Multiple ISC Databases

• Two ISC Databases Fcst and Official
• Purpose
• Separate out the official ISC grids from the Fcst
  ISC grids
• Good for product generation (to get official data
  only)
• Issues
• Performance

ISC
ISC Official
ISC Grid
28
Show ISC Mode

• Current problem with QPF/SnowAmt
• If your Fcst grid is 12hr long, and you have ISC
  grids that are only a few hours long, you will be
  comparing QPFs for different time periods.
• Possible solution
• Virtual weather elements (hidden), but will sum
  up the data to compare apples with apples.

29
Show ISC Mode
30
Show ISC Mode
You fcst 0.70, So do your surrounding WFOs.
Comparing apples to apples
Fcst Grid
0.70
ISC Grid
ISC Grid
ISC Grid
ISC QPF
0.40
0.20
0.30
31
Virtual ISC Grids

• Transparent to user
• History Update Times?
• What to do, use the oldest of the times, or the
  newest? What about gaps?
• Do we lose other features?
• Original ISC data is always available

Virtual ISC Grid
0301z, 0542z, or 613z?, who edited it?, original
source?
ISC Grid
ISC Grid
ISC Grid
ISC QPF
0301z, Eta, mark
0613z, Create Scratch, Tom
0542z, interpolated, Tracy
32
Show ISC Mode virtual elements

• Applies to accumulative weather elements
• SnowAmt, QPF
• Applies to maximum or minimum-based weather
  elements
• MaxT, MinT, PoP

33
ISC Aging in the database

• ISC grids are never aged.
• If a site last transmitted a week ago, then those
  grids are still present
• Aging is different from Purging
• Should we eliminate all grids from a site when we
  dont receive data for xxx hours?

34
ISC Data Request

• ISC data is broadcast only
• Do we need a request facility, that could look at
  another sites database and get their current
  state?

35
Remove some of the send options

• Do we have too many options available for sending
  ISC grids?
• For Example Can we remove the capability of
  sending to an individual site?
• If so, then bookkeeping gets easier for other
  possible enhancements.

36
Other Possibilities

• Automatic running of discrepancy tools as ISC
  data arrives
• More informative messages in status bar on
  arrival of ISC data
• Auto loading of ISC weather elements when needed
  (e.g., when running tools)

37
Summary

• Performance is currently a big issue.
• H/W solutions
• Forecaster methodology
• S/W performance improvements
• S/W Changes
• Functionality changes are on-the-way
• RPP ISC Focus Group will determine these.