Evolution of the SZ-2 Algorithm

1
Evolution of the SZ-2 Algorithm
Sebastian Torres CIMMS/NSSL Technical
Interchange Meeting Fall 2006
2
Recent Evolution of SZ-2

• Several areas of interest identified and ranked
  by the ROC
• Solutions to critical AOIs were discussed at
  previous TIM
• Assessment of engineering fixes
• After-TIM discussions
• Provided draft AEL
• Analyzed ROCs implementation
• Last AEL delivered on June 9, 2006
• Real-time implementation validated with off-line
  MATLAB simulator
• SZ-2 implemented following latest AEL
• Many cases presented to the DQ team
• Problem with noisy velocities due to bug in the
  code
• TAC approved SZ-2 for Build 9 of ORDA

3
06/09/06 AEL Revisions

• Algorithm description fits real-time
  implementation
• dB-for-dB censoring
• Strong-point clutter suppression
• Required outputs (T0, R0, R1, R2)
• Addition of overall logic flow
• Efficient processing of non-overlaid echoes
• Dynamic use of data windows
• Spectrum width computations
• Unbiased autocorrelation for data windows
• Adaptable spectrum width estimator
• Censoring
• dB-for-dB
• Rules and classification
• Strong-point clutter suppression

4
Dynamic Use of Data Windows

• SZ-2 uses three data windows depending on the
  situation
• The PNF needs the von Hann (or more aggressive)
  window
• GMAP needs the Blackman window to achieve
  required clutter suppression
• Dynamic data windowing rules
• Use the rectangular window with non-overlaid,
  non-clutter-contaminated echoes
• Implemented algorithm uses the default window
  (currently the Hamming window)
• Use the von Hann window with overlaid,
  non-clutter-contaminated echoes
• Use the Blackman window with clutter-contaminated
  echoes

5
Spectrum Width Computations

• Rules for choosing a spectrum width estimator
• Use the R0/R1 estimator with non-overlaid echoes
• Use the R1/R2 estimator with overlaid echoes
• Unbiased spectrum width estimates require
  unbiased autocorrelation estimates
• The data window must be accounted for in the
  autocorrelation estimator

6
Censoring in SZ-2

• What is black and what is purple?
• Aimed at clear classification rules
• Black corresponds to non-significant returns
• Purple corresponds to gates that have
  significant returns but cannot be recovered
• Need to maintain accepted system behavior
• For example, dB-for-dB censoring is tagged as
  black
• Gates are classified as having one of the
  following three types of returns
• Signal return above adjusted SNR threshold and
  recoverable (passes all tests)
• Noise return below SNR threshold or strong
  clutter in non-overlaid case
• Overlaid return Unrecoverable with two or more
  overlaid trips (at least fails one of the tests)

7
Censoring Rules
Strong Trip Censoring Rules
8
Censoring Rules
Weak Trip Censoring Rules
9
Censoring Rules
Other Trip Censoring Rules
This censoring applies to the two weakest trips
10
Data Windowing Issues
Sebastian Torres CIMMS/NSSL Technical
Interchange Meeting Fall 2006
11
The Effect of Data Windows

• Data windows reduce the equivalent number of
  independent samples available to estimate
  spectral moments
• Non-rectangular windows are tapered so end
  samples contribute less to the estimation
  process
• The more aggressive the data window, the larger
  the errors of estimates for all spectral moments
• Data windows need to be accounted for in the
  autocorrelation estimator
• Normalization by lag window (DZ, ch5)

12
Standard Error of Moment Estimates for Different
Windows vs. SNR
Parameters of VCP 211 M 64, PRT 780 ms (PRI
8), f 2800 MHz, sv 4 m/s
13
Relative Standard Error of Moment Estimates for
Different Windows vs. SNR
Parameters of VCP 211 M 64, PRT 780 ms (PRI
8), f 2800 MHz, sv 4 m/s
14
Data Windows in SZ-2

• Default window with non-overlaid,
  non-clutter-contaminated echoes
• AEL recommends using the rectangular window
• ORDA uses the Hamming window as the default
  window
• von Hann with overlaid, non-clutter-contaminated
  echoes
• Blackman with clutter-contaminated echoes

Parameters of VCP 211 M 64, PRT 780 ms (PRI
8), f 2800 MHzTrue sv 4 m/s, SNR for Z sv
10 dB, SNR for v 8 dB
15
SZ-2 VelocityHamming Window
KCRI (ORDA)VCP 211 - 03/19/06
16
SZ-2 VelocityRectangular Window
KCRI (ORDA)VCP 211 - 03/19/06
17
SZ-2 and Super Resolution(NPI)
Sebastian Torres CIMMS/NSSL Technical
Interchange Meeting Fall 2006
18
What is Super Resolution?

• Legacy Resolution spatial sampling
• Reflectivity 1-km by 1-deg grid
• Doppler 250-m by 1-deg grid
• Super Resolution spatial sampling
• All moments 250-m by 0.5-deg grid
• Finer spatial sampling and smaller resolution
  volume lead to about 50 improvement in range of
  detection for mesocyclone and tornado signatures
  (Brown et al. 2002)

Tornado outbreak in Oklahoma City9 May 2003
(Curtis et al. 2003)
19
Super Resolution for NEXRAD
Z
RDA
RPG
RDA
v
Products
w

• Super-resolution data scheduled for operational
  use on NEXRAD
• Short-term goals - Phase I ORDA Build 10 (FY
  2008)
• Data used for visualization only
• Legacy- and super-resolution data available in
  the RPG
• RPG algorithms ingest legacy-resolution data
• Long-term goals - Phase II ORDA Build 13? (FY
  2012)
• Data used by the algorithms
• Super-resolution data produced on lower-elevation
  scans (split cuts)
• Higher likelihood of finding tornado and meso
  signatures
• SZ-2 may also run on these scans

20
Super Resolution on the ORDA (I)

• RDA must produce base data with finer spatial
  sampling and resolution
• Finer spatial sampling grid
• Radials collected at 0.5 deg azimuth intervals
• Legacy-resolution radials collected at 0.5, 1.5,
  2.5, deg
• Super-resolution radials collected at 0.25,
  0.75, 1.25, deg
• Recombined radials created at 0.5, 1.5, 2.5,
  deg
• No range averaging to maintain 250 m sampling in
  range
• Finer resolution
• Selective data windowing

Legacy-resolutionreflectivity grid
Super-resolutionreflectivity grid
21
Super Resolution on the ORDA (II)

• Solution Overlapping 1-deg radials with data
  windowing sampled every 0.5 deg and no range
  averaging
• For each range gate, M time-series data samples
  are weighted
• with von Hann window if clutter filtering is
  not needed
• with Blackman window if clutter filtering is
  needed

22
Super Resolution on the ORDA (III)

• Doppler-derived reflectivities and noise power
  are needed in the ORPG to produce legacy-like
  data
• Reflectivity is range unfolded together with the
  Doppler moments
• Noise power is added to metadata
• Doppler moments produced up to 300 km
• Data beyond 230 km is not discarded
• Throughput is doubled
• Twice the number of radials in an elevation cut
• Computational complexity is doubled
• Super-resolution radials have the same number of
  samples as legacy-resolution radials

23
Super Resolution on the ORPG (I)

• ORPG algorithms expect data with legacy
  resolution and quality
• Super-resolution data does not have the required
  resolution or quality for the algorithms (its OK
  for visualization)
• Super resolution radial recombination
• Two super-resolution radials are recombined to
  form one legacy-resolution radial
• Recombination assumes a bimodal spectrum
• Approach exhibits low risk and provides data
  with acceptable quality
• Algorithm must deal with missing data
• SNR thresholds
• Overlaid echoes

24
Super Resolution on the ORPG (II)

• Velocity dealiasing algorithm must run on
  super-resolution and recombined legacy-resolution
  data
• Both velocity fields will available for
  visualization
• Legacy-resolution fields will be fed to the
  algorithms
• Increased processing requirements

25
SZ-2 Changes to run inSuper Resolution mode

• ORDA
• Finer spatial sampling grid no changes
• Finer resolution no changes
• Default window for super resolution will be set
  to von Hann
• Range unfolding of Doppler reflectivities no
  changes
• Unfolded Doppler powers already exist within SZ-2
• Addition of noise power to metadata no changes
• Doppler data up to 300 km no changes
• Data already exist within SZ-2
• Doubled throughput no changes
• Doubled CPU load needs testing
• Current maximum CPU usage with SZ-2 is 35
• Expect maximum CPU usage of 70 with
  super-resolution SZ-2
• ORPG
• SZ-2 is already transparent to the ORPG no
  changes

26
Reflectivity FFTLegacy resolution
27
Velocity FFTLegacy resolution
28
Reflectivity SZ-2Legacy resolution
29
Reflectivity FFTLegacy resolution
30
Velocity SZ-2Legacy resolution
31
Velocity FFTLegacy resolution
32
Reflectivity SZ-2Super resolution
33
Reflectivity SZ-2Legacy resolution (Recombined)
34
Reflectivity SZ-2Legacy resolution
35
Velocity SZ-2Super resolution
36
Velocity SZ-2Legacy resolution (Recombined)
37
Velocity SZ-2Legacy resolution
38
All-Bins Clutter Filtering and SZ-2
Sebastian Torres CIMMS/NSSL Technical
Interchange Meeting Fall 2006
39
All-Bins Clutter Filteringand SZ-2

• SZ-2 cannot recover overlaid signals if multiple
  trips have clutter contamination (overlaid
  clutter)
• Operator-selected all-bins clutter filtering
  forces overlaid clutter in every bin
• A large percentage of the bins will not have
  clutter contamination
• Is there a simple way to detect which bins have
  clutter contamination?
• Answer GMAP

40
All-Bins Clutter Filteringand SZ-2

• GMAP is used to detect clutter
• The clutter power removed by GMAP during the long
  PRT is used as an indicator of the presence of
  clutter
• Recommended SZ-2 algorithm uses the long-PRT CSR
• Preliminary tests show that the long-PRT CNR may
  be a better indicator
• We rely on only one estimate from GMAP

41
Reflectivity All bins
KCRI (ORDA)VCP 211 - 03/19/06
42
Reflectivity Bypass Map
KCRI (ORDA)VCP 211 - 03/19/06
43
Velocity All bins GCF Clutter if CSR gt 15 dB
KCRI (ORDA)VCP 211 - 03/19/06
44
Velocity Bypass Map
KCRI (ORDA)VCP 211 - 03/19/06
45
Velocity All bins GCFClutter if CSR gt 10 dB
KCRI (ORDA)VCP 211 - 03/19/06
46
Velocity All bins GCF Clutter if CSR gt 15 dB
KCRI (ORDA)VCP 211 - 03/19/06
47
Censoring All bins GCFClutter if CSR gt 10 dB
KCRI (ORDA)VCP 211 - 03/19/06
48
Velocity All bins GCFClutter if CNR gt 10 dB
KCRI (ORDA)VCP 211 - 03/19/06
49
Censoring All bins GCFClutter if CNR gt 10 dB
KCRI (ORDA)VCP 211 - 03/19/06
50
Velocity All bins GCFClutter if CNR gt 20 dB
KCRI (ORDA)VCP 211 - 03/19/06
51
Velocity All bins GCFClutter if CNR gt 30 dB
KCRI (ORDA)VCP 211 - 03/19/06
52
Velocity All bins GCFClutter if CNR gt 40 dB
KCRI (ORDA)VCP 211 - 03/19/06
53
Velocity Bypass Map
KCRI (ORDA)VCP 211 - 03/19/06
54
Velocity All bins GCF Clutter if CSR gt 15 dB
KCRI (ORDA)VCP 211 - 03/19/06
55
Velocity All bins GCFClutter if CNR gt 40 dB
KCRI (ORDA)VCP 211 - 03/19/06
56
Thank you!