An Independent Technical Review Process for Government Developed Models

1
An Independent Technical Review Process for
Government Developed Models

• HASDM As a Case Study
• Michael J. Gabor

2
Outline

• Introduction
• JAWG
• HASDM
• Technical Review Process
• Lessons Learned
• Summary and Conclusions

3
Introduction

• U.S. Space Command
• JAWG
• HASDM

4
JAWG

• Originally associated with SSWG
• Led by USSPACE/J33 and AN
• Then leadership transferred to AN
• Coordinate RD of SSA (especially space
  surveillance) between services
• Attempted USSPACE Astro standards

5
HASDM

• DCA
• Use catalog to determine common corrections that
  are expected to account for common atmospheric
  characteristics
• Various approaches to DCA exist
• HASDM
• SSULI - Special Sensor Ultraviolet Limb Imager
  (NRL)
• Nazarenko
• EDR (Storz)
• E10.7
• New solar flux proxy intended to replace F10.7
• Based on satellite EUV measurements

6
Technical Review Process

• Pre-meeting info dissemination
• Pre-meeting QA
• Briefings
• QA
• Feedback forms
• Organizational memos
• Feedback filtered for biases, inaccuracies,
  ignorance, attribution

7
Feedback Survey

• Respondent information (not used for report)
• HASDM Project Comments
• Assumptions
• Consistency of methods
• Appropriateness of tests (complete, independent)
• Meaningfulness of results
• Traceability of conclusions
• Additional thoughts

8
Feedback Survey

• Technical Review Comments
• Success of review
• Prior expectations
• Difference between expectations and results
• Value and effectiveness of process
• Recommendations to improve
• Additional thoughts

9
Review Outcome

• HASDM evaluation - too much to present - see
  paper for examples

10
Lessons Learned

• Model development
• Specific criteria need to be established to
  enable objective evaluations
• Options should be investigated and documented
  first (approach, baseline, parameterization,
  validations)
• Design tests that are independent

11
Lessons Learned

• Review process
• Coordination of review needs to be improved
• Participants were not sure who briefs, who pays,
  etc.
• Insufficient time for review
• Lack of formalism in process
• No mechanisms for tracking resolution of
  criticism
• After USSPACECOM demise, no accountability to
  process

12
Recommendations - HASDM

• E10.7 was not ready for implementation
• The progress of the DCA portion of HASDM merited
  further evaluation
• Establish evaluation criteria
• Investigate other options
• Conduct/Improve tests for performance evaluation
  and operational suitability
• Perform cost/benefit analysis
• Distribute as much data as possible to the
  research community to improve the science and
  allow for other ideas to mature

13
Recommendations - Technical Review

• Process should be applied new models or
  significant modification
• Review needs to become requirement for
  operational implementation
• Organization needs to be assigned responsibility
• Needs authority, independence, and impartiality
• Need mechanisms for tracking resolution

14
Summary and Conclusions

• Review process used only once for HASDM
• Identified criticisms of execution in model
  development and highlighted areas of concern
• Serves as example for future reviews
• Technical review is necessary to gain community
  buy-in and identify loose ends
• Data and technical info should be made available
  to industry and academia
• Overcome institutional inertia obstructing team
  work, cooperation, and interoperability