Scoring:%20a%20novel%20approach%20towards%20automated%20certification%20of%20pipeline%20products

1
Scoring a novel approach towards automated
certification of pipeline products
Reinhard Hanuschik, Wolfgang Hummel, Mark Neeser,
Burkhard WolffData Processing and Quality
Control Group European Southern Observatory
2
Complexity and mass production

• VLT 11 instruments
• VLTI 2 instruments
• 2009/10 2 survey telescopes
• roughly 50 modes
• about 150 data types
• most of them pipeline-supported

3
Complexity and mass production

• each data type or instrument component controlled
  by QC1 parameters
• metrics for instrument performance extracted by
  data pipelines
• stored in database
• data frequency varies on timescales between a day
  and a semester

4
Complexity and mass production
two challenges

• the complexity challenge!

5
Complexity and mass production

• current VLT instruments

• the volume challenge!

6
The solution scoring
The solution

• scoring well-established outside astronomy

• QC process
• measure quality
• compare quality
• assess quality score and certify

7
Scoring the solution

• measure quality done by automatic procedures
• pipelines
• other QC procedures
• implemented for CALIB data
• progress on SCIENCE data

8
Scoring the solution

• compare quality trending
• first step towards assessment
• same/different behaviour as others?

9
Scoring the solution

• assess quality scoring
• compare new result to trending
• pick out outliers and flag them

10
Scoring the solution

• required thresholds
• statistical thresholds e.g. /- 3 sigmas
  independent from specs- sensitivity
  variablecontrol charts
• specified thresholds e.g. lower/upper level
  stable thresholds- requires careful
  configuration and thinking

11
Scoring of pipeline products

• each new pipeline product is scored
• most relevant QC1 parameterscompare to
  trending, score as
• OK 0
• NOK 1 (outlier)
• count scores, assign total score, e.g.

12
Scoring of pipeline products
score report
13
Scoring of pipeline products
score reports for multi-detector instruments
(CRIRES N4)
detailed information on demand!
14
Scoring of instrument health

• all scores go into database
• re-arrange them by instrument properties
• detector health
• calibration lamp performance
• system efficiency
• ?score-based instrument QC

15
Scoring of instrument health

• example detector properties for GIRAFFE

instead of
16
Scoring of instrument health

• and finally

www.eso.org/HEALTH
17
Conclusions

• scoring automatic flagging of outliers
• useful to auto-certify pipeline products
• quick-look monitoring of instrument health
• powerful concept to handle large and complex data
  sets
• hierarchical approach from global overview
  (instrument score) to QC1 parameter per product
  and detector (6 levels)