Quality Audits

1
Quality Audits

2
Supporting documentation

• Bernhard, G., and G, Seckmeyer, Uncertainty of
  measurements of spectral solar UV irradiance,
  J. Geophys. Res., Vol. 104, No. D12,
  14,321-14,345 (1999).
• Gardiner, B.G., QA in ultraviolet
  spectroradiometry the state of the art, WMO
  QC/QC working group.
• Heikkilä, A., et al., SUVDAMA data storage
  Edition 4, Final SUVDAMA Report (1999).
• Seckmeyer, G., et al., Instruments to measure
  solar ultraviolet radiation Part 1 Spectral
  instruments, version 18 from the Final SUVDAMA
  Report (1999).
• Webb, A., et al., Guidelines for site quality
  control of UV monitoring, WMO/GAW Rep. 126,
  World Meteorological Organization, Geneva, WMO
  TD No. 884, 39 pp.

3
Test audit
4
Objectives (1)

• To provide assurance to data users that a certain
  level of quality has been achieved
• The site operator should be able to persuade an
  interested visitor that the measurement and
  collection of data are carried out diligently and
  to a standard that justifies any claims made to
  the quality of the data.

5
Objectives (2)

• To provide site operators with information that
  will allow them to improve the standard of their
  QC and QA procedures, and thus the value of their
  data
• The final report will attempt to identify and
  promote best practice

6
Requisites (1)

• Mission statement
• The site operator should give a description of
  the site objectives in relation to the quantity
  and quality of data collected.

7
Requisites (2)

• The site operator should be able to present a
  detailed and comprehensive description of the
  calibration, measurement and data analysis
  procedures, together with the results of their
  customary application.
• The site operator should be able to make a
  quantitative statement about the accuracy of the
  measurements and be able to justify this
  assessment.

8
Checklist
5.    Measurement regime 5.1.     standard
operating procedures 5.2.     type and quantity
of data collected 5.3.     resolution and freq.
of spectral measurements 5.4.     typical
downtime for repairs/maintenance 5.5.    
documentation 5.6.     demonstrable
repeatability 6.    Data Analysis 6.1.   
documentation 6.2.    analysis of errors 6.3.
  correction for errors 6.4.     use of
ancillary measurements 6.5.     estimates of
relative/absolute errors 6.6.     application of
WMO guidelines 6.7.     measurements of slit
functions 6.8.     procedures for the
application of retrospective corrections 6.9.  
   data storage 6.10.  QC tools available and
applied 7.    Quality control 7.1.    
documentation 7.2.     standard
procedures 7.3.     details of past/future
instrument intercomparisons 7.4.     details of
data version control
3.    Instrumentation 3.1.     list description
of instruments 3.2.     documentation of
instrument characteristics 3.3.     Freq. of
regular maintenance characterisation 3.4.    
maintenance record 3.5.     past changes in
instrumentation 3.6.     planned changes to
instrumentation 3.7.     measurement of angular
response 3.8.     measurement procedure for slit
function(s) 3.9.     photographs of
instruments   4.    Calibrations 4.1.    
documentation 4.2.     certificates 4.3.    
frequency of calibrations 4.4.     definition
of fiducial plane 4.5.     measurement of lamp
distance 4.6.     method for wavelength
calibration 4.7.     typical instrument drift
between calibrations 4.8.     evidence that the
calibration is reliable at the quoted level of
accuracy 4.9.     age of lamps 4.10.  level of
agreement between lamps (internal
consistency) 4.11.  level of repeatability/consis
tency in calibration 4.12.  steps taken to
eliminate stray light 4.13.  evaluation with the
JRC calibration equipment
1.    Location 1.1.   the longitude, latitude and
altitude of site 1.2.  the source and accuracy
of these coordinates 1.3.  a description of the
local environment 1.4. availability accuracy
of a horizon/alt map 1.5.  a description of
nearby and tall objects 1.6.  site
representative to local environment 1.7.  the
local climate 1.8. photographs of horizon and
measuring site 2.    Operational
matters 2.1. site mission statement, 2.2. detai
ls of who is responsible for what 2.3. time
dedicated to measurements 2.4.  time spent on
maintenance etc. 2.5.  of operators working
week on measurements 2.6. the experience of
staff 2.7. how often staff change 2.8. the
training of new staff 2.9. details of how often
procedures have changed 2.10. description of
instrument characteristics 2.11. the records of
past intercomparisons 2.12 description of any
publications read/adopted 2.13.  photographs of
the operators
9
Presentation of results (1)
10
Presentation of results (2)
11
Formal statement (1)

• Brief summary of the audit
• Containing an overview (i.e., site objectives
  and practices), major findings, conclusions
• The institute will have the opportunity to
  comment on this statement, and to append further
  information or comments they consider
  appropriate, before it is circulated
• Blank sections left for future updates
• The formal statement will be in a format that can
  be used within the final report on the quality
  audits.

12
Formal statement (2)
Introduction This document is intended to
provide an informal summary of the quality audit
performed at RIVM over the course of the three
days Monday 19 to Wednesday 21 March 2001. This
audit was t he first of some one dozen site
visits that will be conducted over the next 18
months within the EDU CE project. It was planned
as a test audit, so that the audit procedures
and timetable could be considered and refined.
While I hope that the notes presented below will
prove useful to both RIV M and the EDUCE
coordinator, the trial nature of this first visit
and the incompleteness of the audit mean that
impressions and concl usions are necessarily
preliminary. Tim Martin Overview This document
is intended to provide an informal summary of the
quality audit performed at RIVM over the course
of the three days Monday 19 to Wednesday 21 March
2001. This audit was t he first of some one dozen
site visits that will be conducted over the next
18 months within the EDU CE project. It was
planned as a test audit, so that the audit
procedures and timetable could be considered and
refined. While I hope that the notes presented
below will prove useful to both RIV M and the
EDUCE coordinator, the trial nature of this first
visit and the incompleteness of the audit mean
that impressions and concl usions are necessarily
preliminary. In February, it was agreed that it
would not be possible to observe a full
calibration of a spectroradiometer during the
course of the first visit to RIVM. For this
reason, the sections of the audit rep ort and
summary relating to instrument calibration are
incomplete. A calibration is planned for the
return visit in O ctober 2002, when RIVM also
hope to be able to make use of the JRC travelling
calibration kit. Three full days were provided
for this visit. In future, only two days will be
allowed. I am most grateful to all at RIVM for
their time, hospitality and helpful advice during
the course of the visit. This document is sent
to Best practice This document is intended to
provide an informal summary of the quality audit
performed at RIVM over the course of the three
days Monday 19 to Wednesday 21 March 2001. This
audit was t he first of some one dozen site
visits that will be conducted over the next 18
months within the EDU CE project. It was planned
as a test audit, so that the audit procedures
and timetable could be considered and refined.
While I hope that the notes presented below will
prove useful to both RIV M and the EDUCE
coordinator, the trial nature of this first visit
and the incompleteness of the audit mean that
impressions and concl usions are necessarily
preliminary. In February, it was agreed that it
would not be possible to observe a full
calibration of a spectroradiometer during the
course of the first visit to RIVM. For this
reason, the sections of the audit rep ort and
summary relating to instrument calibration are
incomplete. A calibration is planned for the
return visit in O ctober 2002, when RIVM also
hope to be able to make use of the JRC travelling
calibration kit. Three full days were provided
for this visit. In future, only two days will be
allowed. Room for improvement This document is
intended to provide an informal summary of the
quality audit performed at RIVM over the course
of the three days Monday 19 to Wednesday 21 March
2001. This audit was t he first of some one dozen
site visits that will be conducted over the next
18 months within the EDU CE project. It was
planned as a test audit, so that the audit
procedures and timetable could be considered and
refined. While I hope that the notes presented
below will prove useful to both RIV M and the
EDUCE coordinator, the trial nature of this first
visit and the incompleteness of the audit mean
that impressions and concl I am most grateful
to all at RIVM for their time, hospitality and
helpful advice during the course of the
visit.This document is sent to Conclusions This
document is intended to provide an informal
summary of the quality audit performed at RIVM
over the course of the three days Monday 19 to
Wednesday 21 March 2001. This audit was the
first of some one dozen site visits that will be
conducted over the next 18 months within the EDU
CE project. It was planned as a test audit, so
that the audit procedures and timetabl e could be
considered and refined. While I hope that the
notes presented below will prove useful to both
RIV EDUCE coordinator, the trial nature of this
first visit and the incompleteness of the audit
mean that impressions and conclusions are
necessarily preliminary. In February, it was
agreed that it would not be possible to observe a
full calibration of ort and summary relating to
instrument calibration are incomplete. A
calibration is planned for the return visit in
O ctober 2002, when RIVM also hope to be able to
make use of the JRC travelling calibration
kit. Three full days were provided for this
visit. In future, only two days will be allowed.
Quality Audit at Institute April 2002
13
Final report

• The report will be circulated to all EDUCE PIs
• The report will be made available at the database
  website (accessible only to registered users).
• Some non-contentious information about the visit
  will also be placed at the public pages of the
  EDUCE web site.