Repeatability and Reproducibility Analysis of the Round Robin Results

1

• Repeatability and Reproducibility Analysis of
  the Round Robin Results

2
Round Robin Objectives

• The familiarization of the participating
  laboratories within the project with the specific
  sampling system use (particulates system)
• The examination of the consistency of the results
  obtained when applying this methodology in
  different physical environments

3
Participating Laboratories

• VTT Energy (VTT) Finland
• MTC Sweden
• IFP France
• EMPA Switzerland
• Technical University Graz (TUG) Austria
• AEA Technology (AEAT) UK
• SHELL UK
• Lab of Applied Thermodynamics (LAT) - Greece

4
Round robin execution table
Laboratory Arrival Day Departure Day
VTT 11/09/2001 24/09/2001
MTC 28/09/2001 05/10/2001
IFP 12/10/2001 22/10/2001
EMPA 02/11/2001 09/11/2001
TUG 09/11/2001 20/11/2001
AEAT 29/11/2001 07/12/2001
Shell 08/12/2001 15/12/2002
LAT 16/12/2002 29/01/2002
5
Test Vehicle

• VW Golf TDI 1.9 l, M Y 1996 with the oxidation
  catalyst removed
• Fuel Fulfilling the specification EN590 (Year
  2000 specs)
• Lubricant BP Vanellus C4 multi SAE 20W 50
  grade (pre aged for 874 km)

6
Instrumentation

• Primary Porous Diluter
• TYLAN MFC 2920 dilution air mass flow controller
• Ageing Chamber
• Thermodenuder
• Dekatis Electrical Low Pressure Impactor (ELPI)
  ELPI Filter Stage
• Vacuum Pump
• 2 Ejector dilutors
• Dekatis Diffusion Charger (DDC)
• TSIs 3022 Condensation Particle Counter (CPC)
• Dekatis Gravimetric Cascade Impactor (DGI)

7
Testing

• Transient Cycles
• NEDC
• Artemis Road
• Steady State Modes
• 50 km/h
• 120 km/h
• (road loads)

8
Properties measured in round robin

• Particulates System
• Mass emission rate (mg/km) DGI
• Active surface (cm2/cm3) DDC
• Number Concentration (cm-3) ELPI stages 1 to 8
• Number Concentration (cm-3) ELPI end filter
  stage
• Mean aerodynamic diameter (nm)
• Number concentration (cm-3) CPC
• Total number concentration (cm-3) mean median
  mobility diameters SMPS
• Regulated emissions
• NOx
• CO2
• CO
• HC
• PM

9
Repeatability Reproducibility Analysis proposed
in ASTM E691-99 Standard

• Inspection of the repeatability and
  reproducibility of the results obtained from each
  laboratory.
• Derivation of a precision statement in terms of
  the total within laboratory and between
  laboratories variances.

10
Factors affecting the outcome of a test method

• The equipment used
• The calibration of the equipment
• The operators using the equipment
• The environmental conditions

11
Precision and Accuracy
Accurate but imprecise results
True Value
Precise but inaccurate results
12
Within Lab and between laboratories variability

• Within Laboratory Variability (s2).
• Between Laboratories Variability (st2).
• Within Laboratory Between Laboratories
  Variances reflect
• The precision of the test method
• Any vagueness in the protocol
• Departures from the protocol
• Coefficient of Variance s/µ

13
Repeatable Results
Average value
14
Reproducible Results
Overall mean value
15
Repeatability Reproducibility Statistics and
their critical values

• Repeatability Index k
• x laboratory
• Reproducibility Index h
• Critical h and k values.

16
h plots by laboratories(reproducibility)
17
h plot by driving cycles
18
k plots by laboratories(repeatability)
19
k plots by driving cycles
20
Repeatability problems in regulated emission
results
VTT IFP EMPA
CO2 NEDC cold
CO
HC NEDC cold 50 km/h Artemis NEDC hot
NOx NEDC cold Artemis
PM Artemis NEDC hot
21
Reproducibility problems in regulated emission
results
EMPA SHELL VTT IFP AEAT
CO2 50 km/h All cycles
CO 50 km/h
HC NEDC hot NEDC cold
NOx Artemis NEDC cold
PM All cycles
22
Within Laboratories Coefficients Of Variance for
Regulated Emissions Results
23
Between Laboratories Coefficients Of Variance for
Regulated Emissions Results
24
Repeatability Problems in Particulates
instruments results
MTC IFP EMPA AEAT SHELL TUG LAT
CPC 120 km/h 50 km/h NEDC hot
DDC NEDC cold 50 120 km/h
ELPI 1-8 Artemis NEDC cold 50 120 km/h NEDC cold
ELPI filter Artemis NEDC
ELPI mean dp 120 km/h
SMPS total conc. 120 km/h 50 120 km/h
25
Reproducibility Problems in Particulates
instruments results
MTC IFP EMPA AEAT TUG SHELL
CPC 120 km/h 50 km/h NEDC hot NEDC cold
DDC Artemis NEDC 120 km/h
ELPI Filter Artemis NEDC
ELPI 1-8 Artemis NEDC hot 120 km/h NEDC cold
ELPI mean dp 50 km/h NEDC cold
DGI total mass NEDC Hot Artemis
26
Within laboratories Coefficients Of Variance for
Particle properties
27
Between Laboratories Coefficients of Variance for
particle properties
28
Comparison of the Within Laboratory Coefficients
of Variance of the particulates instruments
results with the regulated PM ones
29
Comparison of the Between Laboratory Coefficients
of Variance of the particulates instruments
results with the regulated PM ones
30
Conclusions - Recomendations

• Great variability observed in ELPI results
  (possibly thermodenuder?).
• DDC results were the most repeatable and
  reproducible.
• Possible dilution ratio determination problems.
• Indications that the method is robust.
• Importance of conducting another round robin now
  that all laboratories are familiar with the
  protocol.