Principles of Good Data Collection

1
Principles of Good Data Collection

• Region V ERP Auto Body Training
• Chicago, IL
• November 18, 2009

2
Goals of Presentation

• Develop understanding of quality issues connected
  with
• Formulating measures
• Designing questionnaire
• Collecting data
• Processing data

3
Step 1 Selection of Indicators

• Purpose of Indicators
• Types of Indicators
• How to Formulate

4
Purpose of Indicators

• Provide shorthand understanding of the
  performance of a facility (or group) at a point
  in time.

5
Different Types of Indicators

• Activity-based measures facility performance,
  (e.g., minimization plan written and posted where
  paint stripping is conducted?)
• Outcome-based measures quantitative
  environmental outcomes by estimating reductions
  in air emissions, hazardous waste, and materials
  use, (e.g., measure reduction in VOC emissions
  associated with use of low-VOC/water based
  solvents for all auto body shops.)
• Regulatory required practices, (e.g., filter
  capture efficiency rating of booth exhaust/filter
  system at least 98 percent)
• Beyond-compliance best management practices,
  (e.g., installed specialized controls (timers,
  motion sensors) that turn off or throttle back
  lights, heat, or equipment when areas are not
  occupied and/or in use)

6
Different Ways of Formulating Indicators

• Quantitative v. Y/N questions
• Single-item questions v. rolled-up questions

7
Example of Rolled-Up Indicators

• Has your shop implemented all the required
  management practices that minimize emissions of
  MeCl use, (evaluating each application for
  alternative to MeCl reducing exposure of MeCl
  strippers to air optimizing conditions when
  using MeCl to reduce evaporation and using
  proper storage and disposal techniques)?

8
Step 2 Questionnaire Design

• Questionnaire Design Tips
• Questionnaire Testing

9
Design Tip Avoid Double Negatives

• Unclear Does the facility avoid using unapproved
  spray booths/stations to coat miscellaneous parts
  or products or vehicle subassemblies?
• Explicit Does the facility only use approved
  spray booths/stations that have a full roof, at
  least three complete walls or side curtains, and
  is ventilated so that air is drawn into the
  booth, to coat miscellaneous parts or products or
  vehicle subassemblies?

10
Design Tip Definite Time Frames

• Unclear Does the facility maintain records of
  MeCl use for paint stripping?
• Explicit Does the facility have records of
  annual usage of MeCl for paint stripping?

11
Design Tip Consider Measurability

• Are employees aware of P2 practices?
  difficult to observe
• Do employees receive documented P2 training at
  least once a year? better
• "Are filters changed regularly to ensure good
  airflow? -- even better

12
Before Using Questionnaires in the Field

• Test them!
• This training is the last test before we go
  live
• Testing helps to ensure that questions are
  interpreted the same way (precision and
  comparability)
• Testing helps identify any errors or ambiguities
  (precision and completeness)

13
Step 3 Data Collection

• Who Collects the Data?
• Tips for Inspector Training
• Timing Considerations
• Electronic v. Paper

14
Who Is to Answer Questionnaire?

• SBEAP Field Observers
• EPA Inspectors
• Facilities (self-reporting)
• Each option can be evaluated in terms of quality

15
Who Is to Answer Questionnaire?

• SBEAP Field Observers
• May have less experience conducting inspections
• Has no authority to enter facilities (will have
  to request entry)
• Follow-up action in the form of assistance

16
Who Is to Answer Questionnaire?

• EPA Inspectors
• May have more experience conducting inspections
• Has authority to enter facilities
• Can take follow-up action in the form of informal
  or formal enforcement

17
Who Is to Answer Questionnaire?

• Facility self-reporting
• Raises bias concerns
• Training less reliable
• Note In ERPs in various sectors (dry cleaning,
  printing, farming, etc) around the country,
  verification inspections by regulators show that
  self-reporting starts at reasonable accuracy
  rates (70) and improves in successive
  self-reporting cycles.

18
Musts for Data Collection

• Train field observers so that they...
• Interpret questions consistently
• Understand and follow sampling protocol
• Answer all questions (no blanks)
• Always use ink
• Use consistent protocol for corrections
• e.g., double-strikeout plus initials (will
  discuss protocol in Planning for Site Visits
  presentation)

19
More on Consistent Interpretation

• Interpretation by field observer
• e.g., are observers looking for faults and
  violations, or trying to gather a holistic
  impression of facility performance, or both?
• Training observers together can help align
  approach
• But there will be differences and potential
  biases
• e.g., more violations in one state might mean
  simply that those observers had a more strict
  interpretation of compliance.
• Best strategy is to make indicators as specific
  and explicit as possible.

20
Timeframe of Data Collection

• Short timeframe desirable, E.g., 2-3 months
• Consider seasonality issues
• VOC usage may not be same peak for each shop
• Not all states, or regions of all states, are in
  same snow belt. May impact prime season for
  collisions
• Reminder Project goal to complete baseline
  inspections by first quarter 2010 (Jan 1st March
  31st).

21
Step 4 Data Processing

• Data Entry Options
• Certification of Data Quality

22
Data Entry Options for Checklists

• Transfer data from checklist into electronic
  format, i.e. using on-line survey developed by WI
  SBEAP Project Lead.
• Scan checklists and send electronic copy (PDF) to
  WI SBEAP Project Lead.
• Mail hard copies of completed checklists to WI
  SBEAP Project Lead for data entry.
• Some SBEAPs may not want to keep records of
  shop-specific data in order to preserve their
  confidentiality and assistance stance. However,
  to protect against lost mail, make copies of
  checklists and destroy after they are
  electronically filed by WI SBEAP Project Lead.

23
Data Entry Tip

• If field observers transfer checklists into
  electronic survey, do so as soon as possible
  after inspection

24
Certification of Data Quality

• Each state lead will be responsible for the
  completeness of their states data.
• Copies of completed checklists will be sent to
  Project Lead/Quality Assurance Officer (WI SBEAP)
  along with a Certification of Data Quality
  signed by each state project.
• Certification of Data Quality Statement will be
  provided to each state lead prior to data
  collection.
• Please send completed checklists and
  certification statement all at once to minimize
  error.
• Statement will look something like this
• I, ..(project lead), from the state of
  .., certify that the enclosed field
  observation checklists meet the data quality
  standards described in the Region V ERP Auto Body
  Training Workshop on November 18-19 2009.

25
For more information

• Tara Acker, NEWMOA
• taraacker_at_gmail.com
• [phone]
• or
• Renee Bashel, WI Department of Commerce
• renee.bashel_at_wisconsin.gov
• [phone]