Instruments and Data Collection Strategies

1
Instruments and Data Collection Strategies

• Florida AmeriCorps Training
• March 3, 2006

2
Agenda

• Purposes of Performance Measurement?
• Where do I stand?
• Key considerations in selecting/designing
  instruments.
• Improving practices for high quality data
  collection.

3
Purpose of Performance Measurement

• Documents changes that addresses a Community
  Need.
• Conveys information about the change to
  stakeholders (interested parties).
• What else?

4
Performance Measurement Steps
ContinuousProgramImprovement
5
Exercise

• Where Do I Stand?

6
Group Discussion

• What instruments are you using?
• What other methods could you use?
• What are some of your instrument challenges?
• How did you obtain the instrument(s) youre using
  to collect data?

7
Key considerations in selecting/designing
instruments

• Instrument alignment
• Appropriate data source
• Manageable
• Timeliness of data collection/availability
• What else?

8
Instrument Considerations

• Connect to PMW--Measure the target.
• Aim for the appropriate method.
• Measure correct population (data source).
• Make it informative and manageable.
• How many questions do you really need?
• Balance best manageable method.
• Be sure members understand instrument.
• Ensure site/respondent has instrument.

9
Tips on Measuring Targets

• Determine early who can give you the needed
  information (source).
• Clearly define questions to answer target-survey,
  interview, checklist, etc.
• Plan how to collect data (instrument)
• Pilot the instrumentas best you can!
• Develop a PM Plan!

10
Improving Practices for High Quality Data
Collection

• Data Collection Definition
• Data Collection Issues
• Data Sources
• Data collection methods
• Developing a Plan
• Streamlining Data Collection
• Data Collection Tips

11
Data Collection Definition

• A systematic process of measuring performance by
  documenting indicators.
• Involves gathering information needed to
  determine if you met your targets.
• Well planned data collection strategies are
  critical to obtaining reliable, consistent,
  useful information about the effect of your
  program.

12
Developing High Quality Data

• Starts with clear performance measure output and
  outcome statements
• Select appropriate data collection methods and
  instruments
• Maintain accuracy and integrity throughout the
  data collection, analyses, and reporting processes

13
Data Collection Methods

• Logs
• Surveyopinion
• Questionnaire (Checklist, Scale)
• Test / Experimental Designs
• Observation (Observation Guide)
• Focus Group (Predetermined Script/Guide)
• Interview (Interview Guide)
• Case Study (Measurable change over time)
• Secondary Data (Existing data-reliable!)

14
Data Collection Issues

• Plan for success
• Make sure that your data sources (e.g.,
  beneficiaries, organizations) are willing and
  available.
• Choose appropriate methods.
• Develop efficient data collection plan/process.
• Develop feasible timelines.
• Prepare your staff.
• Train members/data collectors.
• Increase incentives/accountability.

15
Data Source

• Data Source addresses the question, Who will
  provide the information? and/or From where will
  the data come?
• A data source may come from another agency or
  from within your project, such as
• Members
• Volunteers
• Agency records
• Program staff
• Community beneficiaries

16
Data From Other Agencies

• Many partners have data available other agencies
  may agree to share data with you.
• Determine if data are useful to you
• Access Will you have permission to access data?
• Format Will data be in a useable format?
• Timing Will data be available when you need
  them?
• Aggregation Will you be able to transfer data to
  your own records?

17
Project-Generated Data

• Data collected within your project and by your
  staff.
• In-house data provides more control over data
  collection and more data analysis capabilities.
• Determine cost/benefit
• Access Will respondents and staff be willing?
• Format Does your program have the capacity to
  develop a database/records?
• Timing Will data/sources be available when you
  need them?
• Aggregation Will you be able to analyze your
  data to answer questions?

18
Choosing Data Collection Methods

• Instrument Form used to record data measure
  indicators.
• Method Set of procedures for collecting data.
• Consider procedures
• When will data/sources be available?
• When should you administer instrument?
• How will you administer instrument?
• Who will administer instrument?
• Who will complete instrument?

19
Possible Data Collection Methods

• Archival review Copying information from
  existing paper or electronic records.
• Questionnaires Distributing surveys by hand,
  postal mail, or Internet.
• Interviews Conducting face-to-face or telephone
  surveys.
• Focus groups Leading group discussions.
• Observation Recording actions using checklist
  and notes.

20
Methods Considerations

• Be aware of each method's advantages and
  challenges.
• Resources
• Time
• Cost
• Intrusiveness/respondent burden
• Cultural competence
• Pilot test data collection process.

21
Methods Advantages Challenges

• Example Questionnaires (Distributing surveys
  by hand, postal mail, or Internet.)
• Advantages
• Can complete anonymously.
• Inexpensive to administer.
• Easy to compare and analyze.
• Administer to many people.
• Can gets a lot of data.
• Many sample questionnaires already exist.
• Disadvantages
• Might not get careful feedback
• Wording can bias clients responses
• They are impersonal
• Doesnt get full story

22
Make a Plan!
23
Performance Measurement Plan

• Performance Measurement Plan (who and when)
• Identify individuals and schedules for
  administration of instruments (pre-, post,
  follow-up) with feasible target dates.
• Indicate who/how you will train staff/members.
• Identify/develop materials you might need (e.g.
  instruction guides, translations).
• Specify who will be responsible for data cleaning
  and entry processes.
• Develop procedures for aggregating and
  summarizing results.

24
Aggregate Summarize Results
Common Problems
Possible Solutions

• Not knowing who will aggregate.
• Not planning enough time to compile and analyze
  instruments.
• Not having summary tools to compile data.

• Specify a point person to which all data will
  flow.
• Think about where you need to be in a year, and
  work backward to set deadlines.
• Search for available resources for building
  database capacity.

25
Expedite Streamline Data Collection
Use available technology when accessible

• Existing MIS data
• Computerized participant monitoring
• On-line interviewing/surveys
• Web-based data management

26
Expedite Streamline Data Collection cont
Provide incentives to

• Program Participants
• Program Staff
• Data Collectors
• Other Data Providers

27
Top 11 Data Collection Tips
11. Pilot your process 10. Set a schedule 9. Use
low burden methods 8. Encourage responsiveness
7. Provide confidentiality
28
Top 11 Data Collection Tips cont
6. Identify appropriate data sources 5. Identify
appropriate methods 4. Identify your
helpers 3. Train your helpers 2. Start with
success
1. Get early buy-in
29
Communication Builds Trust and Cooperation
30
Additional Resources

• Contact your State Commission
• Contact Project STAR
• www.nationalserviceresources.org//sites/star/
• 1-800-548-3656
• star_at_aiweb.com