Partnering with an Evaluator

1

• Partnering with an Evaluator
• Presented by Joanne Kahn

2
Session goals

• Understand how to hire and partner with a
  consultant to assist you with evaluation.
• Set the stage for an effective evaluation.

3
Steps in the partnership

• 1 - Project definition
• 2- Vendor selection
• 3- Contract negotiation
• 4- Data collection
• 5- Reporting

4
Project Definition what to include?

• Things you could do in-house
• Library use statistics
• Point-of-use inquiry by staff
• Skills tests
• Feedback forms/surveys (but could get design
  help)
• Outside firm helpful for
• Survey design
• Focus groups
• Interviews
• Observation
• Instructor assessments

5
Steps in the partnership
Project definition Vendor selection Contrac
t negotiation Data collection
Reporting
6
Vendor selection

• Does your library have an existing relationship
  with a particular research firm?
• To find an evaluator, ask peers or check the
  American Evaluators Association website
  www.eval.org.
• Members commit to
• Systematic inquiry
• Competence
• Integrity/honesty
• Respect for people
• Responsibilities for general and public welfare

7
Vendor selection

• Interview for
• Compatibility with your goals
• Flexibility
• Experience with methods you want to use
• Staff who speak/write in needed languages
• Schedule availability
• Willing to work within your budget constraints
• A portfolio of similar work
• References are they easy to work with?
• See handout article from Usable Knowledge.
• www.usablellc.net/White_Papers/Hiring20an20Evalu
  ation20Consultant.pdf

8
Points to discuss with the evaluator

• Timing and frequency
• Who will do each task?
• Which participants? How many?
• Protecting the confidentiality of participants
  personal information
• How will participants be recruited?
• Incentives and perhaps transportation for
  participants

9
Evaluators perspective

• Focus on the story you want to tell stakeholders,
  and collect the data you need to tell it.
• If you change staff (or delegate), make sure
  everyone knows the goals.
• Plan how you will handle unexpected or negative
  feedback. The evaluator will take the approach
  that all information is useful.

Special thanks to Juliette Mackin of NPC
Research.
10
Steps in the partnership
Project definition Vendor selection Contrac
t negotiation Data collection
Reporting
11
Contract negotiation

• Ask for a written proposal and fixed-price bid,
  and understand how the total was reached (hourly
  rates and line-item costs).
• If bid exceeds budget, look for ways to reduce
  costs
• Junior staff members may be less expensive than
  partners
• You may be able to reduce their time by
• Arranging the venue, incentives, and refreshments
• Providing pre-screened participants with
  phone/email contact info
• Bundling this project with another

12
Contract negotiation

• Set a schedule that includes time for review and
  revisions.
• Agree up front
• What intermediate deliverables will be provided?
• Who needs to sign-off on the report?

13
Steps in the partnership
Project definition Vendor selection Contrac
t negotiation Data collection
Reporting
14
Data collection surveys

• You may draft questions for the evaluator to
  review and include.
• Have you made the process as participant-friendly
  as possible?
• Measure before/after results.
• Are the questions clear? For multiple choice
  questions, are the choices adequate?
• Are there open-ended questions?

15
Data collection focus groups

• How will you recruit/screen participants?
• Have you made the logistics as participant-friendl
  y as possible?
• The evaluator will probably want you out of the
  room.
• Review script for completeness, especially
  follow-on questions.
• Will there be a transcript or recording?

16
Steps in the partnership
Project definition Vendor selection Contrac
t negotiation Data collection
Reporting
17
Reporting

• Respect the firms objectivity!
• But its fair to ask them to have a positive
  outlook.

18
Reporting make it readable!

• Executive summary conveys the main points
• Outline is easy to follow
• Visually attractive photos!!
• Do tables/graphs improve the report?
• Are patron comments or anecdotes included?
• Patterns do data sources corroborate each
  other?
• Conclusions should follow from the data