Smart Investing your library Orientation

1
A partnership between the American Library
Association and FINRA Investor Education
Foundation
Orientation ALA Midwinter Meeting January 11, 2008
2
Workshop Goals

• Create a common understanding and increased
  knowledge of outcome-based planning and
  evaluation so grantees may refine and finalize
  Smart Investing evaluation and marketing plans.
• Set the stage for effective program
  implementation.

3
Overview Agenda

• Why Measure Outcomes?
• What is Outcome-Based Evaluation?
• How to Measure Outcomes
• How to Collect Data
• How to Use Data
• Closing Thoughts and Next Steps

4
Why Measure Outcomes?

• See if programs really make a difference in the
  lives of people.
• Improve programs.
• Improve planning.
• Increase accountability.
• Ensure best use of funds.
• Compare programs consistently.
• Demonstrate impact.
• Funders demand it.

5
Why Measure Outcomes?

• "Outcome measurement has been a tremendous
  benefit to our agency by clarifying the services
  we provide. It has helped the staff, clients,
  board, constituents, and funders. We have a whole
  list of topics to talk about and we are all on
  the same page. This is concrete data. We can
  compare success versus instances where there was
  not success and draw parallels."
• -- Nina Waters, Executive Director,Practical and
  Cultural Education Center for Girls,Jacksonville,
  Florida

6
What Is Outcome-Based Evaluation?

• Outcome-based evaluation is a systematic way to
  assess how well a program has achieved its
  intended results.
• How has my program made a difference?
• How are the skills of the program participants
  improved as a result of my program?

7
What Is Outcome-Based Evaluation?
8
Get Ready

• Audience is key
• Users
• Staff
• Program partners
• Key terms
• Situation
• Inputs
• Outputs
• Activities
• Outcomes

9
Get Ready
10
Outputs/Outcomes

• Three programs held
• Web site developed
• Print, electronic collection increased 10
• 100 PSAs run

• 50 people from target audience attend each
  report they would recommend the program
• Increased use of investment Web site from
  beginning to end of the program.
• Increased use of investment collection.
• 10 of participants report PSA motivated
  attendance

11
Get Ready

• Smart Investing program elements
• Staff training
• Web presence
• Classes/programs
• Marketing/outreach
• Partnerships

12
Choose the Outcomes You Want to Measure

• Smart Investing overall outcomes
• Community members will view the library as a
  reliable place for unbiased investor information
• Community members will make increased use of
  library programs and resources
• Community members will be more knowledgeable
  about key investment issues.

13
Choose the Outcomes You Want to Measure

• Smart Investing outcomes at the library level
• Increased requests for investment materials
• Increased visits to library Web site
• Increased awareness of library resources
• Increased staff competencies

14
Choose the Outcomes You Want to Measure

• Smart Investing outcomes at the library level
• Users demonstrate increased skills and/or
  knowledge
• Users take action with new skills (i.e. create a
  personal budget)
• Program partners report a positive experience
  working with the library
• PR/Marketing activities boost participation

15
Specify indicators for your outcomes

• Smart Investing indicator examples
• Program attendance
• Increased circulation of related materials
• Web hits
• Survey feedback
• New investing club established
• Increased use of Earned Income Tax Credit

16
Specify indicators for your outcomes

• Measurable That is, you can test for the change
  or observe it. But if you made a movie of
  success, the camera would focus on people, not on
  mechanisms or processes used to create the
  hoped-for results.
• Changes in participants Remember weve defined
  an outcome as a change in a target audiences
  skills, attitudes, knowledge, behavior, status,
  or life condition brought about by experiencing a
  program.
• Define success  Does the outcome represent a
  benefit for the target audience? Do key
  stakeholders accept the outcome as valid for your
  program? Finally, is the outcome realistic and
  attainable?
• Participating in your program  Is it sensible to
  claim your program services influenced the
  outcome?

17
Marketing Example

• GOAL Develop an outreach plan to identify and
  communicate with community members who will
  benefit from Smart Investing _at_ your library.
• The marketing plan will identify community
  members in each targeted audience who will
  benefit most from the project.
• Community members will be well-informed of the
  spectrum of activities available throughout the
  project.

18
Partner Example

• GOAL Partnerships with community and regional
  organizations will be sustained.
• Requests for speaking engagements at local civic
  groups
• Attendance at programs offered during financial
  education annual events
• Resource sharing with library systems and
  organizations

19
Staff Example

• GOAL Reference staff will know sources of
  accurate and unbiased information on investing.
• By 12-1-08, staff testing will show an 80
  accuracy rate in referring patrons to accurate
  and unbiased information on investments.
• By the end of the grant period, staff will
  successfully refer 90 of inquiries for
  investment information to accurate and unbiased
  sources in multiple formats.

20
Prepare to collect data on your indicators

• Pre-test and Post-test
• You cant measure success without a baseline.
• What is the current state of affairs what do
  people know, perceive and do before the program
  and how does the program move the audience
  forward?
• Include all stakeholders

21
Prepare to collect data on your indicators

• Data source options
• Feedback forms/short surveys
• Point-of-use inquiry by staff
• Focus groups
• Interviews
• Skills tests
• Observation
• Instructor assessments
• Library use statistics

22
Prepare to collect data on your indicators

• Other considerations
• When will you collect data?
• Include all participants or a sample?
• Who will collect data?
• Who will record/compile data?
• How will confidentiality be protected?
• How will participants be informed about the data
  collection process?

23
Hands-on

• Take 10 minutes to review your goals and outcomes
  and make two to three improvements based on the
  data you gathered for todays workshop and what
  weve discussed this morning.

24
Try out Your Measurement System

• Pilot or beta test your surveys or
  questionnaires.
• Clarity of questions
• Ease of use
• Are you measuring what you intended to measure?
  Are you asking the most appropriate questions?
• REVISE instrument as needed!

25
Analyze and Report Findings

• Collect, input data at regular intervals
• Mid-project reporting
• Troubleshooting
• Revising outcomes as appropriate
• Its as much about the journey as the destination

26
Analyze and Report Findings

• Review feedback from participants
• Get very familiar with the data
• Look for and note oddities in reporting
• Peruse the data and identify patterns
• Substantiate patterns do data sources
  corroborate each other?

27
Analyze and Report Findings

• Organize data logically (tables/charts)
• Analyze and interpret data to develop narrative
  for final report
• Document findings
• Maintain files or database of outcomes and
  activities
• Determine outcomes you want to continue monitoring

28
Improve Your System

• More important to discontinue something that
  isnt working than to report back that you did
  everything you said you would do at start of the
  program when it failed.

29
Use Your Findings

• Tell your story!
• Marketing
• Accountability and long-term assessment
• Improved services and/or programs
• Resource (re)allocation

30
Marketing

• Outcomes can inform key messages about the
  library and what it does well.
• Outcomes allow stakeholders to understand, in
  users own words, the powerful role of the
  library.

31
Marketing

• Identify your audiences
• Funder
• Targeted media outlets
• Partners
• Match audiences and outcomes
• Determine dissemination strategies
• Annual report
• Press release
• Flyers and brochures

32
Closing

• Questions?
• One thing youve learned today
• One thing youre going to do as a result of what
  youve learned today
• Evaluation form

33
Additional resources

• IMLS National Leadership Grants tutorial
  http//www.imls.gov/project_planning/index1.asp
• Texas State Library resources http//www.tsl.stat
  e.tx.us/outcomes/resources.html
• Florida State Library workbook
  http//dlis.dos.state.fl.us/bld/Research_Office/Ou
  tcomeEvalWkbk.doc

34
ALA Office for Research Statistics

• Denise Davis, Director
• 312-280-4273
• Larra Clark, Project Manager
• 312-280-2129