Dr. Kimberly KempfLeonard

1
Methods Sources of Data How, Who What
outcome measures subgrantees should try to know
provide

• Presented by
• Dr. Kimberly Kempf-Leonard
• School of Social Sciences
• University of Texas at Dallas
• kleonard_at_utdallas.edu

Performance Measurement Training
2
Why are we doing this?

• Monitoring Assessment
• Improve our juvenile justice system
• Convince other systems they can benefit from what
  weve done
• Describe Explain Evaluate Predict

3
How do you know .something (lets call it X)..?

• X is common sense or obvious
• X is a generally agreed upon belief
• X is based on tradition
• Experts told me X
• I learned X through my own experience
• I observed X myself

• Could you be wrong?
• Inaccurate observations
• Overgeneralization
• Selective Observations
• Illogical reasoning
• Resist change
• Ego-based commitments
• too devoted to tradition
• Uncritical agreement with authority

4
Whats it mean to know somethings true or real?

• Is it real in eye of beholder?
• Subjective reality
• Is it the same reality for everyone?
• Objective reality
• Goal Real Information on Outcomes
• distinguish objective subjective

5
(No Transcript)
6
We all know how to lose weight

• 1980-90s Hypothesis Weight loss a longer life
  are both more likely when you eat a low fat diet.
• 
  FAT
  CAUSES FAT
• Fat is bad for you it makes sense
• pasta, bagels, bran muffins
• U.S. Food Pyramid
• 2000s Hypothesis Weight loss and a longer life
  are both more likely when you eat a diet low in
  carbohydrates.
• 
  CARBS CAUSE FAT
• Atkins diet Red Meat
• Endocrinology 101 theory carbs
  insulin/blood sugarmetabolize fatappetite
• Epidemic of obesity and Type II diabetes
  clinical trials low fat diet fails
• 2005 Hypothesis There isnt just 1 right
  answer
• Move More Eat Less X, Y X
  CAUSE FAT

7
Time Data Collection Design

• Cross-sectional designs
• Data are collected for a single point in time
• Quickest
• Only correlations, cant assess cause/effect
• Longitudinal designs
• Add a time dimension with at least 2 data
  collection periods (6-12 months follow-up)
• Case Study provides rich details
  illustration Difficult to generalize to other
  cases

8
Appropriate Data Collection METHODS?Ask...Watch
ExperienceUse whats there

• Archival records
• Documents, Information Systems, Content
• Field Observations
• Personal Experiences
• Surveys
• Questionnaires, Interviews, Focus Groups

9
For which subjects (youths, programs, services)
should they get data?

• Good answers
• The population
• 2) A sample that represents the population well

10
(No Transcript)
11
We can only get ..Is that enough?

• How the subgroup came to exist (i.e. Selection)
  is critical information
• Front end select to represent population (No
  CREAMING)
• Back end minimal attrition bias (No MORTALITY
  problem)
• lottery, equal chance of being in subgroup
  (probability sampling)
• Size or number of subjects isnt nearly as
  important
• Larger samples more breadth less depth
• Want more if targeted behavior or outcome is
  infrequent

12
Key know traits of subjects present absent
13
What sort of information is recorded? (variables)

• Output measures of Program Activities
• hours of training
• of youth served
• Average days in program services
• of services implemented
• Average days from intake to disposition

• Outcome measures
• (Short- Long-term)
• of youth who reoffend
• of youth who complete program requirements
• of youth with improved targeted behaviors

14
(No Transcript)
15
Assessing variables

• Characteristics
• Multiple categories
• Mutually exclusive
• options are exhausted
• Seldom perfect measures
• Fail to get true meaning equate measure proxy
• Random mistakes
• Systematic mistakes
• Multiple measures improve confidence
• 2 eyes are better than 1

16
A variable should try to hit
17
What do we know about unemployment rate.

• U.S. unemployment rate is measured by dividing
  the number of unemployed individuals by the
  number of people in the civilian labor force
  multiplying by 100
• Civilian Labor force exclusions military,
  inmates,
  retirees, disabled, voluntarily idle.
• Unemployed individuals include working at paid
  job (inc. part-time), unpaid work at family
  business, absent from work (paid unpaid) due to
  labor dispute, bad weather, vacations, personal
  reason

18
more confident that crime declined
19

• What happened to the exit polls in the 2004
  Presidential election?
• What does it mean for a youth to re-offend or
  recidivate?
• What does it mean for youth to participate in the
  program?
• Is there any type of youth not included? Does
  that make a difference?
• What does if mean if theres a handwritten A in
  the blank for race?

20
Survey Methods

• Standardized, systematic asking
• Uniform stimulus so responses can be compared
• Efficient data collection
• Price, Speed, Quality Pick Two
• Ideal for characteristics of a population
• Individuals attitudes, values, behaviors,
  experiences, opinions, knowledge, circumstances
• Organizations/Institutions culture, policies,
  finances

21
The heart of surveys is the questionnaire
instrument
Wake up Pay attention theres a quiz coming!
22
Anticipate how they might answer

• Interpretation of the Question
• Retrieve from memory
• Form judgement
• Edit response for social desirability (strategic)
• Trying out (pretesting) survey questions helps
  avoid problems

23
Need to be Clear

• Avoid extreme loaded positions
• Should State spend more money on youth services?
• Should State spend more money to help serious
  juvenile offenders?
• Limit to a single topic (Double-barreled
  Conditional phrases)
• Do you think your outlook and performance
  benefited from the curriculum and the teachers?
• Avoid ambiguity
• What is the number of youths living in the
  facility most of the time while working, even if
  they have another place to live?
• Complexity creates confusion
• Please choose one or more categories that best
  indicate your race.
• Avoid Presupposition (varied word tenses, forced
  choices)
• What are your usual practices for studying?
• Have you stopped attending a program?
• Are you truant or delinquent?

24
Confidence that we Know!Quality Control
Standards.. because we dont use the same
vocabulary, have the same experiences, or
interpret the same way.

• Precision Does it achieve the level of
  generality intended?
• Validity Is it accurate?
• Does it capture reality of
  whats intended?
• Reliability Do repeated tries yield the same
  result?
• Assess comparisons cross-checks
• Do both eyes see it the same way?
• Use established measures
• data collection instruments
• cross-check data collectors
• test-retest method (2 different times)
• split-half method (1/2 one kind ½ another)

25
Final advice Dont worry!

• a lot of examples exist already that can be
  easily adapted
• There are a lot of college professors and
  graduate students in every state just itching to
  help you (and we work cheap)