Interpreting Correlations

1
Program Evaluation Designs
2
Programs for Class Discussion

• Counseling program for prison inmates
• Stress reduction training for employees
• Study skills for at-risk college students
• Parental training for problem children

3
Why Should I Care About Research?

• Answering questions and making decisions
• We encounter research every day
• Common sense is often wrong

4
While Youre on Vacation, Burglars Go to
Work According to FBI statistics, over 26 of
home burglaries take place between Memorial Day
and Labor Day. To help you protect your home, we
are offering a special summer sale price for
installation of a home security system. Home
Safety Experts Route 11 and Wayland Road 555-6676
5
How Do I Know What to Research?

• Ideas
• Hypotheses - well thought-out suggestions or
  ideas
• Theories - systematic sets of assumptions
  regarding the nature and cause of particular
  events

6
Idea Hypothesis Theory
It seems that people dont work as hard in a group as when they are alone When pulling on a rope, a person working by himself will exert more force than a person working in a group Sucker effect Free-rider Individual effort will not be noticed
Social Loafing Example
7
Idea Hypothesis Theory
It seems that poor people are more violent than rich people There will be a correlation between income and the number of times arrested for being violent 1. 2. 3. 4. 5.
Aggression Example
8
How Do I Find Previous Research?

• Written sources
• Journals
• Trade Magazines
• Magazines
• Books
• Electronic Resources
• First Search
• Psych Info
• InfoTrac
• The Web

9
Where Will I Conduct Research?

• Locations
• Laboratory
• Field
• Office
• Issues
• External validity (generalizability)
• Control

10
What Research Method Should I Use?Independent
and Dependent Variables

• Independent Variable
• Experimental group
• Control group
• Dependent Variable

11
A researcher thinks that smaller groups will be
more cohesive than larger groups Independent
variable Group size Dependent variable Level
of cohesion
Number of Group Members
3 5 7 9 11 13
Cohesiveness rating 87 77 65 60 60 58
12
A researcher thinks that asking for a large
request and then a smaller request will increase
the chance of a person agreeing to the smaller
request Independent variable Size of initial
request Dependent variable agreeing to small
request
Experimental Condition
Small request only Large request first
agreeing 18 79
13
Identify the Independent and Dependent Variables

• Are people from large families more likely to
  commit crimes than people from small families?
  (A)
• Will taking an SAT prep class increase scores on
  the SAT? (B)
• Will taking Vitamin A decrease night blindness?
  (c)

14
Identify the Independent and Dependent Variables

• A researcher found that a higher percentage of
  serial killers have head injuries than do
  nonviolent criminals (d)
• A researcher found that people who set goals
  perform better at work than people who do not set
  goals (e)
• A researcher found that the amount of time that
  passes between viewing a crime and reporting a
  crime influences the number of errors made by
  eyewitnesses (f)

15
Evaluation of Program ResultsCriteria

• Content validity
• Participant reactions
• Participant learning
• Behavioral change
• Bottom-line measures

16
Dependent Measures in Program Evaluation

• Did people know about the program?
• Did people participate?
• Did they stay in the program?

17
Dependent Measures in Program Evaluation

• Did they change?
• Reality versus perception
• Measures
• Attitudes
• Behavior
• Skills
• Health
• How much of a change?
• Any change
• Enough to meet some criterion

18
Dependent Measures in Program Evaluation

• What did they think about the program
• Process
• Effectiveness

19
What Research Method Should I Use?Research
Methods

• Experiment
• independent variable is manipulated and
• Subjects are randomly assigned to conditions
• dependent variable
• Quasi-experiment
• Independent variable is not manipulated or
• Subjects are not randomly assigned to conditions
• Case study
• Survey
• Interview
• Natural observation

20
Evaluation of Program ResultsSingle-Group,
Nonexperimental Designs
Program
Posttest
21
Evaluation of Program ResultsSingle-Group,
Nonexperimental Designs
Pretest
Program
Posttest
22
Threats to Internal Validity

• Changes Unrelated to the Program
• Maturation
• History
• Participant Issues
• Selection
• Attrition
• Regression toward the mean
• Data Collection Issues
• Testing
• Instrumentation

23
Construct Validity in Pretest-Posttest Designs

• Of concern when self-report measures are used
• Types of self-reported change
• Alpha (real change)
• Beta (respondents change their understanding of
  the scale)
• Gamma (respondents change their understanding of
  the variable being measured)
• Examples
• Asking students about school safety
• Asking couples about their marital satisfaction

24
ExampleNew Child Care Center

• 2000 Employee absenteeism rate 5.09
• 2001 On-site child-care center established
  (Jan 1)
• 2001 Employee absenteeism rate 3.13

25
Date Absenteeism External Factor Internal Factor
1/00 2.8
2/00 3.1
3/00 4.7 Unemployment rate at 4.1
4/00 4.7
5/00 4.8
6/00 6.7 Main highway closed
7/00 6.5
8/00 4.9 Highway reopens
9/00 4.5
10/00 4.4
11/00 8.7 Terrible snowstorm
12/00 5.3
2000 Total 5.09
26
Date Absenteeism External Factor Internal Factor
1/01 5.3 Child care center started
2/01 5.2
3/01 5.1 Flextime program started
4/01 2.0 Unemployment rate at 9.3
5/01 2.0
6/01 2.0
7/01 1.8 Wellness program started
8/01 1.8
9/01 2.0 New attendance policy
10/01 2.1
11/01 4.0 Mild weather
12/01 4.2 Mild weather
2001 Total 3.13
27
Demonstrating Effects in Designs Without a
Control Group

• Means
• Show means for the periods before and after
  intervention
• Use a t-test to determine statistical
  significance
• Report d to show practical significance
• (Mafter Mbefore) SDoverall
• Time Series Analysis
• Looks at trends over time
• Interrupted time series involves an intervention
• Most powerful is ABAB design
• Most common is AB design

28
Child Care Center Example
Child Care Center Started
29
Child Care Center Example
Child Care Center Started
30
Child Care Center Example
Child Care Center Started
31
Conformity SuicidePhillips (1980)
32
Conformity and Commercial Airline
Fatalities(Phillips, 1983)
33
Time Series Charts

• Include actual plotted data
• Include Lowess lines
• Smoothed data
• Plotted point is the median of the 3 adjacent
  points
• Should include notations for other important
  events

34
Control Group Designs

• Control for many of the threats to internal
  validity
• Groups need to be as similar as possible
• Using multiple groups increases accuracy of
  interpretation
• Placebos are often used for the control group
• Control group treatment
• Never get treatment
• Get treatment later

35
Evaluation of Program ResultsControl Group
Designs
Pretest
Program
Posttest
Pretest
Posttest
36
Evaluation of Program ResultsSolomon four-groups
design
Program
Posttest
Group 1
Pretest
Program
Posttest
Group 2
Pretest
Posttest
Group 3
Posttest
Group 4
37
Assigning Subjects to Conditions

• Two Choices
• Independent groups
• Repeated measures
• Considerations
• Number of available subjects
• Order effects
• Will participating in one condition affect
  responses in another

38
Where Do I Get My Subjects?Who Will Participate?

• Size
• Students vs. real world

Does it Matter? If you were investigating whether
the type of evidence (e.g., eyewitness testimony,
DNA, fingerprints) influenced jurors decisions,
would students as subjects be different from
having people from the community?
39
Where Do I Get My Subjects?Sampling

• Types of Samples
• Random
• Representative
• Non-random/representative
• Sampling Methods
• Random selection
• Simple
• Block randomization
• Proportionate
• Convenience
• Random assignment

40
Randomized Block Designs

• Used when
• there are several conditions
• you want equal numbers in each condition
• You are concerned about time-related variables
• Example
• 4 conditions (A, B, C, D), 8 subjects per block
• Number of blocks the number of subjects desired
  for each condition
• Random blocks might be CBAD, DABC, BCDA, ACDB,
  CEBA, DBCA, BCAD, ADBC,

41
What Type of Sampling Method is Being Used?

• A researcher has the students in her classes fill
  out a questionnaire
• A researcher gives 6 to people who will
  participate in his study. As the people arrive,
  he flips a coin to see if they will be in the
  experimental or the control condition.
• A manager wants to see if a training program will
  increase performance. She selects every third
  name from the company roster to participate.
  Employees with an odd number at the end of their
  social security number are given one training
  program and those with an even number are given
  another.

42
Where Do I Get My Subjects?Inducements to
Participate

• Extra credit
• Money
• Intrinsic reasons
• Ordered to participate

Does it Matter? Would the inducement used affect
the type of person agreeing to participate? In
what ways?
43
Where Do I Get My Subjects?Informed Consent

• Ethically required
• Can be waived when
• Research involves minimal risk
• Waiver will not adversely affect rights of
  participants
• Research could not be done without the waiver

44
Is Informed Consent Needed?

• An experimenter wants to study the effects of
  electric shock on reducing patients depression
  levels
• A researcher wants to conduct a telephone survey
  in which she asks people their five favorite TV
  shows. She will then determine if males and
  females like different shows.
• A researcher wants to determine the types of
  people who litter. He plans to hide above a road
  and record information about the people who
  litter or dont litter (e.g., age, sex, type of
  car).

45
Running the Study

• Informed consent
• Instructions
• Task completion
• Deception?
• Debriefing

46
External Validity

• Can our findings be generalized to other
  situations?
• Considerations
• Sample
• How realistic was the task?
• How realistic was the dependent variable?
• Has the study been replicated in other situations?

47
Putting it all Together
Redesign Your Study