Topics in Special Education Research

1
Topics in Special Education Research

• Session 4 Single Subject Research Methodology

2
Proposal Assignment Group Work

• A detailed explanation of the assignment is
  posted on the wiki
• What should you be doing in your groups?
• At this point you should have a topic and start
  coming up with your framework for your research
  project (based on literature).
• Start to draft your conceptual framework,
  research questions identify your dependent and
  independent variables
• You should walk away from your group time with a
  list of tasks to complete.

3

• Socially Important Issue
• 2. Conceptual Model/Hypothesis
• 3. Research Question(s)
• 4. Dependent Variable
• 5. Dependent Variable Measure
• 6. Independent Variable
• 7. Independent Variable Measure
• 8. Research Design

4
This Afternoons Agenda

• Short presentation to concisely answer questions
  regarding
• last class
• use of statistics in analyses
• Review, Take Correct Quiz
• Discussion with classmates regarding Single-Case
  Design Readings
• Lecture on Single-Case Design
• In-Class Activity Designing a Single-Case Study

5
Difference between Inter-observer agreement
Treatment integrity/fidelity

• Inter-observer agreement (IOA)- involves 2
  observers measuring the same behaviors at the
  same time
• It is most often used to determine the
  reliability of observations of the DEPENDENT
  VARIABLE
• Treatment Integrity/Fidelity
• (of treatment/intervention/ INDEPENDENT
  VARIABLE)
• This is how the researcher measured how well the
  treatment/intervention was implemented
• Commonly done using checklists and other
  observers recording the completion of these
  checklists

6
Experimental and Quasi-Experimental vs other
designs

• The main difference between experimental/quasi-exp
  erimental research designs AND other designs is
• They MANIPULATE the independent variable
• Basically..they introduce an intervention, while
  other methods (except for single-case designs) do
  not systematically introduce an intervention
• .seeks to make CAUSAL CONCLUSIONS

7
Statistics, statistics
Descriptive Statistics
Inferential Statistics
Who is in your data?
What your sample says about the population
sample
population
Mean, Median, Mode, Standard Deviation, Variance
Tests of significance (t-, F-Tests)
8
Tests of Significance

• Statistical analyses to determine whether a
  difference is statistically significant
  (probability for result to occur by chance).
• Yes or No answer
• Alpha level (p)
• An established probability level which serves as
  the criterion to determine whether to accept or
  reject the null hypothesis
• Common levels in education
• .01
• .05
• .10

Objectives 4.1 6.1
9
Statistics, statistics
Descriptive Statistics
Inferential Statistics
Who is in your data?
What your sample says about the population
sample
population
Mean, Median, Mode, Standard Deviation, Variance
Tests of significance (t-, F-Tests)
10
Inferential Statistics

• T tests- used when have two groups to compare.
• Independent samples t- if groups are independent
• Different people in each group
• Dependent samples t- if two sets of scores are
  available for the same people (e.g., pre and
  post-tests of same group)
• Matched groups
• ANOVA (analysis of variance)- when you have more
  than 2 groups to compare OR more than one
  independent variable (reports an F-statistic,
  which is basically a t-value squared)
• ANCOVA (analysis of covariance)- ANOVA that
  allows for control of the influence of an IV
  (e.g., characteristics of people) that may vary
  between your groups before treatment is
  introduced.
• Post-hoc method for matching groups on variables
  such as age, prior education, SES, or a measure
  of performance

11
Effect Size

• Way of quantifying the difference between two
  groups.
• Not just was there an effect, but the magnitude
  of the effect.
• Many ways to calculate
• ES Mean of experimental group Mean of
  control group/Standard Deviation
• R-squared, Cohens-D
• Standard deviation is how well the mean
  summarizes the data

12
(No Transcript)
13
(No Transcript)
14
Visible Learningby John Hattie

• Over 800 Meta-analyses
• https//www.youtube.com/watch?vsng4p3Vsu7Y

15
Review for Quiz
16
Steps in the Research/Scientific Process

• 1. Identify socially important issue
• 2. Review current literature
• 3. Define conceptual model
• 4. Define specific hypothesis(es) and research
  question(s)
• 5. Define dependent variable(s)/measure

• 6. Identify independent variable(s)/measures
• 7. Select appropriate research design
• 8. Obtain consents
• 9. Collect data
• 10. Analyze data
• 11. Communicate results
• Written presentation
• Oral presentation

17
Experimental Design

• Seeks to make causal conclusions.
• Direct manipulation of an independent variable
  (intervention)
• Difference between experimental design and
  quasi-experimental design is the use of random
  selection of participants and conditions.

18
Validity

• Refers to whether a study is able to
  scientifically answer the questions it is
  intended to answer.
• Extent to which your test (or study) measures
  what it intends to measure.

19
Internal Validity

• Changes observed in the dependent variable
  (outcome) are due to the effect of the
  independent variable (intervention)..
• not to some other unintended variables
  (extraneous, alternative explanations)
• 12 threats to internal validity (noted by
  Mertens, 2010)
• E.g.., History, maturation, testing,
  instrumentation, mortality, etc.

20
External Validity (think generalizability)

• External Validity extent to which findings in
  one study can be applied to another situation.
• AKA ecological validity, generalizability
• 10 threats posed as questions (noted by Mertens,
  2010)
• E.g., detail/description of procedures,
  experimenter effects, sensitization, etc.

21
Quiz
22
Correct Quiz
23
Discussion
24
Lecture

• Please get into your research groups for the
  lecture portion.
• You will be completing the in-class activity
  together with your group.

25
Single Subject Research

• Systematic analysis using individual subjects as
  their own experimental control.
• Main message
• Single subject research is an approach to
  rigorous experimentation that involves small
  numbers of subjects, repeated observations of
  subjects over time, and employs research designs
  that allow each subject to provide his/her own
  experimental control.
• Within-subject analysis
• Fine-grained analysis across time and conditions

26
Defining Features of Single Subject Research

• An experimental research method focused on
  defining causal (e.g., functional) relations
  between independent and dependent variables.
• Focus is on individuals as unit of analysis
• can treat groups as participants with focus on
  the group as a single unit
• Repeated measures of participants behavior (DV)
  over time
• Within-subject comparison to analyze effect
• Observed change in individuals behavior from
  Baseline to Intervention

27
Reasons for using single subject methodology

• Focus on an individual rather than group means
• Interest is in the behavior of a single
  individual or on within-subject variability
• A group may be treated as an individual
• Group descriptive statistics may not "describe"
  any actual individual
• Generalizations from a group to an individual are
  problematic in many instances
• Predicting the behavior of a specific individual
  is different from predicting that of a typical
  individual

28
Reasons for Using Single Subject Methodology
(continued)

• Many populations of interest are low incidence
  populations
• Practically, large numbers of subjects may not be
  available
• Assumptions of normal distribution and
  homogeneity of variance may not be valid
• Can be used in clinical practice contexts
• Single subject research studies may develop out
  of and be conducted on a specific problem or need
  of an individual(s) in a practical context
• Scientist-practitioner model

29
Using Single Subject Research to Establish
Evidence-based Practices

• A practice may be considered evidence-based
  when
• The practice is operationally defined, and
  implemented with fidelity.
• The outcomes associated with the practice are
  operationally defined.
• The context in which the practice in use is
  operationally defined
• Results from the single subject studies used to
  assess the practice demonstrate experimental
  control.
• The effects are replicated across 5 single
  subject studies conducted in at least 3
  locations, and with at least 20 different
  participants.

30
Dependent and independent variables

• Dependent variable (DV) the behavior (measure)
  that you are analyzing
• You want to produce change (variability) in the
  dependent variable
• Studies may have multiple DVs
• Independent variable (IV) the variable (event,
  intervention, condition) that is of experimental
  interest and that the researcher manipulates in
  an experimental research design
• May be discrete or continuous
• May be a single element or multi-component
  compound
• Studies may have multiple IVs

31
Internal Validity

• The degree to which observed differences/changes
  in the dependent variable are a direct result of
  manipulation of the independent variable, and not
  some other extraneous variable
• Extent to which a functional relation can be
  documented. Control of extraneous variables that
  provide alternative explanations for results.
• It is okay to try to maximize internal validity,
  especially in initial documentation of a
  functional relationship
• Doing this may come with a cost, however

32
Threats to Internal Validity

• History everything happening outside of the
  research study
• Maturation
• Testing - repeated measurement
• Instrumentation
• with human observers, observer bias and drift
• Attrition - loss of participants
• Multiple treatment interference
• Diffusion of treatment - intervention is
  inadvertently provided when not intended

33
Threats to Internal Validity (continued)

• Loss of baseline through generalization or spread
  of effects (across settings, behaviors, or
  participants)
• Instability and/or high variability of behavior
• cyclical variability
• Statistical regression toward mean
• Selection biases with participants
• Inconsistent or inaccurate implementation of the
  IV (Treatment Drift/Treatment Integrity)

34
External Validity

• Defined The extent to which results can be
  applied to settings, activities, people, etc.
  other than those involved in the study.
• Given that you have found an effect for this
  intervention with this participant under one set
  of conditions, will it work with other
  participants, in other settings, when implemented
  by other interventionists, and when implemented
  with minor variations in the basic procedures?
• What can we generalize from this single study?
• Importance of systematic and direct replication.

35
Threats to External Validity

• Reactive experimental arrangements - Hawthorne
  effect
• Reactive assessment - reactivity to observers
• Pretest sensitization
• Experimenter bias
• Interaction between selection bias and treatment
  effects - i.e., intervention only works if the
  "right" participants are selected
• Specificity of effects

36
The Research Question

• In single subject designs the research question
  typically examines a causal, or functional
  relation, between the independent and dependent
  variable. As such the research question should
  have three features
• Identify the dependent variable(s)
• Identify the independent variable(s)
• Proclaim intention to determine if change in the
  IV is functionally related to change in the DV.

37
Research Question Features

• Dependent variable is socially important
• Independent variable(s) can be controlled (e.g.
  manipulated) across time.
• Both the dependent and independent variable(s)
  can be operationally described and measured.
• For experimental research, the question must
  ask if change in the DV is caused by (or
  functionally related to) change in the IV.

38
Research Question Examples

• Is there a functional relation between
  development of reading fluency and scores on
  comprehensive reading assessments?
• Will walking in water facilitate development of
  appropriate gait by individuals with gait
  imbalance hypertension?
• Is there a functional relation between use of
  escape-extinction and reduction of
  escape-motivated food refusal?
• Does Jason act out because he has ADHD?

39
For your research study define your DV, IV,
SSD research question

• 1. Dependent Variable (Outcome)
• 2. Independent Variable (Intervention)
• 3. Research question Is there a functional
  relationship between and ?

40
Phase A
Phase B
Phase A
Phase B
Immediacy of Effect
Variability
Level
Trend
Overlap
Research Question???
41
Phase A
Phase B
Phase A
Phase B
Immediacy of Effect
Variability
Level
Trend
Overlap
Research Question???
42
In SSD, a Functional Relationship/Experimental
Control has occurred when

• There are 3 demonstrations of an effect at 3
  points in time.
• Effect could be change in trend or level
• Also want to see immediacy of effect
• Good research has at least 5 data points in each
  phase to establish a consistent pattern in the
  data.

43
Establishing a Baseline

• Baseline - phase in a design that serves as the
  reference point or comparator for analysis of
  change in behavior (effect of IV)
• Used in withdrawal/reversal and multiple baseline
  designs may be included in alternating
  treatments design (but not needed)
• Generally, the first phase, but not always
• Returned to periodically in withdrawal/reversal
  designs
• Provides (should provide) a representative
  picture of behavior under pre-intervention
  (typical, status quo) conditions
• Baseline is the control condition in within
  subject analysis
• May involve some alternative intervention/treatmen
  t

44
High Variability in Baseline?

• Use baseline phase to do close observation to
  reveal potential sources of variability
• Control variability through elimination or
  holding constant extraneous variable(s)
• Consider whether sources of variability should be
  studied as IVs
• Be alert to dramatic changes within the phase and
  identify potential causes
• Balance logistical and clinical needs with
  research goal of stability
• Recognize potential limitations and threats to
  internal validity if you have high variability

45
Trends in baseline data?

• Trends (increasing or decreasing slope) can be
  accepted, if the trend is in the opposite
  direction of the anticipated effect of the IV
• Visual analysis does consider changes in trend
  across/between phases
• Trend in the expected change direction is
  problematic
• Collect more data points
• Consider whether intervention is warranted
• If substantial change in slope is expected, you
  may go forward with intervention
• Statistical analysis may be used to supplement
  visual analysis

46
Guidelines for Implementing IVs

• Implement based on data collected in baseline (or
  previous phases), rather than on a predetermined
  schedule that is independent of the data
• Establish effects of IV on one baseline (data
  path) before implementing IV in another baseline
  (data path) in a multiple baseline
• Collect and report measures of IV implementation
  fidelity

47
Length of Phases

• Phases should be long enough to establish
  representativeness of data within the phase
• Reach stability within the phase (at least 5
  points)
• Some have argued that for power, the number of
  data points in SS design is comparable to number
  of subjects in group design
• Researchers often want to use relatively short
  phases
• Because of logistical issues, ethical issues,
  impatience, costs
• Be aware of limitations and threats to validity
• Phases of very different lengths within a design
  (particularly ABAB) can create issues for visual
  analysis and interpretation of effects

48
Defining Features of Multiple Baseline Designs

• A multiple baseline design involves three or more
  AB interventions (series) with phase changes
  staggered across at least three points in time.
• Key Features
• Series are independent of each other
• People, places, materials, behaviors/skills
• The same IV is applied in each series
• Staggered implementation of IV

49
Interpreting MBL Designs

• Identify Research Question(s)
• Assess Baselines for each series
• Do the Baselines document a predictable pattern?
• Do Baselines allow opportunity to document IV
  effect?
• Are Baselines similar?
• Horizontal Analysis of Effect (per series)
• Level, trend, variability, overlap, immediacy of
  effect
• Vertical Analysis
• DV change in one series is associated with NO
  change in other series?
• Similar effect (consistent effect) across series?
• Functional Relationship?
• At least three demonstrations of effect at three
  points in time

50
Lollipop for R
BL
Treatment
6
100
80
60
40
20
Vivian
0
Lollipop for R
100
80
60
Percentage of Correct Responding
40
20
Tammy
0
Lollipop for R
100
80
60
40
20
Dr. Cathy
0
10
20
30
40
50
60
70
Sessions
51
Defining features of withdrawal and reversal
designs

• Sequential phases of data collection involving
  the implementation and withdrawal of an
  independent variable(s)
• within each phase, multiple data points are
  collected to establish a representative pattern
  of behavior
• phase change should occur only after stability of
  behavior within the phase is established
• traditionally, the first phase is Baseline,
  followed by implementation of the IV
  (Intervention)
• this is not required, however, as you may begin a
  study with an intervention phase

52
When are reversal and withdrawal designs
appropriate?

• Behavior measured as DV is reversible
• Learning will not occur
• Limited carryover effects between phases
• Ethical concerns
• Can do a reversal
• DV is not a dangerous behavior, or you can
  protect participant
• Staff cooperation
• Can compare multiple conditions
• Comparison of too many conditions makes design
  cumbersome

53
4B
FCT
Baseline
Baseline
FCT
6
5
4
Total SIB per minute
3
2
1
0
1
5
10
15
20
25
30
35
Sessions
54
ATD/ MED Defined

• Alternating Treatment (Multi-Element) Designs
  employ rapid phase reversals across 2 or more
  conditions to assess sensitivity of change in the
  dependent variable to change in condition.

55
Student 1Hypothesis Escape Math Work
2. Is Esc different than Attn?
1. Is Esc different than Control?
56
In-class Activity 4

• State the single-case research design you would
  use for your study and why?

57
Measurement in Single Subject Designs

• The selection of measures is PART of building a
  single subject design.
• All single subject designs require measures that
  allow documentations of
• A stable pre-intervention pattern of performance,
  and
• A rapid and dramatic change in performance
  following intervention.
• Measures must be reliable/consistent enough to
  document pre-intervention stability, and
  sensitive enough to document rapid, dramatic
  change.

58
Fundamental Dimensions of Behavior

• Frequency
• The number of occurrences of a response within an
  observation period.
• Duration
• The total time taken to perform a response
  (typically indexed as the mean duration)
• Latency
• The time between the presentation of the Sd, and
  the initiation of a response.
• Perseveration
• The proportion of the observation period/interval
  in which responding was occurring. (Total time
  for all occurrences)
• Rate
• The frequency of a response divided by the total
  time for an interval (typically occurrences per
  minuteor occurrences per second).

59
Measurement Procedures

• Event recording
• Observe number of occurrences within an
  observation period
• Duration recording
• Observe the mean time of responding per
  occurrence (tempo)
• Interval recording
• Observe the proportion of intervals in which the
  behavior occurs.
• Whole interval versus partial interval recording.
• Time sampling
• Proportion of time sampled moments in which
  behavior is occurring.
• Permanent product
• Count of products from behavior. Note No direct
  observation
• Narrative
• Continuous description of behavior in real time

60
In-Class Activity 5 6 select measures for
your variables.

• Define a research question
• For the Dependent Variable
• Select a measure
• Select a measurement process
• For the Independent Variable
• Select a measure
• Select a measurement process

61
Building Data Collection Forms

• Paper/Pencil or Computer Entry/PDA
• Key Features
• Logistical Information
• Date, observer, observed,
• Ease of recording (eyes on context)
• Key strokes or checks instead of writing words.
• Number of variables recorded simultaneously (3 is
  plenty)
• Operational definitions
• Fit the context and range of observed behavior
• Instructions on setting up a data session
• System for summarizing session results.

62
Nifty Observation Form   Date ___________________
_____   Observer _____________________   Context
______________________   Request Statement from
teacher requesting response by target
student Compliance Initiation of requested
response within 5 s of request Noncompliance
Absence of initiation of requested response
within 5 s of request. Problem behavior Talking
out, aggression, property destruction,
disruption.  
10 s Interval Request Compliance ()/ Noncompliance (0) Problem Behavior   Comments/Issues
1        
2        
3        
4        
5        
6        
7        
8        
9        
10        
 
63
In-class Activity 7

• Build a data collection form based on how you
  plan to measure the data.

64
Inter-observer Agreement

• Proxy for reliability but not really a measure of
  reliability.
• Poor IOA means poor reliability, but good IOA
  does not prove good reliability.
• Two practical measures
• Percent agreement (Total, Occurrence Only)
• Kappa

65
Percent Agreement

• Defined The extent to which two, independent
  observers agree they observed the same events at
  the same time.
• Operationalized. Given a group of observation
  intervals, to what extent do the frequencies or
  interval recordings co-vary across two,
  independent observers. What percent of the
  intervals index agreement?
• Calculation.
• (Frequency of observations with agreement/ total
  number of observations) 100
• Frequency observed by Observer 1/Frequency
  observed by Observer 2 (correlation)

66
Percent Agreement

• Advantages
• Easy to compute
• Easy to understand
• Failure to obtain criterion level is informative.
• Disadvantages
• Is not a measure of reliability
• Provides an over-estimate of agreement
  (especially when lt10 or gt90 of intervals
  include occurrence.

67
Percent Agreement

• Professional Standards
• 85 agreement is expected for good IOA
• Occurrence Only vs Total Percent Agreement
• Occurrence/Nonoccurrence Only is used to assess
  agreement when lt10 or gt 90 of intervals include
  occurrence.
• Calculate (use in denominator) only using those
  intervals in which either of the observers
  recorded a response (Occurrence Only) or only
  those intervals with either of the observers did
  not record a response (non-occurrence only).
• Controls for one source of bias.

68
Cohens Kappa

• Purpose of Kappa is to provide an index of
  observer agreement that controls for chance
  agreements.
• Kappa can range from 1.00 to 1.00
• .40-.60 fair agreement
• .60-.75 good agreement
• .75 generally needed for publication in
  Tier 1 journals

69
Kappa

• Calculation
• Kappa (Po- Pc) / (1 Pc)
• Where Po the proportion of observed agreements
• Where Pc the proportion of agreements expected
  by chance.
• Recommendation
• Report both percent agreement and Kappa.
• Use Occurrence/Non-occurrence Only when
  appropriate

70
Ethics in Single Subject Research
71
Issues related to single subject research design
features

• Withdrawal/Reversal Designs
• Implementing withdrawal/reversal phases length
  of phases when DV is problematic
• End study with participants in the "optimal"
  phase
• Adequate baseline length
• Multiple Baseline Designs
• Extended baselines treatment phases
• No treatment/intervention "control" baselines
• Reaction to measurement or other research
  procedures
• Set research session termination guidelines
  criteria to protect everyone terminate sessions
  when criteria are met
• Have a plan to protect participants and others,
  and to bring situations under control if crisis
  occurs

72
Issues related to applied research in natural
settings

• Minimize negative images and stigma
• Use unobtrusive measurement (as possible)
• Appropriate selection of DV measures
• For example, use latency to problem behavior
  rather than rate in community settings
• Dignified procedures
• Responding to "citizen" questions or comments
• Ensuring cooperation and support of others in
  natural settings
• Open communication before and during study
• Obtain appropriate permissions consents
• Be courteous respectful
• Allow people in the setting (teachers, families,
  staff) some voice
• Include community "others" as research
  partners/collaborators

73
Exiting research projects gracefully

• Plan for exit
• Leave participants in "optimal" phase or state of
  performance
• Provide training and support (i.e., plan,
  materials, etc) for natural community members to
  assume and maintain implementation of
  intervention
• Provide information on results and their
  implications for natural setting
• Provide follow-up if necessary
• Agree on researcher responsibilities on the front
  end (before study)

74
Activity 8. Draw your research design and
proposed data
75
4B
FCT
Baseline
Baseline
FCT
6
5
4
Total SIB per minute
3
2
1
0
1
5
10
15
20
25
30
35
Sessions