Fixing a Broken Model

1
Fixing a Broken Model

• How to Use Significance Testing to Assess for
  Disproportional Representation

2
Presenters

• Clifford V. Hatt, Ed.D., ABPP
• Coordinator, Psychological Services,
• Virginia Beach City Public Schools, Virginia
• Jon C. Thompson, Psy.D., HSPP
• Pediatric Neuropsychologist
• St. Vincents Hospital, Indianapolis, Indiana

3
Whats Broken?

• No way to tell if two different percentages are
  really different
• Using the white student population as the
  comparison group
• Using adjustment amounts to determine if the
  percentages are meaningful
• Considering a disproportion instead of incidence
  of a disorder

4
The Questions and Issues

• The What and The How of Disproportionate
  Representation

5
Proportional Concept

• Assumes same ethnic proportions will/should be
  reflected in all educational situations
• Special Education Classifications
• Referrals for educational assistance
• Discipline referrals
• AP classes
• Drop out rates
• Seems simple and self-evident

6
Disproportion vs. Incidence

• Do disabilities and illnesses occur in the
  general population and in subpopulations at the
  same rate?
• If the incidence of mental retardation occurs in
  1 of the general school age population (based on
  USDOE data), can we expect it to occur at that
  same rate in specific school-aged ethnic groups?
  

7
2004 of Students Age 6-21 Served Under IDEA,
Part B
All LD SLI MR ED Other
United States 9.16 4.24 1.73 0.84 0.73 1.62
Virginia 9.62 4.12 1.45 0.80 0.78 2.47
8
Incidence Rates Reported in DSM-IV-TR

• Learning Disability
• Approximately 5 of students in public schools
  are diagnosed with a learning disability
  (reading, written language, math)
• Mental Retardation
• Approximately 1 of general population is
  diagnosed with mental retardation

9
Is Special Education Data Consistent?
MR LD
United States 2004 IDEA Data 0.84 4.24
Virginia 2004 IDEA Data 0.80 4.12
DSM-IV-TR Reported Incidence Rate 1.0 5.0
10
Consistency of Comparative Data

• If Special Education Data is consistent, which
  incidence rate should be used for comparison?
• United States Data
• Virginia Data
• DSM-IV-TR

11

• For every complex problem there is an answer
  that is clear, simple, and wrong.
• H. L. Mencken (1880-1956)

12
Problems with the Methods

• Risk Index
• Number of students from a given ethnic background
  identified with a disability/Total number of that
  ethnicity in the general student population
• Similar to incidence rate
• Only descriptive analysis

13
Problems with the Methods

• Odds Ratio (Relative Risk Index)
• Uses the Risk Index of an ethnic minority as the
  nominator and the Risk Index of whites as the
  denominator
• Gives simple comparison of groups with whites
  serving as the control group
• Can determine whether a particular ethnic group
  will be identified as having a disability more or
  less often compared to whites

14
Problems with the Methods

• Odds Ratio (Relative Risk Index)
• No way to evaluate for significance
• Assumes whites are the majority and are
  representative of a disability group as a whole
• Assumes all disabilities are equally represented
  across ethnic groups

15
Problems with the Methods

• Composition Index
• Calculated by dividing the of students of a
  racial or ethnic group in a category by the total
  of students in that ethnic or racial group in
  the total student population.
• Provides composition for a pre-determined
  population so that when all compositions of the
  different ethnic groups are added together, total
  equals 100

16
Problems with the Methods

• Composition Index
• Composition of the identified category is then
  compared to composition of the population as a
  whole
• No way to evaluate for significance
• Highly susceptible to interpretation flaws when
  population under consideration is small
• Assumes all disabilities occur with equal
  frequency across all populations

17
Arbitrary Numbers

• It may be recognized that composition ratios are
  not very useful when comparing subgroups.
• Some type of statistical adjustment is often
  suggested without any clear rationale.
• Some states may use /- 20 percentage points
  to suggest significant disproportionality
  (Coutinho Oswald, 2004).

18
A Fix for the Problems of a Broken Model

• What if you could determine if two percentages
  (proportions) were really different?
• What if you could compare local with state data
  and state data with federal data using the same
  method?
• What if you could track progress made after
  interventions were implemented?

19
A Fix for the Problems of a Broken Model

• What if you actually saw measurable differences
  in what you did instead of always never quite
  measuring up?
• What if this all made sense?!!!

20
One Solution

• Determine the significance of the difference
  between two proportions (percentages are just a
  special type of proportion)
• Determine if real statistical differences are
  found
• Use a z-score as the common measure

21
Standard normal distribution
-2 -1 0 1 2
22
Within 2 SDs ( really 1.96)
95
-2 -1 0 1 2
23
Formula Key
Population Population proportion Sample size ID Sp Ed Sample proportion
1 p1 n1 X1
2 p2 n2 X2
24
Formula for z
25
Formula in Special Ed Terms

• z ( min spec ed - min students)/ sqrt(x
  1-x)(1/n1 1/n2)
• x ( min students spec ed min students)/(n1
  n2)
• n1 total minority students
• n2 total spec ed students

26
A Simple Way to Do It

• Excel spreadsheet with the formula imbedded Excel
  compared to state population
• Enter the number of minority students in special
  ed and the number of minority students in the
  local student population
• Select a comparison group state data or national
  data
• Enter the respective information

27
Results for Virginia
Current Method New Method
Districts Demonstrating Disproportionality Districts Demonstrating Disproportionality
Amer. Indian 23 25
Asian 4 9
Black 96 47
Hawaiian 7 9
Hispanic 37 2
White 58 17
28
Implications

• Different choices in the use of the comparison
  group make a difference!
• The method you chose to compare data makes a
  difference!
• Disproportionality is a problem regardless of
  method.
• More specific methodology allows for more
  appropriate, targeted remedies.
• Can be easily calculated in an Excel spreadsheet
  format.

29
Implications

• Modified z-score method provides a way to
  determine whether the differences between
  predicted and obtained proportions are
  meaningful.
• More appropriately identifies districts that need
  to have corrective action.
• Also provides a way to determine progress or
  change following interventions.
• Accounts for differences in size or number of
  students under consideration.

30
A Little Comic Wisdom

• Politics is the art of looking for trouble,
  finding it everywhere, diagnosing it incorrectly
  and applying the wrong remedies.
• Groucho Marx (1890-1977)