Advances in Monitoring Progress in Mathematics

1
Advances in Monitoring Progress in Mathematics

• Anne Foegen, Ph.D.
• Iowa State University
• Advances in Progress Monitoring
  Curriculum-Based Measurement Research and
  Innovations
• May 5, 2006

2
Contributors

• Stan Deno, University of Minnesota
• Erica Lembke, University of Missouri
• Cindy Jiban, University of Minnesota
• Deanna Spanjers, University of Minnesota

3
Session Overview

• RIPM Math strand
• Developing new measures
• K/1 numeracy measures
• Elementary numeracy measures
• Middle/high school algebra measures
• Questions/Discussion

4
RIPM Math Strand

• Analysis of existing research in math progress
  monitoring measures
• Unanswered questions and gaps at some levels
• Moving toward a seamless and flexible system of
  measures

5
Developing New Measuresin Mathematics

• Focus on brief measures that are viable for
  frequent progress monitoring
• Considerations of curriculum-sampling approaches
  vs. general outcome measurement
• Similar research process across studies and
  levels reliability, validity, sensitivity to
  growth

6
New Measures forKindergarten and 1st Grade
7
Previous Research

• Clarke Shinn (2004), Chard et al. (2005)
• Number identification
• Quantity discrimination
• Missing number
• Oral counting
• VanDerHeyden, et al. (2001 and 2004)
• Circle number
• Write number
• Draw circles
• Choose number
• Number naming (number identification)
• Count objects
• Free count
• Choose shape

8
Initial Studies (2004-05)

• Examined reliability and validity of four
  potential measures

Quantity Discrimination
Quantity Array
3
7
? ? ? ? ? ? ? ?
Missing Number
Number ID
0 ___ 2 3
6
9
Initial Studies (2004-05)

• Examined reliability and validity of four
  potential measures

Quantity Discrimination
Quantity Array
3
7
? ? ? ? ? ? ? ?
Missing Number
Number ID
0 ___ 2 3
6
10
Measure construction

• Quantity Discrimination
• Used number sets 0-10 and 0-20 (randomly
  selected)
• Number Identification
• Random numbers 0-100 50 of the numbers are
  between 0-20, 30 are between 0-50, and 20 are
  between 0-100
• Missing number
• Forward counting sequences by 1s (80), 5s, and
  10s
• Range for 1s 0-10, 5s 0-50, 10s 0-100
• Samples in handouts

11
Administration Procedures

• Individually administered (piloting group
  administration this year in 1st)
• Oral responses from students
• Each task is 1 minute

12
Early Numeracy Research Results
13
Method

• Parallel studies in MO and IA
• Fall, Winter, Spring
• Administer 2 forms of QD, NI, and MN to each
  student individually
• Each month, MO
• Administer 1-2 forms of QD, NI, and MN to each
  student individually

14
Participants, MO

• 107 students (77 K, 30 1st) in 6 K and 2 1st
  grade classes
• Ethnicity94 Caucasian
• 57 Male, 50 Female
• 25 free and reduced lunch
• 8 identified with special needs (YCDD,
  speech/language, MR)

15
Means/Standard Deviations for theInitial
Administration
K (n77) 1st (n30)
Average of 2 forms

• Sensitivity to growthFloor or ceiling effects?
• Discriminate among students at different skill
  levelslarge SD?

16
Technical Adequacy

• Reliability and Criterion Validity

17
Reliability

• Alternate form

• Test-retest (2004-05)

Reliabilitystronger for K students QD, NI
strongest for both grades
18
Criterion Validity
2005-06 Teacher ratings and Round 3 average of 2
forms 2005-06 SEA and Round 1 average of 2
forms 2004 Mini Battery of Achievement and median
of 3 forms
plt.05, plt.01, plt.001
Strongest validity, varies by measure
19
Monitoring Student Progress

• Growth, Predictive Validity

20
Weekly Growth Rates, 2005-06Quantity
Discrimination
21
Weekly Growth Rates, 2005-06Number Identification
22
Weekly Growth Rates, 2005-06Missing Number
23
Predictive ValidityStanford Early Achievement
Scores

• Fall of 2004( measures) to Spring of 2005
  criteria (n72)
• Test of Early Math Achievement and Average
  Teacher Ratings
• Kindergarten
• QD--.33, .64
• NI--.39, .63
• MN--.41, .44
• 1st
• QD--.41, .55
• NI--.59, .59
• MN--.59, .65
• Fall of 2004 (measures) to Fall of 2005 (SEA)
• QD--.57
• NI--.51
• MN.49
• n16-20, 1st grade only
• Median of 3 forms

24
Discussion/Implications

• What measures?
• PerformanceQD and NI for both MN for 1st grade
• ProgressBoth K and 1st grade students showing
  growth on all measures.
• Across 1 year (small sample)
• Within a year (K growing more quickly on NI and
  QD, but both K and 1 growing on all measures)
• Predictive validity
• For K, QD and NI have the best predictive
  validity with teacher ratings
• For 1st, NI and MN were strong predictors within
  a year and QD across 1 year

25
New Numeracy Measuresfor Elementary Students
26
Existing vs. New Measures

• Best existing MBSP Math (Fuchs, Hamlett,
  Fuchs, 1990/1994)
• Sample yearly curriculum differ at each grade
  level
• Range from 2 - 7 minutes per administration
• New measures
• Based on general proficiency with number
• Same at each grade level
• Brief 1 or 2 minutes

27
New Progress Measures

• Brief duration (1-2 minutes per task)
• Common forms and difficulty across elementary
  grades
• General outcome measurement model

28
Progress Measures
Complex Quantity Discrimination
Estimation
29
Progress Measures, cont.
Missing Number in Pattern
Quantity Arrays
Cloze Math
30
Progress Measures, cont.
Basic Facts
Mixed Numeracy
31
Administration Procedures

• Group administration
• Paper/pencil format
• Most measures are 1 minute easy estimation is 2
  minutes

32
Elementary NumeracyResearch Results
33
Participants

• Grade 2 n 107
• Grade 4 n 87
• Grade 6 n 75
• from 2 schools in 1 urban Minnesota district

34
Criterion Measures

• Northwest Achievement Levels Test (NALT) in
  Mathematics
• Teacher Judgment
• Rating of Math Proficiency (7 point Likert)

35
Procedures

• Battery of measures administered in late fall and
  spring
• Two rounds in fall to evaluate test-retest
  reliability
• Criterion variables collected fall (teacher
  ratings) and spring (NALT)

36
Technical Adequacy

• Reliability and Criterion Validity

37
Test-Retest Reliability (1 Week)

• Gr. 2 Gr. 4 Gr. 6
• Cloze Math .69 .87 .90
• Basic Facts .83 .79 .91
• Missing Number .89 .78 .93
• Quantity Arrays .63 .69 .77
• Estimation .72 .80 .76
• Complex Qty Discrim. .72 .83 .81
• _____________
• Note Scores are an average of 3 probes. All
  product-moment correlations significant, plt.01.

38
Alternate Form Reliability

• Gr. 2 Gr. 4 Gr. 6
• Cloze Math .72 .76 .85
• Basic Facts .77 .83 .86
• Missing Number .76 .76 .89
• Quantity Arrays .63 .69 .77
• Estimation .72 .80 .76
• Complex Qty Discrim. .72 .83 .81
• _____________
• Note Scores are an average of 3 correlations.
  All product-moment correlations significant,
  plt.01.

39
Criterion Validity with NALT

• Gr. 2 Gr. 4 Gr. 6
• Cloze Math .54 .57 .68
• Basic Facts .55 .53 .60
• Missing Number .49 .61 .66
• Mixed Numeracy¹ .40 .58 .66
• Quantity Arrays .40 .09 .33
• Easy Estimation .40 .23 .50
• Complex Qty Discrim. .37 .43 .66
• ___________________
• Note Scores are average of 3 probes.
• Significant, plt.01 .
• Significant, plt.05 .
• 1 Mixed Numeracy replaces Quantity Arrays in
  spring

40
Concurrent Validity with Teacher Ratings

• Gr. 2 Gr. 4 Gr.
  6
• Fall / Spr Fall / Spr
  Fall / Spr
• Cloze Math .50/.50 .39/.42
  .47/.63
• Basic Facts .59/.46 .36/.41
  .50/.54
• Missing Number .55/.43 .44/.37
  .55/.57
• Quantity Arrays .36/ -- .26 / --
  .28 / --
• Mixed Numeracy -- /.42 -- /.44
  -- /.58
• Easy Estimation .29/.35 .12 /.26
  .21 /.45
• Complex Qty Dis .35/.32 .22 /.27
  .46/.67
• ___________________
• Note Teacher ratings combined by grade, across
  teachers, for grades 2 4. Scores are average of
  3 probes. QA administered fall only MX spring
  only.
• Significant, plt.01 . Significant, plt.05 .

41
Monitoring Student Progress

• Growth, Predictive Validity

42
Growth Across Grades
43
Predictive Validity with NALT

• Gr. 2 Gr. 4 Gr. 6
• Cloze Math .51 .59 .64
• Basic Facts .53 .43 .58
• Missing Number .55 .59 .69
• Quantity Arrays .40 .09 .33
• Easy Estimation .09 .35 .59
• Complex Qty Discrim. .31 .43 .62
• ___________________
• Note Scores are average of 3 probes.
• Significant, plt.01 .
• Significant, plt.05 .

44
Discussion/Implications

• Cloze Math, Missing Number in Pattern, and Basic
  Facts maintain moderate validity ( .5)
  across elementary grades Of these, Missing
  Number and Basic Facts show reliability .8
  across elementary grades
• Alone, these measures do not demonstrate the
  level of technical adequacy produced by existing
  mathematics measures for elementary
• Number correct in one minute shows growth across
  grades for all measures
• Average growth of less than 4 items correct per
  school year limits utility for within-grade
  progress monitoring

45
New Measures for Algebra 1and PreAlgebra
46
Existing Options for Secondary Math

• Middle School
• Estimation (Foegen)
• Facts (RIPM, Foegen)
• Concept-based measures (Helwig Tindal)
• Stretching Fuchs Fuchs MBSP measures for
  lower functioning students
• All measures are designed for general mathematics
• High School
• No existing measures specifically for high school
  content

47
Monitoring Progress inSecondary Mathematics

• Content diversity issues
• Content is a greater driving force than grade
  level
• National trends for challenging curriculum often
  include algebra as a graduation requirement
• Algebra as a gatekeeper

48
Developing Algebra Measures

• Initial pool of 5 alternative measures to
  identify those that are most promising
• Technical adequacy studies distributions,
  reliability, criterion validity
• Pool narrowed to three most promising measures

49
Algebra Progress Measures

• Basic Skills A
• Algebra Foundations B
• Content Analysis- C
• Constructed Response
• Translations D
• Content Analysis- E
• Multiple Choice

50
Basic Skills (in Algebra) A

• 60 items 5 minutes
• Problems include
• Solving basic fact equations
• Applying the distributive property
• Working with integers
• Combining like terms
• Simplifying expressions
• Applying proportional reasoning
• Scoring of problems correct

51
Algebra Foundations B

• 42 items (50 points) 5 minutes
• Problems represent five core concepts/skills
  essential to conceptual understanding in algebra
• Writing and evaluating variables and expressions
• Computing expression (integers, exponents, and
  order of operations)
• Graphing expressions and linear equations
• Solving 1-step equations and simplifying
  expressions
• Identifying and extending patterns in data tables
• Scoring of problems correct

52
Content Analysis-Multiple Choice E

• 16 items 7 minutes
• Problems are sampled from 2-3 core concepts in
  each chapter from chapters 1-8
• Must show work to obtain partial credit
• Scoring Up to 3 points per problem, -1 pt.
  penalty for circling an incorrect answer without
  showing any work (guessing)

53
Algebra Research Results
54
Setting and Context

• Three districts participating in project
• District A small bedroom community
  predominately white 16 free/reduced lunch 600
  students in 7-12
• District B blue collar town 85 white 23
  free/reduced lunch 1350 students in 9-12
• District C rural area including a Native
  American settlement school 75 white 44
  free/reduced lunch 500 students in 9-12
• Three rounds of studies conducted spring 2004
  through spring 2005 measures revised between
  studies
• Focus today will be on spring 2005 technical
  adequacy studies

55
Participants and Measures

• Spring 2005
• 78 students (6 with IEPs) in districts B and C
  completed two forms of each type of probe
• Probe administration process repeated 7-10 days
  later
• Criterion measures included algebra grade, GPA,
  standardized test scores, teacher ratings, Iowa
  Algebra Aptitude Test scores

56
Reliability of Aggregated Probes
Note Mean of two probes
57
Criterion Validity for Aggregated Probes
58
Criterion Validity for Aggregated Probes
59
Monitoring Student Progress

• Growth Studies

60
Growth Study Method

• Participants
• 217 students in 3 districts 25 with IEPs
• 7 general education teachers
• Measures
• Three algebra progress measures revised versions
  used
• District A Algebra Foundations and Content
  Analysis probes
• Districts B and C Basic Skills and Content
  Analysis probes
• Procedures
• 1 semester duration
• Students completed two probes twice each month,
  type of probe alternated mean scores used for
  analysis
• Data obtained for 4 time periods for each type of
  probe

61
Participating Classes

• District A Traditional Schedule
• 8th grade algebra
• Pre-algebra (1st half of Algebra 1)
• Algebra 1
• Districts B and C Block Schedule
• Algebra 1A (1st half of Algebra 1)
• Algebra 1B (2nd half of Algebra 1)
• Algebra 1

62
District A Mean Scores
63
Districts B and C Mean Scores
64
Mean Slope Values(based on individual student
slopes)
65
Discussion/Implications

• Reliability is acceptable when the mean of two
  probes is used
• Criterion validity in moderate range with algebra
  grades, IAAT, teacher ratings
• Content Analysis measure tends to be more
  sensitive to changes in performance than Basic
  Skills or Algebra Foundations for most students
• For lower performing students, the pattern of
  results is less clear. The growth for these
  students, particularly those in Districts B and
  C, was more similar across probe types.

66
Questions and Discussion

• afoegen_at_iastate.edu