Use of SOL Test Data at the Student, School, and District Level

1
Use of SOL Test Data at the Student, School, and
District Level

• James H. McMillan
• Lauren M. Miller
• Virginia Commonwealth University

Presentation at the 2005 Annual Conference of the
Virginia Association of Test Directors
Slides available at http//www.soe.vcu.edu/merc/
2
(No Transcript)
3
Data Driven
Data Deluged
Data Doped
4
Keep Standardized Tests in Perspective

• These tests dont measure
• Motivation Self-efficacy
• Attitudes Empathy
• Compassion Creativity
• Citizenship Persistence
• Curiosity Altruism

5
The Good Ole Days

• Large Scale
• Assessment

• Classroom
• Assessment

6
(No Transcript)
7
These Days

• Large Scale
• Assessment

• Classroom
• Assessment

8
(No Transcript)
9
This is what we are doing
InferenceUse Conclusion Claim Consequence
Research Question
SOL Scores
Validity
10
This is what we want to say

• Higher SOL test scores better education
• (conclusion)

This is what we should not say

• Higher SOL test scores better education
• (conclusion)

11
(No Transcript)
12
Why?

• Error in testing
• Limited coverage of important objectives
• Unethical test preparation
• Single indicator
• Inflation (not your )

13
Testing Error Pyramid
Least Error
State
Most Error
14
More accurately
InferenceUse Conclusion Claim Consequence
Research Question
Score Error
15
(No Transcript)
16
How Much and What Kind of Error for Individual
Scores??

• Question Score Error Conclusion
  

17
For Example

• Scores
• Johnny 401
• Sally 396

Conclusion Johnny is proficient Sally isnt
Question What is the math proficiency of Johnny
Reed compared to the proficiency of Sally Smith?
18

• SOL test score validity is only concerned with
  inferences about the domain of knowledge and
  skills being tested.

Virginia Social Studies Domain
SOL Domain
Number of Items

15
History
8
40 Item Test
Civics
7
Geography
Econ
10
Virginia Studies
19
Grade 8 History and Social Science US History to
1877
Year 1 SOL Test Items
Year 2 SOL Test Items
7 SOLs
19 Areas
0
1
1
0
1
1
1
0
2
1
1
1
2
1
1
0
1
0
1
0
1
2
2
1
1
1
Total Number of items
12
12
20
Grade 5 History/Social ScienceSpring 2003

• Student Raw and Standard Scores

600
0
399
21
Grade 5 History/Social Science Spring 2003

• Student Raw and Standard Scores

1 SEM
600
0
399
416
22
Grade 5 History/Social Science Spring 2003

• Student Raw and Standard Scores

23
Grade 5 History/Social Science Spring 2003

• Student Raw and Standard Scores

24
(No Transcript)
25
Measuring Score Changes at the Classroom
School Level

• Error
• Actual amount of change from year to year
• Statistically significant changes
• Effect size

26
How Much and What Kind of Error for Class,
Teacher, or School Scores??

• Score Error Conclusion
  

Research Question
Test
Expressed As
1. Random
Standard error
Student
2. SOL Sampling
? ?
3. Nonrandom human (bias)
? ?
4. Linking
? ?
? ?
5. Cohort effect
27
Classroom Case in Point Actual Change in Ms.
Lopezs Math Class

• 2004 2005 Change
  419.20 419.32 .12
• My class did better in 2005
• My teaching was better in 2005
• I had smarter kids in 2005

28
2004-2005 Significant Fifth Grade Class
Change Due to

• Changes in students?
• Better alignment?
• More parental involvement?
• Different curriculum?
• Better classroom assessments?
• Grading practices?
• Instruction?
• Luck?

29
School Case in Point Jefferson Elementary

• 00 01 02 03 04 05
• 70.1 70.2 70.3 70.4 70.5 70.6

• Parents and members of the board of education -
  our percentage of students passing has increased
  every year!

30
Grade 3 English School Level Percent of Students
Passing
31
Grade 8 Writing School Level Percent of Students
Passing
32
Geometry School Level Percent of Students Passing
33
Grade 3 SOL Pass Rate
34
Grade 8 SOL Pass Rate
35
School Level Mean Percent Change in Passing
36
School Level Percent Passing Grade 5 English
Change from Year to Year
01-02 02-03 -
01-02 02-03
2002Stars
01-02 - 02-03 -
01-02 - 02-03
r -.37
37
School Level Percent Passing Algebra I Change
from Year to Year
r -.243
38
(No Transcript)
39
Statistically Significance Change 5th Grade
Math Classroom

• N 67 Classrooms
• X 450, SD 34.8, SE 4.25
• 95 Confidence Interval for a
• classroom score of 420 411.50 428.50
• Score of 430 significantly different from
• 420

40
Trend Analysis Classrooms with Standard Error
4
472
470
460
459
454
450
446
440
450
2004
2001
2003
2002
41
School Level Statistical Significance Testing
42
Two Significant Problems with Statistical
Significance

• 1) Interpreting results can be difficult.
• 2) Depends heavily on the number of students,
  classrooms, and schools.

43
(No Transcript)
44
Using Effect Size to Measure Change in SOL Scores

• Practical Significance
• Not linked to statistical significance a
  school may have a small effect size when the
  difference between groups is statistically
  significant

45
Formula for Calculating Effect Size
46
Effect Size Values Interpretation

• Effect Size
• .25 Small Effect
• .50 Moderate Effect
• .80 Large Effect

47
Using Effect Size to Show Practically Significant
Changes

• Division A Classrooms
• Grade 5 English 2004
• X 480.07
• SD 28

488.89 Small Effect 497.71
Moderate Effect 508.29 Large Effect
48
Using Effect Size to Measure School Change from
Year to Year

• School A
• Grade 3 Math
• 2000 1999

1.24

49
Using Effect Size to Measure School Change from
Year to Year

• School B
• Grade 3 Math
• 2000 1999

50
Using Effect Size to Measure School Change from
Year to Year

• School C
• Grade 3 Math
• 2000 1999

51
Using Effect Size to Measure School Change from
Year to Year

• School D
• Grade 3 Math
• 2000 1999

52
2003 School Level Effect Sizes
53
Some Suggestions

• Remember, tests dont measure much of what is
  really important about school.
• Focus on trends over time in full scale and
  reporting category scores taking into account
  sources of error and influence, especially cohort
  effects.
• Align with SOL cognitive skills, not just
  content, and give classroom tests that assess
  these skills.
• Use individual item analysis sparingly.
• Verify possible conclusions with additional
  evidence.
• Dont rank order (miniscule differences give
  wrong conclusions)

54
(No Transcript)
55
(No Transcript)
56
Use of SOL Test Data at the Student, School, and
District Level

• James H. McMillan
• Lauren M. Miller
• Virginia Commonwealth University

Presentation at the 2005 Annual Conference of the
Virginia Association of Test Directors
Slides available at http//www.soe.vcu.edu/merc/