Andrew C. Porter Vanderbilt University

1
Measuring the Content of Instruction Uses in
Research and Practice

• Andrew C. PorterVanderbilt University

2
Tools

• Teacher surveys of instruction
• Daily logs
• End-of-semester or end-of-year surveys
• Content analyses of
• Standards
• Tests
• Curriculum materials
• Alignment indicese.g, alignment between
  assessment and standards

3
Content Matrix
4
Vertical and Horizontal Alignment
Achievement
Instruction
District
Assessment
Standards
State
Assessment
Standards
5
Example Matrices to Measure Alignment
Cognitive Demand
Assessment
Standards
.3 0 .1
0 .1 0
0 .2 .1
0 .1 .1
.2 0 .1
0 .2 0
.1 .2 .1
0 0 .1
Topics
XAssessment Cell Proportions YStandards Cell
Proportions
Alignment Index 1 -
6
Alignment of Assessments with Standards7th-Grade
MathGoals Study
Average Within-State Alignment .40 Average
Between-State Alignment .39 Average
State-Test-to-NCTM Alignment .39
7
Alignment of Instruction with Assessment8th-Grade
MathSCASS Study
Average Within-State Alignment .22 Average
Between-State Alignment .23 Average
State-to-NAPE Alignment .39
8
Alignment of Instruction with Instruction
8th-Grade Math SCASS Study
Average Alignment .69
9
7th Grade Standards
State F
NCTM
State E
10
Quality of Data

• Response rates
• Interrater agreement for content analyses
• Validity of teacher self-report
• Explaining between-teacher variance in alignment
  to NAEP
• Predicting student achievement gains
• Note The need for a reform-neutral language

11
Uses of Tools Describing Instructional Practices

• Research
• Serve as dependent variable in teacher
  decision-making research
• Describe the implemented curriculum
• Measure implementation of new curricula
• Assess the validity of transcript studies
• Practice
• Inform teacher reflections on their own
  instructional practices Note Should not be
  used for teacher accountability.

12
Uses of Tools Describing Instructional Materials

• Research
• Research effects of textbooks on instruction
• Assess the breadth and depth of content in
  instructional materials
• Practice
• Build tests
• Write content standards
• Develop national, state, or district indicator
  systems

13
Uses of Indices of Alignment

• Research
• Serve as a control variable
• Serve as a dependent variable
• Serve as a descriptive variable
• Practice
• Align state tests to state standards
• Align instructional materials to standards or
  course frameworks

14
Increasing Validity and Value

• Getting the content language right
• Using time samples to describe instruction for an
  entire school year
• Replicating the finding that alignment predicts
  student achievement gains
• Identifying contexts in which teacher self-report
  on the content of instruction is more or less
  accurate
• Improving the level and consistency of interrater
  agreement in content analyses
• Understanding the distributional properties of
  the alignment statistics
• Building powerful professional development
  programs for data-based decision making
• Developing a content language for reading

15
Conclusions

• Much progress has been made in recognizing the
  importance of instructional content as a variable
  in education research.
• Some progress has been made in building tools for
  including content in education research.
• There have been several innovative uses of these
  new tools in both research and practice, and more
  are on the horizon.
• But there is much more work to be done.

16
(No Transcript)
17
7th-Grade Standards--Close View
Number Sense and Numeration
State E
State F
NCTM
18
7th-Grade Standards--Close View
Data Analysis and Probability
State E
State F
NCTM
19
Response Rates for Survey

• Eisenhower Longitudinal
• Wave 1 75
• Wave 2 74
• Wave 3 75
• Eisenhower Cross-Sectional 72
• Reform Up Close 75

20
Interrater Agreement

• Assessment Mean Range
• Goals Study .51 .77 to .34
• CCSSO Study .47 .60 to .37
• Standards
• Goals Study .48 .59 to .33
• Note In each study, there was one outlier
  rater.

21
Eisenhower Longitudinal Study
Longitudinal data on instruction alignment to
NAEP yielded

• 42 of variance explained by level (elementary,
  middle, high school) and subject
• 27 of variance explained by teachers in the same
  school
• 0 of variance explained by between school or
  between years

22
Alignment to Predict Achievement Gains
23
Index Intercorrelations