Content Analysis Workshop

1
Content Analysis Workshop
Minneapolis, MN Nov 7-9, 2007
Describing the Content of
Standards Assessments
John L. Smithson, Director, Measures of the
Enacted Curriculum Alissa Minor, Projects
Manager, Measures of the Enacted
Curriculum Wisconsin Center for Education
Research, University of Wisconsin-Madison johns_at_wc
er.wisc.edu
2
The Goal
To render quantitative descriptions of
instruction, standards, and assessments using a
common language in order to facilitate
comparisons and analyses of these three domains
of a standards-based approach to education reform
and their relationship to one another.
3
The Goal
To render quantitative descriptions of
instruction, standards, and assessments using a
common language in order to facilitate
comparisons and analyses of these three domains
of a standards-based approach to education reform
and their relationship to one another.
4
Needs Assessment
Content Descriptions
Curriculum Management
SEC Taxonomy
Monitoring Change
Program Evaluation
5
Uses of Content Analysis Results
Descriptive (Tile Charts and Content Maps)

• Visual, curriculum-based descriptions of
  Instructional Targets for teacher reflection,
  discussion and planning.

Analytic (Alignment)

• Predict student achievement gains
• Control for content to examine other factors
• As an outcome measure for change over time
• Examine alignment of Standards Assessments

6
To Describe Instructional Content
SEC utilizes a two-dimensional taxonomy based on
Topic by Cognitive Demand
7
The Content Matrix
8
adding levels of relative emphasis yields a 3-D
construct
9
(No Transcript)
10
Content Map Data Displays
11
(No Transcript)
12
To Facilitate Comparisons
Assessments
Curriculum
Alignment
Standards
13
Uses of Content Analysis Results
Descriptive (Tile Charts and Content Maps)

• Visual, curriculum-based descriptions of
  Instructional Targets for teacher reflection,
  discussion and planning.

Analytic (Alignment)

• Predict student achievement gains
• Control for content to examine other factors
• As an outcome measure for change over time
• Examine alignment of Standards Assessments

14
Alignment as a Quantity
Aligning Tests to Standards
0.27 (Avg. Alignment Test to Standard)
Range of Alignment Test to Standard)
0.14
0.40
0.00
0.50
0.25
0.75
1.00
State U Grade 8 Mathematics Alignment Test to
Standard (0.23)
(Based on results for 10 states, across grades 4,
6 and 8 SEC Collaborative 2003)
15
Instructional Alignment
Instruction to Standards Assessments
Fine Grain
Standards
0.05
0.29
0.17
Avg.
Min.
Max.
1.00
0.00
0.50
0.75
0.25
0.03
0.31
0.19
Avg.
Min.
Max.
Assessments
Based upon results for 168 teachers, across 3
states MSP PD Study 2004
16
Explaining variation in student learning gains
17
Alignment Analyses for School Improvement
Using alignment as an outcome measure
Alignment Index Instruction to
Standards Mathematics Across 4 Districts
Counts
Treatment 99 Control 124 Leaders 16
(Measuring change in alignment over time)
18
Content Analysis Procedures
Exploring the Dimensions of Content
But first.
Lets take a 10 minute break!
19
The Two Dimensions Of Content What students
should know Topics And Be Able to Do
Expectations for student performance
20
Describing the Cognitive Domain
How Many Categories?
3
SCASS Science
4
DOK (Webb)
SEC
5
6
Blooms
21
Dimensions of Knowing Inquiry
Acquire
Use
Extend
(From Dimensions of Knowing and Inquiring about
Science, State Collaborative on Assessments
Student Standards Science Project, Council of
Chief State School Officers, 1997)
22
Depth of Knowledge
Level
1 2 3 4
Recall
Skill/Concept
Strategic Thinking
Extended Thinking
Webb, N. 1999. Alignment of Science and
Mathematics Standards in Four State. NISE
Research Monograph 18. MadisonWisconsin Center
for Education Research.
23
Exploring Cognitive Demand
Acquire
Use
Extend
Skill/Concept
Recall
Strategic Thinking
Extended Thinking
24
Cognitive Demand (or Expectations for Student
Performance)
Acquire
Use
Extend
Skill/Concept
Recall
Strategic Thinking
Extended Thinking
Memorize
Perform Procedures
Demonstrate Understanding
Conjecture, Generalize Prove
Solve non- routine/ make connections
Memorize
Conduct Investigations
Communicate Understanding
Analyze Information
Apply concepts /make connections
Recall
Perform Procedures
Analyze/ Investigate
Evaluate
Generate /Demonstrate
25
Cognitive Demand (or Expectations for Student
Performance)
Acquire
Use
Extend
Skill/Concept
Recall
Strategic Thinking
Extended Thinking
Memorize
Perform Procedures
Demonstrate Understanding
Analyze Information
Evaluate/Apply
Recall
Understanding
Application
Analyze
Evaluate
Create
26
Exploring Cognitive Demand
CgD Immersion Activity

• Organize into Groups/Tables
• Each Table w/ CgD Pie
• Each Person w/ Cgd Descriptors

Step 1 Place CgD cards on Pie Slices face-down
Step 2 Turn cards over ID agreements e.g. 2
cards w/ same descriptor in same slice if Group
Agrees ... discuss key words if not Discuss
operational definition to distinguish
Step 3 Discuss disagreements if consensus
reached put in envelope / if not, set aside
27
Content Analysis Materials

• Cognitive Demand List
• Topics Lists
• Comments Suggestions Worksheet
• Coding Forms
• Documents to be analyzed

28
Cognitive Demand Lists

• Five categories of cognitive demand
• Slightly different for each subject
• Each category is defined by a list of
  descriptors
• The list of descriptors are not exhaustive
• Each category stands on its own
• Each category has an associated letter (B-F)

29
(No Transcript)
30
(No Transcript)
31
(No Transcript)
32
Dimension A Content Topics
Topics List (In your packet of material)
Organized at two levels Content Areas (16
for Mathematics) (27 for Science) (14 for
ELAR) Topics (identified by number) (182
Mathematics Topics) (211 Science Topics) (114
ELAR Topics)
Plus non-specific other
33
Topics Lists

• Topics Lists
• Mathematics
• Science
• English Language Arts Reading
• Cover grades K-12
• Organized into Content Areas
• Topics Content Areas have an associated

34
Comments Suggestion Worksheet

• One for each reviewer - more available
• Use to
• Record coding conventions/decision rules
• Suggest/identify additional topics not listed
• Suggest/identify additional CGD descriptors
• Provide other comments suggestions
• Be sure to turn in at end of workshop (and with
  mail-in materials, as necessary).

35
Coding Forms

• Assessment Coding Forms
• Benchmark Coding Forms
• Each is used to record content descriptions
• Each content description consists of
• A topic number
• A cognitive demand category letter

36
Assessment Coding Forms
37
Standards Coding Forms
38
Standards Coding Forms
39
(No Transcript)
40
(No Transcript)
41
(No Transcript)
42
(No Transcript)
43
(No Transcript)
44
Practice Coding Exercise
Content Analyzing Assessments (Three code maximum)
45
Practice Coding Exercise
Content Analyzing Standards (Six code maximum)
46
The Content Analysis Process
Coding Teams of 4-5 Content Experts
Independent Coding by each Analyst w/ Group
Discussion
Should not be necessary to discuss every item
select by team
Goal for Process Generalizability not
Inter-rater Reliability
Pick-up and return documents / coding sheets to
Alissa
Sign return to Alissa non-disclosure forms
47
Content Analysis Workshop
The intended curriculum State content
standardsWhat students should learn
The assessed curriculum State (and other)
assessmentstested learning
The learned curriculum Student outcomes based
on school learning
The enacted curriculum What teachers teach