Exploring Educational Standard Alignment: In Search of

1
Exploring Educational Standard AlignmentIn
Search of Relevance

• René Reitsma, Byron MarshallMichael Dalton,
  Martha Cyr
• Oregon State University
• Worcester Polytechnic Institute

2
Exploring Educational Standard AlignmentIn
Search of Relevance

• Problem aligning DL learning objects with
  educational standards
• Need tantalizing promise
• National Science Digital Library (NSDL) efforts
  accomplishments
• Early results show low Inter-Rater Reliability
  (IRR)
• Hypothesis low IRR is partially a methodological
  artifact
• Proposal multifactor concept of alignment
• Experiment 10-factor alignment model
• High IRR
• Four factor regression model (R.75) of overall
  alignment

3
Aligning DL Learning Objects with Educational
Standards

• Expanding DL learning resource base e.g.,
• National Science Digital Library (NSDL) 928
  collections
• K-12 TeachEngineering.org, TeachersDomain.org,
  Engineering is Elementary, etc.
• NSF-GK-12 program (ongoing).
• 84,500 math, science technology standards
  (changing frequently)

4
Curriculum Standard Alignment Efforts

• NSDL leadership
• JesCo
• Achievement Standards Network (ASN)
• Center for Natural Language Processing (CNLP)
• Curriculum Alignment Tool (CAT)
• Standard Alignment Tool (SAT)
• WGBH Teachers Domain standard alignment
  lexicon
• Others
• Academic Benchmarks
• AAAS/NSDL Strandmap server
• Etc.

5
NSDL-based Curriculum Alignment Services
6
How Good are these Alignments?
7
Low Inter-Rater Reliability (IRR)

• Devaul, H., Diekema, A.R., Ostwald, J. (2007)
• Bar-Ilan, J. Keenoy, K., Yaari, E., Levene, M.
  (2007)
• There is no average user, and even if the users
  have the same basic knowledge of a topic, they
  evaluate information in their own context
• Hypothesis Low IRR is partially a methodological
  artifact
• Alignment is a multifactor, multidimensional
  concept
• Learning objects may align with certain
  dimensions but not with others
• Levins, R, Lewontin, R.C. (1980) Abstraction
  becomes destructive when the abstract becomes
  reified so that the abstract descriptions are
  taken for descriptions of the actual objects

8
Dimensions of Alignment

• Seracevic, T. (2007) Relevance A Review of the
  Literature and a Framework for Thinking on the
  Notion in Information Science. Part II Nature
  and Manifestations of Relevance

9
Hypotheses

• H One-dimensional alignment/relevance IRR is
  partially a methodological artifact.
• H-1 At least some dimensional IRRs will be
  high(er)
• H-2 Dimensional IRR will vary
• H-3 Overall alignment/relevance IRR will be
  low, even when asked in the context of
  dimensional relevance testing.

10
Experiment
11
Experiment Cont.d

• 14 Subjects all familiar with the
  TeachEngineering system
• Two teaching tasks
• As a third grade Massachusetts teacher you are
  assigned to teach material related to the
  standard Relate earthquakes, volcanic activity,
  mountain building, and tectonic uplift to plate
  movements. You have two hours of class time to
  spend on instruction.
• Judge the alignment of three curricular objects
  (R-1 R-10, six-point Likert scale)

12
Results

• 91 IRR comparisons 10 alignment dimensions 6
  alignments
• H-1 IRRs are relatively high IRR-1 (binary)
  64-95
• H-2 IRR variability
• H-3 Overall relevance (R-10) among the weaker
  ones

13
How about Overall Relevance (R-10)?
14
MLR Model of Overall Relevance (R-10)

• R2 .75
• Overall alignment/relevance is meaningful as a
  complex variable.
• Some high IRR alignment dimensions do not
  contribute to overall alignment.

15
Conclusion

• K-12 educational DL content is expanding
  educational standard alignment is needed.
• Innovative and promising resources are available
  but reported IRR of assessment of alignments is
  low.
• Propose that Alignment is a complex concept
• Recognize alignment dimensions
• Experiment suggests that dimension-specific IRR
  will be (much) higher
• Overall alignment has a very specific
  interpretation.
• What do we need
• Continued assessment and IRR collection
• Collections making their assessment data
  available.
• Alignment methods that can assimilate evidence
  from the multiple dimensions that comprise
  alignment of a learning resource with a
  teaching standard.