Measuring Changes in Teachers

1
Measuring Changes in Teachers Mathematics
Content Knowledge

• Dr. Amy Germuth
• Compass Consulting Group, LLC

2
Overview of Session

• Introductions
• Importance of Measuring Teacher Content Knowledge
• Measuring Teacher Content Knowledge
• Discussion

3
Importance of Measuring Teacher Content Knowledge

• US ED MSP Program Office Perspective
• The Government Performance Results Act (GPRA)
• A law passed in 1993 that requires federally
  funded agencies to develop and implement an
  accountability system based on performance
  measurement.

Importance of Measuring Teacher Content Knowledge
4

• Requires all federally funded program to
• Outline long-term and annual performance goals
  that include outcomes,
• Develop indicators to assess performance goals,
• Collect and analyze data on the indicators, and
• Report progress toward achieving performance
  goals based on GPRA data and data from grantees
  evaluations.
• MSP GPRA indicator related to content knowledge
• The percentage of MSP teachers who significantly
  increase their content knowledge, as reflected in
  project-level pre- and post-assessment

Importance of Measuring Teacher Content Knowledge
5

• GPRA and grantee evaluation findings are used by
  Congress to inform appropriations they want
  concrete evidence of results
• More uniform and higher quality data will enable
  the MSP Program office to more accurately and
  convincingly assess progress toward MSP
  performance goals
• This is challenging for measuring changes in
  teachers content knowledge

Importance of Measuring Teacher Content Knowledge
6

• MSP Grantee Perspective
• Has the potential to affect funding of the
  program at federal level
• US ED annual reporting requirement
• One of the more direct variables for individual
  partnerships to measure and demonstrate program
  outcomes

Importance of Measuring Teacher Content Knowledge
7
Measuring Teacher Content Knowledge

• Because of the variability in the activities
  projects are implementing to improve teacher
  content knowledge the use of one instrument is
  not a viable solution
• Some instruments are better than others
• Search for or develop an instrument that
• Is aligned with the content of your MSP
  professional development activity
• Has been piloted with teachers similar to those
  participating in your program

Measuring Teacher Content Knowledge
8

• Provides evidence of reliability or the extent to
  which an instrument yields consistent, stable,
  and uniform results over repeated administrations
  under the same conditions each time
• Provides evidence of validity or the extent to
  which 1) the instrument measures the skills it
  sets out to measure and 2) the inferences and
  actions made on the basis of the scores are
  appropriate and accurate

Figure obtained from the website
http//www.socialresearchmethods.net/KB/relval.ht
m
Discussion Question What instruments have
grantees been using? What are the properties of
those instruments?
Measuring Teacher Content Knowledge
9
Potential Tests

• Learning Mathematics for Teaching (LMT) Project
  University of Michigan
• Includes several surveys that measures teachers
  content knowledge for teaching mathematics in
  Number and Operations(K-6, 6-8), Patterns,
  Functions, and Algebra (K-6, 6-8), and Geometry
  (3-8)
• http//sitemaker.umich.edu/lmt/about
• Diagnostic Teacher Assessments in Mathematics and
  Science (D-TAMS) - University of Louisville
• Six versions of four different pre-post
  assessments designed to assess change in teacher
  content knowldge are available Number /
  Computation, Geometry / Measurement, Probability
  / Statistics, and Algebraic Ideas.
• http//louisville.edu/edu/mathscience/diagnostic.h
  tml
• Knowing Mathematics for Teaching Algebra (KAT)
  Michigan State University
• Instruments not yet available
• http//www.msu.edu//kat/index.htm

10

• Resources
• Overview of reliability and validity
• http//writing.colostate.edu/guides/research/relva
  l/index.cfm
• http//www.socialresearchmethods.net/KB/
• http//www.socialresearchmethods.net/tutorial/Colo
  si/lcolosi2.htm
• Crocker, L. Algina, J. (1986). Introduction to
  classical and modern test theory. Belmont, CA
  Wadsworth.
• Shultz, K. S. Whitney, D. W. (2005).
  Measurement theory in action Case studies and
  exercises. Thousand Oaks, CA Sage.
• Information on established instruments
• http//www.apa.org/science/faq-findtests.html
• http//www.unl.edu/buros/

Measuring Teacher Content Knowledge
11

• Information on developing instruments
• Crocker, L. Algina, J. (1986). Introduction to
  classical and modern test theory. Belmont, CA
  Wadsworth.
• Haladyna, T. M. (1999). Developing and validating
  multiple-choice test items (2nd ed.). Mahwah, NJ
  Erlbaum.
• Osterlind, S. J. (1998). Constructing test items
  Multiple-choice, constructed response,
  performance, and other formats. Boston, MA
  Kluwer.
• Shultz, K. S. Whitney, D. W. (2005).
  Measurement theory in action Case studies and
  exercises. Thousand Oaks, CA Sage.

Measuring Teacher Content Knowledge
12

• When To Administer Your Test
• Administer the pretest immediately prior to the
  MSP activity or college course
• Administer the posttest after the MSP activity
  and follow-up are complete, but within the same
  school year as the initial MSP activity

Measuring Teacher Content Knowledge
13

• How to Determine Whether Teachers Have Made
  Significant Gains
• To report for GPRA, US ED needs a way to tell
  whether the observed gains in teacher content
  knowledge between the pretest and the posttest
  are significant
• US ED is proposing that projects use a
  statistical test called a dependent t-test to
  determine whether teachers have made significant
  gains

Measuring Teacher Content Knowledge
14

• Proposed process
• The federal program office would supply grantees
  with an Excel spreadsheet with embedded formulas
  to do the needed calculations
• Grantees would enter into the spreadsheet the
  pretest scores and posttest scores for the
  teachers they test
• Grantees would use one spreadsheet per test
• The spreadsheet would calculate the needed
  statistics and produce a report for grantees
  showing the total number of teachers and the
  number who showed significant gains
• Grantees would report this information to US ED
  through the Annual Performance Report
• US ED would aggregate the information from all
  grantees and use it for GPRA reporting

Measuring Teacher Content Knowledge
15
Discussion

• Experiences developing content knowledge
  instruments
• Challenges encountered in measuring changes in
  content knowledge
• Questions about measuring changes in content
  knowledge