Reactions to Presentation by the Committee on Test Design for K12 Science Achievement

1
Reactions to Presentation by the Committee on
Test Design for K-12 Science Achievement

• Zalman Usiskin
• University of Chicago

2
Visible faces of the tetrahedron

• Response to the address of Jere Confrey that was
  originally scheduled
• Response to this mornings presentations by
  Meryl, Gerry, and Joe
• Response reflecting the interests of the K-12
  science and mathematics curriculum development
  community

3
What is the purpose of testing in K-12?

• to evaluate students
• to improve students
• to evaluate teachers
• to evaluate the curriculum teachers teach
• to evaluate the curriculum the developers wrote
• to evaluate schools
• to ship out those students, teachers, and/or
  schools that do not shape up

4
A telling requirement States must have
challenging academic content standards in science.
However, no standards are considered
challenging unless lots of people do not meet
them!
5

• Both science and mathematics have progressed by
  asking both what happens or has happened under
  certain conditions and why these events have
  happened.

6
No Body Left Behind

• Base-line measure on the Terol Nova test
• Total score 267
• Higher-order Terol 207
• Lower-order Terol 35
• A total score of over 200 is failing. So is a
  higher-order score of over 150.
• Recommendation Give my body the Zocor
  curriculum, proven to be successful with
  scientifically-based research.

7
No Body Left Behind

• Performance on the Terol Nova test after one
  month
• Total score 198
• Higher-order Terol 151
• Lower-order Terol 37
• Much improvement, but barely passing.
• Recommendation Continue the Zocor curriculum,
  and take another test after 3 more months.

8
No Body Left Behind

• Performance on the Terol Nova test after four
  months
• Total score 151 (orig. 267)
• Higher-order Terol 101 (orig. 207)
• Lower-order Terol 38 (orig. 35)
• Extraordinary results. Body is no longer on the
  list of failing bodies!
• Conclusion The Zocor curriculum works.

9
A little difficulty with this conclusion...

• I didnt use the Zocor curriculum. I made up my
  own curriculum
• so the results were not due to the Zocor
  curriculum, nor were they due to a placebo. They
  were due to a treatment that was not part of the
  study, and any conclusion about the Zocor
  curriculum is faulty.

10
Research in Education and Medicine

• Both deal with subjects in potentially failing
  situations.
• Both have treatments to be administered.
• But in education both the administers of the
  treatment and the subjects can consciously impose
  their will on the treatment.
• And in education market research is critical.

11
If we are testing curriculum,what are we testing?

• The curriculum as it is likely to be used?
• The curriculum as the developers desire it to be
  used?
• The teachers interpretations of the curriculum?
• The students responses to the curriculum?
• The tests coverage of the curriculum?

12

• Every test of a student is also a test of
• the extent to which the material on the test
  was in the curriculum,
• the extent to which the material in the
  curriculum was taught by the teacher,
• the extent to which the student learned what
  was taught by the teacher
• the motivation of the student towards the test.

13
Whats a big idea?
Whats the big idea?

• About a quarter of people in the U.S. do not
  trust science, do not believe in some of the
  basic ideas (evolution, geology, astro-physics,
  etc.) of science, and believe in astrology and
  other pseudosciences.
• Any testing program to improve science education
  for all Americans has to address (if not attack)
  this ignorance.

14
Commenting on design of studies rather than
evaluation of studies

• Similarity a study cannot be evaluated without
  examining its design.
• Difference a study can have an impeccable design
  but be without any interesting questions.