Aligning Science Assessment to Content Standards

1
Aligning Science Assessment to Content Standards

• George DeBoer, Arhonda Gogos, Cari Herrmann
  Abell, Kristen Lennon, An Michiels, Tom Regan, Jo
  Ellen Roseman,
• Paula Wilson
• Center for Curriculum Materials in Science
• Knowledge Sharing Institute
• Ann Arbor, Michigan
• July 10-12, 2006
• This work is funded by the National Science
  Foundation
• ESI 0352473

2
Thanks to

• Abigail Burrows for organizing the pilot testing
  with schools.
• Ed Krafsur for developing the assessment data
  base.
• Brian Sweeney for developing illustrations for
  test items.

3
Strand 6 Part I

• Examining the Project 2061 Criteria for Aligning
  Middle School Assessment Items to Learning Goals

4
Aligning Student Assessment to Content Standards

• What We Are Doing Project Background
• Creating a bank of middle and early high school
  science assessment items that are precisely
  aligned with national content standards
• Providing resources to support the creation and
  use of assessment items aligned to content
  standards
• Developing a data base for these resources and a
  user interface to access the resources
• In this session, we will focus on the criteria we
  use for judging alignment of assessment items to
  content standards.

5
Resources We Will Provide

• Clarifications of the content standards
  (elaboration, boundary setting, i.e., whats in
  and whats out). To add precision to the
  alignment of assessment items.
• Summaries of research on student learning
  (misconceptions and other ideas students hold)
  related to the ideas in the content standards. To
  serve as distractors in assessment items.
• Assessment maps (which include prerequisite
  ideas, related ideas, ideas that come later in
  the learning trajectory). Useful for developing
  test instruments on a specific topic. Also useful
  in item development for deciding what knowledge
  is reasonable to expect students to have (e.g.,
  bedrock).

6
List of Topics

• Atoms, Molecules and States of Matter
• Substances, Chemical Reactions and Conservation
• Processes that shape the Earth / Plate Tectonics
• Weather and Climate
• Solar System
• Energy Transformations
• Force and Motion
• Forces of Nature
• Sight and Vision
• Mathematics Summarizing Data
• Mathematics Relationships among Variables

7
List of Topics, Continued

1. Basic Functions in Humans
2. Cells and Proteins
3. Evolution and Natural Selection
4. Interdependence, Diversity and Survival
5. Matter and Energy Transformations in Living
   Systems
6. Sexual Reproduction, Genes and Heredity
7. Cross-cutting Themes Models
8. Nature of Science Claims of Causal Relationships
9. Nature of Science Inductive Reasoning
10. Nature of Science Empirical Validation of Ideas
    about the World
11. Nature of Science Uncertainty and Durability

8
Examples of

• Clarification statements
• Summaries of research on student learning
• Assessment maps
• How each is used in the item development work.

9
Idea B All atoms are extremely small (from BSL
4D/M1a).

• Students are expected to know that atoms are much
  smaller than very small items with which they are
  familiar, such as dust, blood cells, plant cells,
  and microorganisms, all of which are made up of
  atoms. Students should know that the atoms are
  so small that many millions of them make up these
  small items with which they are familiar. They
  should know that this is true for all atoms. The
  comparison with very small objects can be used to
  test students qualitative understanding of the
  size of atoms in relation to these objects.
  Students will not, however, be expected to know
  the actual size of atoms.

10
Student Misconceptions Related to the Size of
Atoms

• Atoms and/or molecules are similar in size to
  cells, dust, or bacteria (Lee et al., 1993
  Nakhleh et al., 1999 Nakhleh et al., 2005).
• Atoms and/or molecules can be seen with
  magnifying lenses or optical microscopes
  (Griffiths et al., 1992 Lee et al., 1993).

11
(No Transcript)
12
Steps in the Item Development Procedure

• Select a set of benchmarks and standards to
  define the boundaries of a topic
• Tease apart the benchmarks and standards into a
  set of key ideas
• Create an assessment map showing how the key
  ideas build on each other conceptually
• Review the research on student learning to
  identify ideas students may have about the ideas
• Design items
• using student misconceptions as distractors
• using the assessment analysis criteria
• following a list of design specifications

13
Steps in the Item Development Procedure, cont

1. Use open-ended interviewing to supplement
   published research on student learning
2. Use mini item camps to get feedback on items
   from staff
3. Revise items
4. Pilot test items and conduct think aloud
   interviews
5. Analyze pilot test data
6. Revise items
7. Conduct formal reviews of approximately 25 items
   using the assessment analysis criteria
8. Revise items
9. Conduct national field test of items

14
Demonstration of the Database and User Interface

• Items
• Misconception List
• Topics, key ideas, clarifications
• Assessment Maps
• Item Specifications

15

• The Project 2061 Assessment Analysis Procedure

16
There are six parts to the analysis procedure

1. Exploring the Learning Goal
2. Determining Content Alignment
3. Determining Whether the Task Accurately Reveals
   What Students do or do not Know
4. Considering the Tasks Cost Effectiveness
5. Suggesting Revisions
6. Assessment Item Rating Form (not included in this
   version)

17
Reviewers use the following materials

• Assessment Items
• The content standard that is being targeted
• Clarification statements
• Lists of common student misconceptions and other
  ideas students may have.
• Results of student interviews or field test
  results if available

18
I. Exploration Phase

• Determining the alignment of an assessment task
  to a learning goal requires a precise
  understanding of the meaning of the learning goal
  and what knowledge and skills are needed to
  successfully complete the task.

19
A. The Learning Goal

1. Reviewers carefully read the clarification
   statement written for the targeted learning goal
   (content standard or benchmark).
2. Reviewers examine the list of misconceptions
   related to the targeted learning goal.

20
B. The Assessment Task

• Reviewers
• attempt to complete the task themselves.
• list the knowledge and skill needed to
  successfully complete the task.
• consider if there are different strategies that
  can be used to successfully complete the task.
• consider which misconceptions might affect
  student answers.

21

• II. Determining the Content Alignment between the
  Learning Goal and the Assessment Task

22
A. Necessity

1. To be content aligned, knowledge of the ideas
   described in the learning goal or the
   clarification statement, or knowledge that
   certain commonly held misconceptions are not
   true, must be needed to evaluate each of the
   answer choices.

23
Reviewers are told

• If the knowledge in the learning goal is not
  needed to decide if the answer choices are
  correct or incorrect, explain how the answer
  choices can be evaluated using other knowledge.

24
Applying the Necessity Criterion

• Which of the following is the smallest?
• A.  An atom
• B.  A bacterium
• C.  The width of a hair
• D.  A cell in your body

25
Idea B All atoms are extremely small (from BSL
4D/M1a).

• Students are expected to know that atoms are much
  smaller than very small items with which they are
  familiar, such as dust, blood cells, plant cells,
  and microorganisms, all of which are made up of
  atoms. Students should know that the atoms are
  so small that many millions of them make up these
  small items with which they are familiar. They
  should know that this is true for all atoms. The
  comparison with very small objects can be used to
  test students qualitative understanding of the
  size of atoms in relation to these objects.
  Students will not, however, be expected to know
  the actual size of atoms.

26
Applying the Necessity Criterion

• The knowledge in the learning goal is needed to
  evaluate each answer choice.

27
An example of an item for which the targeted
knowledge is not needed

• Targeted Idea Substances may react chemically
  in characteristic ways with other substances to
  form new substances with different characteristic
  properties (based on NSES 5-8BA2a).

28

• Which of the following is an example of a
  chemical reaction?
• A piece of metal hammered into a tree.
• A pot of water being heated and the water
  evaporates.
• A spoonful of salt dissolving in a glass of
  water.
• An iron railing developing an orange, powdery
  surface after standing in air.

29
Applying the Necessity Criterion

• The knowledge in the learning goal is not needed.
• Answer choice D, the correct answer, is a
  specific instance of a general principle (SIGP).
  The student can get the item correct by knowing
  that rusting is a chemical reaction without
  knowing the general principle that new substances
  are formed that have different characteristic
  properties.

30
B. Sufficiency

• To be content aligned, knowledge of the ideas
  described in the learning goal or the
  clarification statement, or knowledge that
  certain commonly held misconceptions are not
  true, must be all that is needed to evaluate
  each of the answer choices. Students should not
  need any additional science knowledge.

31
Reviewers are told

• If the knowledge in the learning goal is not
  enough to evaluate each of the answer choices,
  indicate what additional knowledge is needed.
  (Do not include as additional knowledge those
  things that can be assumed as general knowledge
  and ability of students this age.)
• An example of additional knowledge might include
  science or mathematics terminology that students
  are not expected to know.

32
Applying the Sufficiency Criterion

• Which of the following is the smallest?
• A.  An atom
• B.  A bacterium (clarification statement says
  microorganism)
• C.  The width of a hair
• D.  A cell in your body

33
Applying the Sufficiency Criterion

• The sufficiency criterion is not met. Students
  need to know the term bacterium, which is
  additional knowledge. Although a listed
  misconception includes the word bacteria, in
  pilot testing, 25 of 193 students indicated that
  they did not know what a bacterium was (even
  though most knew what bacteria were). The item
  should say microorganism or bacteria to match
  the clarification statement and/or misconception
  list.

34
Applying the Sufficiency Criterion

• Approximately how many carbon atoms placed next
  to each other would it take to make a line that
  would cross this dot ? ?
• A.  6
• B.  600
• C.  6000
• D.  6,000,000
• Note This item assumes a 1mm dot and a diameter
  of 1.5Å for a carbon atom.

35
Applying the Sufficiency Criterion

• The sufficiency criterion is met. Students need
  to know that like the other small things
  mentioned in the clarification statement, e.g.,
  dust, plant cells, blood cells, and
  microorganisms, this small visible dot is also
  made of millions of atoms.
• Note This item assumes a 1mm dot and a diameter
  of 1.5Å for a carbon atom.

36
Idea B All atoms are extremely small (from BSL
4D/M1a). (Not included in the workshop packet.)

• Students are expected to know that atoms are much
  smaller than very small items with which they are
  familiar, such as dust, blood cells, plant cells,
  and microorganisms, all of which are made up of
  atoms. Students should know that the atoms are
  so small that many millions of them make up these
  small items with which they are familiar. They
  should know that this is true for all atoms. The
  comparison with very small objects can be used to
  test students qualitative understanding of the
  size of atoms in relation to these objects.
  Students will not, however, be expected to know
  the actual size of atoms nor the
  order-of-magnitude relationships to other
  objects.

37
III. Determining Whether the Task Accurately
Reveals What Students Do and Do Not Know

• Its a validity issue. Students should choose
  the correct answer when they know the idea and
  they should choose an incorrect answer when they
  do not know the idea.
• Getting rid of factors not related to the
  knowledge being measured (construct irrelevant
  factors)
• Reducing false negatives and false positives

38
A. Comprehensibility

• 1. It is not clear what question is being
  asked. Explain.
• 2. The task uses unfamiliar general vocabulary
  that is not clearly defined. List potentially
  unfamiliar vocabulary and explain. (Note This is
  referring to general language usage, not
  technical scientific or mathematical terminology,
  which is addressed under Sufficiency.)
• The task uses unnecessarily complex sentence
  structure or ambiguous punctuation that makes the
  task difficult to comprehend when plain language
  could have been used. Explain.
• (Note Rebecca Kopriva, C-SAVE, Maryland.)

39
Comprehensibility Continued

• The task uses words and phrases that have
  unclear, confusing, or ambiguous meanings. This
  may include commonly used words that have special
  meaning in the context of science. For example
  the word finding could be unfamiliar to
  students when referring to a scientific
  finding. Note all places where words, both
  general and scientific) do not have clear and
  straightforward meanings.
• There is inaccurate information (including what
  is in the diagrams and data tables) that may be
  confusing to students who have a correct
  understanding of the science. Explain.
• The diagrams, graphs, and data tables may not be
  clear or comprehensible. (For example, they may
  include extraneous information, inaccurate or
  incomplete labeling, inappropriate size or
  relative size of objects, etc.) Explain.
• Other. Provide a brief explanation.

40
Comprehensibility

• An item with comprehensibility issues.

41
Most sidewalks made out of concrete have cracks
every few yards as shown in the diagram below. 
These are called expansion joints as labeled in
the diagram below.  What happens to the width of
the cracks during a hot day in the summer and
why?

• A.  The cracks get wider because the concrete
  shrinks.
• B.  The cracks get wider because the concrete
  gets softer.
• C.  The cracks get narrower because the concrete
  expands.
• D.  The cracks get narrower because the ground
  underneath the sidewalk shrinks.

42
Most sidewalks made out of solid concrete have
spaces between the sections as shown in the
diagram below.  What happens to the width of the
spaces during a hot day in the summer and why?
 

• A.  The spaces get wider because the concrete
  shrinks.
• B.  The spaces get narrower because the concrete
  expands.
• C.  The spaces get stay the same because the
  concrete does not shrink or expand.
• D.  Some spaces get narrower and some get wider
  because some concrete expands and some concrete
  shrinks

.
43
B. Appropriateness of Task Context

• a. The context may be unfamiliar to most
  students. Explain.
• b. The context may advantage or disadvantage one
  group of students because of their interest or
  familiarity with the context. Explain.
• c. The context is complicated and not easy to
  understand so that students might have to spend a
  lot of time trying to figure out what the context
  means. Explain.

44
Appropriateness of Task Context, Continued

• The information and quantities that are used are
  not reasonable or believable. Explain.
• e. The context does not accurately represent
  scientific or mathematical realities or, if
  idealizations are involved, it is not made clear
  to students that it is an idealized situation.
  Explain.
• f. Other. Explain.

45
C. Resistance to Test-Wiseness

• 1. Some of the distractors are not plausible.
  Explain.
• 2. One of the answer choices differs in length
  or contains a different amount of detail from the
  other answer choices. Explain.
• 3. One of the answer choices is qualified
  differently from the other answer choices, using
  words such as usually or sometimes, or an
  answer choice uses different units of
  measurement. Explain.
• 4. The use of logical opposites may lead
  students to eliminate answer choices. Explain.

46
Resistance to Test-Wiseness, Continued

• One of the answer choices contains vocabulary at
  a different level of difficulty from the other
  answer choices that may make it sound more
  scientific. Explain.
• 6. The language in one of the answer choices
  mirrors the language in the stem. Explain.
• 7. There are other test-taking strategies that
  may be used in responding to this task. Explain

47
An item with test-wiseness issues

• This item is targeted to Idea A from Matter and
  Energy Transformations in Living Systems
• Food is a source of molecules that serve as
  fuel and building material for all organisms.
• Is the oxygen that animals breathe a kind of
  food?
• Yes, because oxygen enters the body. M-A2
• Yes, because all animals need oxygen to survive.
  M-A3
• No, because animals do not get energy from
  oxygen. From clarification of Idea A.
• No, because oxygen can enter an animals body
  through its nose. M-A1, M-A2.

48
Misconceptions and other Ideas students may have
Matter and Energy Transformations Idea A

1. Many children associate the word food with what
   they identify as being edible (Driver, 1984
   Driver, Squires, Rushworth, Wood-Robinson,
   1994 Lee Diong, 1999).
2. Students see food as substances (water, air,
   minerals, etc.) that organisms take directly in
   from their environment (Anderson, Sheldon,
   Dubay, 1990 Simpson Arnold, 1982).
3. Some students think that food is what is needed
   to keep animals and plants alive (Driver et al.,
   1994).

49
Analyzing test-wiseness issues

• Conclusion Answer choice D (No, because oxygen
  can enter an animals body through its nose), is
  not a plausible explanation for why oxygen is not
  food. The answer choice is likely to be
  eliminated because of its implausibility, which
  is one of the factors (C1) used in assessing
  test-wiseness. (In pilot testing, 5 of 29
  students selected this, thinking that the point
  of entry is what determines if something is food.
  Many others questioned how the nose is relevant
  in a question about food.)
• The answer choice could be improved by changing
  it to say that oxygen is not food because it is
  not edible (M-A1) or because it does not enter
  through an animals mouth.

50
IV. Considering the Tasks Cost Effectiveness

• Does the task require an inordinate amount of
  time to complete? Ask whether the time needed
  for students to read the question, make
  calculations, interpret a data table, or read a
  graph is warranted. Provide a brief explanation
  of why the task is not cost effective and how the
  same information might be elicited more
  efficiently.

51
V. Suggesting Revisions

• Based on your analysis of the task, make your
  suggested revisions or indicate if you think the
  task should be eliminated from consideration.

52
Begin Content-Focused Activities
53
Aligning Science Assessment to Content Standards

• George DeBoer, Arhonda Gogos, Cari Herrmann
  Abell, Kristen Lennon, An Michiels, Tom Regan, Jo
  Ellen Roseman,
• Paula Wilson
• Center for Curriculum Materials in Science
• Knowledge Sharing Institute
• Ann Arbor, Michigan
• July 10-12, 2006
• This work is funded by the National Science
  Foundation
• ESI 0352473

54
Thanks to

• Abigail Burrows for organizing the pilot testing
  with schools.
• Ed Krafsur for developing the assessment data
  base.
• Brian Sweeney for developing illustrations for
  test items.

55
Strand 6 Part II

• Using Student Data to Inform the Design of
  Assessment Items in Middle School Science

56
Steps in the Item Development Process

• Select a set of benchmarks and standards to
  define the boundaries of a topic
• Tease apart the benchmarks and standards into a
  set of key ideas
• Create an assessment map showing how the key
  ideas build on each other conceptually
• Review the research on student learning to
  identify ideas students may have about the
  content
• Design items
• using student misconceptions as distractors
• following the assessment analysis criteria
• following a list of design specifications

57
Steps in the Item Development Process, cont

1. Use open-ended interviewing to supplement
   published research on student learning
2. Use mini item camps to get feedback on items
   from staff
3. Revise items
4. Pilot test items and conduct think aloud
   interviews
5. Analyze pilot test data
6. Revise items
7. Conduct formal reviews of approximately 25 items
   using the assessment analysis criteria
8. Revise items
9. Conduct national field test of items

58
Using Pilot Testing and Think Aloud Interviews

1. We use pilot testing and interviewing to probe
   student thinking about the targeted ideas and the
   test items.
2. We compare student answer choices to their
   explanations.
3. When answer selections and explanations dont
   match, we look for problems with the item that
   could produce these mismatches.

59
Interviewing Snapshot for 2005 and 2006

• 7 schools (urban, suburban) 200 interviews
• Free and reduced lunch ranged from 2 to 78
• Some think-aloud some open-ended
• Open-ended interviews were used to inform item
  development. Student comments helped in the
  writing of distractors.
• All interviews done by the item writers.

60
Think-Aloud Interview Procedure

• Please read the question aloud, think about the
  answer choices, and circle the best one. Feel
  free to write down anything on the test paper
  that helps you to answer the question.
• Could you tell me in your own words what the
  question is asking?
• Why did you choose the answer you chose?
• Were there other answer choices that you almost
  chose? (Why?)
• Continued

61

1. Were there any answer choices that you did not
   even consider? (Why?)
2. Was there an answer choice you were expecting to
   see but did not? What was it?
3. Were there any words or diagrams you did not
   really understand or situations that made the
   question confusing? Do you think anything would
   be confusing to your classmates?
4. Are you familiar with the situation that is
   presented in the question?
5. Where did you learn about the topic in this
   question? Have you seen a question like this
   before?

62
Getting permission to conduct interviews

• We inform the school administrators that
• The students responses will be used only to
  judge the quality of the test questions and will
  NOT be used as a measure of students knowledge
  or ability, instructional quality, or the quality
  of the school.  
• The students are coded to protect their identity.
  
• The parents are asked to sign a permission
  letter.
• Some school districts require Institutional
  Review Board (IRB) approval.

63
We provide incentives

1. The revised versions of the items are made
   available to the teachers and administrators. 
2. We provide a report on what we learned regarding
   student knowledge of the targeted ideas and
   misconceptions students may have.
3. We offer a workshop on developing assessment
   items aligned to content standards to
   volunteering teachers and/or participating
   schools.
4. As a token of our appreciation, students receive
   a gift certificate to Borders bookstore for each
   interview.

64
Limitations

1. Considerable time requirement
2. Small student sample
3. Hard to get access to students

65
Piloting snapshot

• Total of 112 classrooms across 5 content areas.
• Atoms and Molecules 726 students
• Force and Motion 610 students
• Flow of Matter and Energy 312 students
• Plate Tectonics 568 students
• Control of Variables 462 students

66
Pilot Test Schools District-level Demographics

• Northeast Suburban/Small Town. Middle School and
  High School.
• 40 White, 48 African American, 8 Hispanic
  25 Free and Reduced Lunch.
• 2. Northeast Suburban. Middle School. 95
  White 10 Free and Reduced Lunch.
• 3. Northeast Rural. (K-8). 98White 49 Free
  and Reduced Lunch.
• 4. Southern Small Town. Middle School (6-8) 70
  White, 24 African American 33 Economically
  Disadvantaged.
• 5. Southwest Small Town. Middle School (7-8).
  95 Hispanic, 95 Free and Reduced Lunch.

67
Teacher Feedback Questionnaire

• Does the class have a special designation (e.g.,
  honors, AP, ELL, special needs, etc.)? Please
  describe.
• Please note the approximate number of students in
  this class with Individualized Education Plans
  (IEPs).
• Approximately how much exposure have your
  students had to the topics hat these assessment
  items test?
• How long did it take to administer the test?
• Was it difficult for the students to understand
  the instructions? Please document on any
  difficulties they had.
• Please add any comments or suggestions you may
  have.

68
Pilot-test questions

• Is there anything about this test question that
  was confusing? Explain.
• Circle any words on the test question you dont
  understand or arent familiar with.
• Is answer choice A correct? Yes No Not Sure
• Is answer choice B correct? Yes No Not Sure
  
• Is answer choice C correct? Yes No Not Sure
  
• Is answer choice D correct? Yes No Not Sure
  
• For items 3-6, students are asked to explain why
  an answer choice is correct or not.

69
Pilot-Test Questions, Continued

• Did you guess when you answered the test
  question? Yes No
• Please suggest additional answer choices that
  could be used.
• Was the picture or graph helpful? If there was no
  picture or graph, would you like to see one?
• Have you studied this topic in school?
  Yes No Not Sure
• Have you learned about it somewhere else? Yes
  No Not Sure
• (TV, museum visit, etc)? Where?

70
Results of Teacher Feedback

• Test took 45min. to an hour to complete on
  average.
• Students sometimes had difficulty providing an
  explanation for each answer choicecognitively
  and motivationally. Not used to doing that.
• Only a very small number of students did not take
  the task seriously for a variety of reasonsend
  of the year, not graded, etc. Most were very
  cooperative.
• Students with learning disabilities expressed
  more difficulty.
• The unfamiliar format was a challenge to some.
• Teachers appreciated the depth of understanding
  that was expected.

71
Examples

• What we learn from pilot testing

72
Targeted Idea Substances may react chemically
in characteristic ways with other substances to
form new substances with different characteristic
properties (based on NSES 5-8BA2a).

• Which of the following is an example of a
  chemical reaction?
• A piece of metal hammered into a tree.
• A pot of water being heated and the water
  evaporates.
• A spoonful of salt dissolving in a glass of
  water.
• An iron railing developing an orange, powdery
  surface after standing in air.

73
Students who Selected Each Answer Choice
A (metal) B (evaporation) C (dissolving) D (rusting) Not sure Total
0 14 18 43 1 76
0 18.4 23.7 56.6 1.3 100
74
Results of piloting

• Only 5 of the 43 students who chose the correct
  answer D said that a new substance formed.
  Approximately half of the 43 students who chose D
  said they recognized it as an example of rusting
  or oxidation. Maybe these students know that
  rusting is a chemical reaction that produces new
  substances with different properties, but they
  may also know rusting only as a specific instance
  of a chemical reaction without knowing that
  chemical reactions involve the formation of a new
  substance.
• None of the students chose answer choice A,
  suggesting that hammering a piece of metal into a
  tree is not a plausible answer choice. Similar
  results were found during interviews.
• A significant number of students (42.1) chose
  either B or C. This supports other research that
  shows that students hold the idea that phase
  change and/or dissolving are chemical reactions.

75
Suggested revisions

• Replace A with a more plausible distractor such
  as Sand being removed from sea water by
  filtration.
• Replace D with a reaction that students are not
  so familiar with, for example, a white solid
  forming when two clear liquids are mixed
  together.

76
Targeted Idea

• Organisms use molecules from food to make complex
  molecules that become part of their body
  structures.

77
When a baby chick develops inside an egg, the
yolk in the egg is its only source of food. As
the chick grows, the yolk becomes smaller.  Why
does the yolk become smaller?

• A. The yolk enters the chick, but none of the
  yolk becomes part of the chick.
• B. The yolk is broken down into simpler
  substances, some of which become part of the
  chick.
• C. The yolk is completely turned into energy for
  the chick.
• D. The yolk gets smaller to make room for the
  growing chick.

78
Students who Selected Each Answer Choice
A (not part of) B (simpler substances) C (turned into energy) D (makes room for chick) Not sure Total
8 16 23 20 7 74
11 22 31 27 9 100
79
Results of piloting

• 6 students commented that they did not understand
  the phrase simpler substance in answer choice
  B.
• Only 8 of the 16 students who chose the correct
  answer B explained that yolk is broken down to
  provide building material that becomes
  incorporated into the body of the chick. The rest
  of the students indicated that the yolk is needed
  for the chick to grow or to become bigger. It
  is not clear that these students understand the
  idea that is being assessed, i.e., that food is
  broken down into smaller molecules that provide
  building material for the chick, which become
  part of the body structures of the chick.
• One of the students who selected answer choice A
  commented that Just like humans, pieces of food
  do not become part of us. This student might
  have a correct molecular understanding of how
  food is made part of body structures but got the
  question wrong because of the students focus on
  the yolk as being broken down into pieces of
  food.

80
Suggested revisions

• Change answer choice A to read The yolk is
  broken down into simpler molecules but none of
  the atoms of these simpler molecules become part
  of the chick.
• Change answer choice B to read The yolk is
  broken down into simpler molecules that are used
  to make the body structures of the chick.

81
The expansion of alcohol in a thermometer
AM42-4 The level of colored alcohol in a
thermometer rises when the thermometer is placed
in hot water.  Why does the level of alcohol
rise?                    A.  The heat molecules
push the alcohol molecules upward. B.  The
alcohol molecules break down into atoms which
take up more space. C.  The alcohol molecules get
farther apart so the alcohol takes up more
space. D.  The water molecules are pushed into
the thermometer and are added to the alcohol
molecules.
82
Student data from pilot testing
Is Answer Choice Correct? A (heat molecules) B (break down) C (farther apart) D (water pushed in) Correct
Yes 38 6 25 6 26
No 24 52 32 58
Not Sure 25 28 30 20
83
Student Responses

• 87 students from grades 7-9 at 3 different
  schools
• 6 students not familiar with alcohol / colored
  alcohol (7)
• 44 chose answer choice A (plausible distractor)
• 6 students wrote heat rises as their
  explanation for A.
• 12 students may have the heat molecules
  misconception.
• Answer choice A is the only one that has the word
  heat in it. (Perhaps add as it is heated to
  the end of one or more answer choices.)

84
Sample student responses

• Answer choice A
• No, because heat molecules cant push alcohol
  molecules because alcohol molecules are denser.
• Yes, I remember learning about heat molecules
  and knew they bump other molecules upward.
• Yes, makes sense heat rises.
• Yes, because heat rises and it is being heated.
• Answer choice B
• No "The molecules dont break down they stay the
  same"
• Answer choice C
• Yes "The space between molecules expands with
  the increase in temperature."
• Answer choice D
• No "Because there is no way that the water can
  get pushed into the thermometer."
• No "Because how could water get through a glass,
  a solid glass."

85

• Examples from plate tectonics of
• Determining appropriateness of terms used in
  assessment items
• Identifying misconceptions
• Identifying implausible ideas for distractors

86
Key Idea a The solid crust of the earth -
including both the continents and the ocean
basins - consists of separate plates.

• Students are expected to know that the rigid,
  outer layer of the earth is made of separate
  sections that are called plates and that the
  plates fit together so that the edge of one plate
  directly touches an adjacent plate with no gaps
  between them. They should know that plates are
  made of solid rock. Students should know that
  each of the major plates encompasses very large
  areas of the earths surface (e.g., an entire
  continent plus adjoining ocean floor or a large
  part of an entire ocean basin) and that the
  boundaries of continents and oceans are not the
  same as the boundaries of plates.

87
1. Determining appropriateness of terminology in
items

• Two items were piloted in order to test student
  knowledge of the term bedrock (after typical
  instruction, i.e., not necessarily targeted to
  the meaning of the word bedrock) to determine if
  the word should be used in assessment and thus be
  part of a clarification statement.
• The two items are identical except one uses the
  term bedrock and the other uses the descriptive
  phrase solid rock.
• These items were piloted at two different middle
  schools in two eastern states at grades 7 and 8.
  Interviews of 9th graders (10 students) in a
  third school in a western state where bedrock is
  readily visible are consistent with these
  findings, but are not presented here.

88
Which of the following are part of earths
plates? A. Solid rock of continents but not
solid rock of ocean floors. B. Solid rock of
ocean floors but not solid rock of continents.
C. Solid rock of both the ocean floors and the
continents.D. Solid rock of neither the ocean
floors or the continents.

• Number of Students 33 (3 classes, two 7th
  grade and one 8th grade)

89
Student data from pilot testing (solid rock)
Is Answer Choice Correct? A (continents only) B (ocean floor only) C (both) D (neither) Correct
Yes 5 3 19 0 57.6
No 24 26 8 29
Not Sure 4 4 4 4
90
Which of the following are part of earths
plates? A. Bedrock of continents but not
bedrock ocean floors.  B. Bedrock of ocean
floors but not bedrock of continents. C.
Bedrock of the ocean floors and the
continents.D. Bedrock of neither ocean floors
nor continents.

• Number of Students 34 (3 classes, one 7th
  grade and two 8th grade)

91
Student data from pilot testing (bedrock)
Is Answer Choice Correct? A (continents only) B (ocean floor only) C (both) D (neither) Correct
Yes 1 2 17 3 50.0
No 20 17 5 19
Not Sure 13 15 12 12
92
Student answers to Bonus Question What is
bedrock?

• Twenty-one of 34 students responded that they did
  not know.
• Students who attempted to define the term said
• The bed of rocks on the ocean floor
• The bottom layer of a rock
• Like the ocean floor
• The bare rock under dirt and sand
• The deep rock of the crust
• Bedrock is rock that is in the ground
• A type of layering of loose pebbles that have
  been fused together
• Rocks and sediments that are on the bottom of
  the continent or ocean
• Rocks on the bottom of the ocean
• Rock Maybe
• It is the rock that is on the bottom of an ocean
  plate

93

• Analysis
• There is a greater number of unsure responses
  when bedrock is used. The item using bedrock
  has 12 to 15 responses of unsure to each answer
  choice, while the item using solid rock has 4
  unsure responses to each of the answer choices.
  Uncertainty about the meaning of the term could
  interfere with student thinking about the idea
  being tested.
• Thirty-two out of the thirty-four students wrote
  responses indicating that they do not know what
  bedrock is. Despite this lack of understanding of
  the term, 50 of the students were able to
  correctly answer this item, compared to 57.6 of
  students answering the item using solid rock.
  Students are apparently translating bedrock to
  mean rock without knowing for sure what it is.
• For now, we have decided not to include the term
  bedrock in the clarification of this idea (even
  though the word is used in a grade 3-5 benchmark)
  and not use it for assessment purposes.

94
2. Identifying misconceptions

• In written comments, a number of students
  expressed misconceptions.
• Which of the following are part of earths
  plates?
• Solid rock of continents but not solid rock of
  ocean floors.
• Plates can be seen and aren't under water.
• The plates do not go down that far.
• Ocean water and solid rock from the bottom is not
  part of a plate.
• B. Solid rock of ocean floors but not solid rock
  of continents.
• Yes, it's only made of rock from the ocean
  surface.

95
3. Identifying implausible distractors

• Which of the following are part of earths
  plates?
• D. Solid rock of neither the ocean floors nor
  the continents.
• None of the 33 students selected this answer
  choice.
• D.  Bedrock of neither ocean floors nor
  continents.
• Three of the 34 students selected this answer
  choice.
• Although students have misconceptions about
  either ocean floors or continents being part of
  plates, the idea that neither ocean floors nor
  continents is part of plates is not plausible.
  This distractor is not informative and should be
  replaced.

96
An example from physics

• Idea d Friction is a force that makes it
  difficult for one object to slide on another
  object (from SFAA 4F-3h).
• From the clarification statement
• Students should know that friction is a force
  that acts in the opposite direction to the
  sliding of one surface on another surface.

97
Alignment/SIGP
FM62-1 (Sixth Grade, n 25, Eighth Grade of
different school, n18) A box slides across the
floor. The arrow labeled "Motion" represents the
box's direction of motion. Which force could be
the force of friction acting on the box?
                                                  
                                         
A.  Force A (40 Sixth / 17 Eighth) B.  Force B
(16 / 0) C.  Force C (40 / 44) D.  Force D
(0 / 17)
98
Possible Misconceptions

• Forces always act in the direction of motion
  (Kuiper, 1994). (Answer choice A)
• Friction is a force in the vertical direction,
  holding an object down (Horizon Research, Inc.).
  (Answer choice B)
• Friction is an upward force gravity is a
  downward force. (Answer choice D)

99
Two routes to the correct answer

• 1. Use targeted learning goal
• Friction opposes the sliding of two surfaces.
• 2. Combine two other ideas
• A backward force slows things down.
• Friction slows things down. (This is a specific
  instance of a general principle-SIGP)
• Therefore, friction is a backward force.
• If students use 2. they have not demonstrated
  knowledge of the learning goal.

100
Student Responses

• Sixth Grade Of the 10 students choosing the
  correct answer
• 2 indicated that they used targeted learning goal
• 2 indicated that they used the other route (false
  positive)
• Eighth Grade Of the 8 students choosing the
  correct answer
• 4 indicated that they used the targeted learning
  goal
• Zero indicated that they used the other route

101
Conclusions

1. Pilot testing can be used successfully to reveal
   what students are thinking about the ideas we are
   testing.
2. Pilot testing provides access to a large number
   of students around the country, but what we learn
   is limited by the questions we ask and what
   students choose to write. Follow-up isnt
   possible.
3. Student interviews allow for flexibility to
   follow up students comments with more probing
   questions, but one-on-one interviews are limited
   to smaller numbers of students.
4. A combination of the two methods is being used to
   provide insights into student thinking and the
   effectiveness of the assessment items that we are
   developing.