Formative Assessment Materials for Large-Enrollment Physics Lecture Classes

1
Formative Assessment Materials for
Large-Enrollment Physics Lecture Classes

• David E. Meltzer
• Department of Physics
• University of Washington
• www.physicseducation.net
• Supported by NSF DUE-0243258, DUE-0311450,
  PHY-0406724, and PHY-0604703

2
Real-time In-class Formative Assessment

• The Problem How can the instructor assess
  students thinking during class and modify
  in-class instructional activities accordingly?
• Our Goal Develop and test materials that both
• provide a basis for in-class instructional
  activities, and
• assist the instructor in monitoring student
  thinking, moment-to-moment?

?in the context of large-enrollment classes
3
Our Materials Carefully sequenced sets of
multiple-choice questions

• Emphasize qualitative, conceptual items
• Make heavy use of multiple representations
• Designed to maximize student-instructor
  interaction in large classes
• Allow rapid assessment of student learning
• Assist instructors in structuring and guiding
  their presentations and instructional activities

4
Our Materials Carefully sequenced sets of
multiple-choice questions

• Emphasize qualitative, conceptual items
• Make heavy use of multiple representations
• Allow rapid assessment of student learning
• Assist in structuring and guiding the
  presentations and instructional activities

5
Motivation Research in physics education
suggests that

• Problem-solving activities with rapid feedback
  yield improved learning gains
• Eliciting and addressing common conceptual
  difficulties improves learning and retention

6
Active-Learning Pedagogy(Interactive
Engagement)

• problem-solving activities during class time
• student group work
• frequent question-and-answer exchanges
• guided-inquiry methodology guide students with
  leading questions, through structured series of
  research-based problems dress common learning
• Goal Guide students to figure things out for
  themselves as much as possibleuide students to
  figure things out for themselves as much as
  possible

7
Key Themes of Research-Based Instruction

• Emphasize qualitative, non-numerical questions to
  reduce unthoughtful plug and chug.
• Make extensive use of multiple representations to
  deepen understanding.
• (Graphs, diagrams, sketches, simulations,
  animations, etc.)
• Require students to explain their reasoning
  (verbally or in writing) to more clearly expose
  their thought processes.

8
Key Themes of Research-Based Instruction

• Emphasize qualitative, non-numerical questions to
  reduce unthoughtful plug and chug.
• Make extensive use of multiple representations to
  deepen understanding.
• (Graphs, diagrams, sketches, simulations,
  animations, etc.)
• Require students to explain their reasoning
  (verbally or in writing) to more clearly expose
  their thought processes.

9
Key Themes of Research-Based Instruction

• Emphasize qualitative, non-numerical questions to
  reduce unthoughtful plug and chug.
• Make extensive use of multiple representations to
  deepen understanding.
• (Graphs, diagrams, sketches, simulations,
  animations, etc.)
• Require students to explain their reasoning
  (verbally or in writing) to more clearly expose
  their thought processes.

10
Key Themes of Research-Based Instruction

• Emphasize qualitative, non-numerical questions to
  reduce unthoughtful plug and chug.
• Make extensive use of multiple representations to
  deepen understanding.
• (Graphs, diagrams, words, simulations,
  animations, etc.)
• Require students to explain their reasoning
  (verbally or in writing) to more clearly expose
  their thought processes.

11
Key Themes of Research-Based Instruction

• Emphasize qualitative, non-numerical questions to
  reduce unthoughtful plug and chug.
• Make extensive use of multiple representations to
  deepen understanding.
• (Graphs, diagrams, words, simulations,
  animations, etc.)
• Require students to explain their reasoning
  (verbally or in writing) to more clearly expose
  their thought processes.

12
Active Learning in Large Physics Classes

• De-emphasis of lecturing Instead, ask students
  to respond to questions targeted at known
  difficulties.
• Use of classroom communication systems to obtain
  instantaneous feedback from entire class.
• Incorporate cooperative group work using both
  multiple-choice and free-response items
• Goal Transform large-class learning environment
  into office learning environment (i.e.,
  instructor one or two students)

13
Active Learning in Large Physics Classes

• De-emphasis of lecturing Instead, ask students
  to respond to questions targeted at known
  difficulties.
• Use of classroom communication systems to obtain
  instantaneous feedback from entire class.
• Incorporate cooperative group work using both
  multiple-choice and free-response items
• Goal Transform large-class learning environment
  into office learning environment (i.e.,
  instructor one or two students)

14
Fully Interactive Physics LectureDEM and K.
Manivannan, Am. J. Phys. 70, 639 (2002)

• Use structured sequences of multiple-choice
  questions, focused on specific concept small
  conceptual step size
• Use student response system to obtain
  instantaneous responses from all students
  simultaneously (e.g., flash cards)

a variant of Mazurs Peer Instruction
15
(No Transcript)
16
(No Transcript)
17
Results of Assessment

• Learning gains on qualitative problems are well
  above national norms for students in traditional
  courses.
• Performance on quantitative problems is
  comparable to (or slightly better than) that of
  students in traditional courses.
• Typical of other research-based instructional
  methods

18
Features of the Interactive Lecture

• High frequency of questioning
• Easy questions used to maintain flow
• Many question variants are possible
• Instructor must be prepared to use diverse
  questioning strategies

19
High frequency of questioning

• Time per question can be as little as 15 seconds,
  as much as several minutes.
• similar to rhythm of one-on-one tutoring
• Maintain small conceptual step size between
  questions for high-precision feedback on student
  understanding.

20
High frequency of questioning

• Time per question can be as little as 15 seconds,
  as much as several minutes.
• similar to rhythm of one-on-one tutoring
• Maintain small conceptual step size between
  questions for high-precision feedback on student
  understanding.

21
High frequency of questioning

• Time per question can be as little as 15 seconds,
  as much as several minutes
• similar to rhythm of one-on-one tutoring
• Maintain small conceptual step size between
  questions for high-precision feedback on student
  understanding.

22
High frequency of questioning

• Time per question can be as little as 15 seconds,
  as much as several minutes
• similar to rhythm of one-on-one tutoring
• Maintain small conceptual step size between
  questions for high-precision feedback on student
  understanding.

23
Curriculum Requirements for Fully Interactive
Lecture

• Many question sequences employing multiple
  representations, covering full range of topics
• Free-response worksheets adaptable for use in
  lecture hall
• Text reference (Lecture Notes) with strong
  focus on conceptual and qualitative questions
• Workbook for Introductory Physics (DEM and K.
  Manivannan, CD-ROM, 2002)

24
Curriculum Requirements for Fully Interactive
Lecture

• Many question sequences employing multiple
  representations, covering full range of topics
• Free-response worksheets adaptable for use in
  lecture hall
• Text reference (Lecture Notes) with strong
  focus on conceptual and qualitative questions
• Workbook for Introductory Physics (DEM and K.
  Manivannan, CD-ROM, 2002)

25
Curriculum Requirements for Fully Interactive
Lecture

• Many question sequences employing multiple
  representations, covering full range of topics
• Free-response worksheets adaptable for use in
  lecture hall
• Text reference (Lecture Notes) with strong
  focus on conceptual and qualitative questions
• Workbook for Introductory Physics (DEM and K.
  Manivannan, CD-ROM, 2002)

26
Curriculum Requirements for Fully Interactive
Lecture

• Many question sequences employing multiple
  representations, covering full range of topics
• Free-response worksheets adaptable for use in
  lecture hall
• Text reference (Lecture Notes) with strong
  focus on conceptual and qualitative questions
• Workbook for Introductory Physics (DEM and K.
  Manivannan, CD-ROM, 2002)

27
Curriculum Requirements for Fully Interactive
Lecture

• Many question sequences employing multiple
  representations, covering full range of topics
• Free-response worksheets adaptable for use in
  lecture hall
• Text reference (Lecture Notes) with strong
  focus on conceptual and qualitative questions
• Workbook for Introductory Physics (DEM and K.
  Manivannan, CD-ROM, Vol. II, preliminary
  edition, 2002)

28
Curriculum Requirements for Fully Interactive
Lecture

• Many question sequences employing multiple
  representations, covering full range of topics
• Free-response worksheets adaptable for use in
  lecture hall
• Text reference (Lecture Notes) with strong
  focus on conceptual and qualitative questions
• Workbook for Introductory Physics (DEM and K.
  Manivannan, CD-ROM, Vol. II, preliminary
  edition, 2002)

Further development supported by NSF under
Assessment of Student Achievement program
29
Curriculum Requirements for Fully Interactive
Lecture

• Many question sequences employing multiple
  representations, covering full range of topics
• Free-response worksheets adaptable for use in
  lecture hall
• Text reference (Lecture Notes) with strong
  focus on conceptual and qualitative questions
• Workbook for Introductory Physics (DEM and K.
  Manivannan, CD-ROM, Vol. II, preliminary
  edition, 2002)

Further development supported by NSF under
Assessment of Student Achievement program
30
(No Transcript)
31
video
32
Overall Tasks

• Develop question sequences for complete
  (two-semester) course
• Obtain feedback through in-class use to aid
  evaluation of assessment items
• Obtain interview data to validate items
• Do pre- and post-testing with standardized
  diagnostics to help monitor effectiveness

33
Design Strategies

• Not possible to pre-determine all possible
  discussion paths
• Knowledge of probable conceptual sticking points
  is important guided by research
• Make use of standard question variants

34
Many question variants are possible

• Minor alterations to question can generate
  provocative change in context
• add/subtract/change system elements (force,
  resistance, etc.)
• Use standard questioning paradigms
• greater than, less than, equal to
• increase, decrease, remain the same
• left, right, up, down, in, out

35
Many question variants are possible

• Minor alterations to question can generate
  provocative change in context.
• add/subtract/change system elements (force,
  resistance, etc.)
• Use standard questioning paradigms
• greater than, less than, equal to
• increase, decrease, remain the same
• left, right, up, down, in, out

36
Many question variants are possible

• Minor alterations to question can generate
  provocative change in context.
• add/subtract/change system elements (force,
  resistance, etc.)
• Use standard questioning paradigms
• greater than, less than, equal to
• increase, decrease, remain the same
• left, right, up, down, in, out

37
Many question variants are possible

• Minor alterations to question can generate
  provocative change in context.
• add/subtract/change system elements (force,
  resistance, etc.)
• Use standard questioning paradigms
• greater than, less than, equal to
• increase, decrease, remain the same
• left, right, up, down, in, out

38
Many question variants are possible

• Minor alterations to question can generate
  provocative change in context.
• add/subtract/change system elements (force,
  resistance, etc.)
• Use standard questioning paradigms
• greater than, less than, equal to
• increase, decrease, remain the same
• left, right, up, down, in, out

39
Many question variants are possible

• Minor alterations to question can generate
  provocative change in context.
• add/subtract/change system elements (force,
  resistance, etc.)
• Use standard questioning paradigms
• greater than, less than, equal to
• increase, decrease, remain the same
• left, right, up, down, in, out

40
Many question variants are possible

• Minor alterations to question can generate
  provocative change in context.
• add/subtract/change system elements (force,
  resistance, etc.)
• Use standard questioning paradigms
• greater than, less than, equal to
• increase, decrease, remain the same
• left, right, up, down, in, out

41
Easy questions used to maintain flow

• Easy questions (gt 90 correct responses) build
  confidence and encourage student participation.
• If discussion bogs down due to confusion, can
  jump start with easier questions.
• Goal is to maintain continuous and productive
  discussion with and among students.

42
Easy questions used to maintain flow

• Easy questions (gt 90 correct responses) build
  confidence and encourage student participation.
• If discussion bogs down due to confusion, can
  jump start with easier questions.
• Goal is to maintain continuous and productive
  discussion with and among students.

43
Easy questions used to maintain flow

• Easy questions (gt 90 correct responses) build
  confidence and encourage student participation.
• If discussion bogs down due to confusion, can
  jump start with easier questions.
• Goal is to maintain continuous and productive
  discussion with and among students.

44
Interactive Question Sequence

• Set of closely related questions addressing
  diverse aspects of single concept
• Progression from easy to hard questions
• Use multiple representations (diagrams, words,
  equations, graphs, etc.)
• Emphasis on qualitative, not quantitative
  questions, to reduce equation-matching behavior
  and promote deeper thinking

45
Interactive Question Sequence

• Set of closely related questions addressing
  diverse aspects of single concept
• Progression from easy to hard questions
• Use multiple representations (diagrams, words,
  equations, graphs, etc.)
• Emphasis on qualitative, not quantitative
  questions, to reduce equation-matching behavior
  and promote deeper thinking

46
Interactive Question Sequence

• Set of closely related questions addressing
  diverse aspects of single concept
• Progression from easy to hard questions
• Use multiple representations (diagrams, words,
  equations, graphs, etc.)
• Emphasis on qualitative, not quantitative
  questions, to reduce equation-matching behavior
  and promote deeper thinking

47
Interactive Question Sequence

• Set of closely related questions addressing
  diverse aspects of single concept
• Progression from easy to hard questions
• Use multiple representations (diagrams, words,
  equations, graphs, etc.)
• Emphasis on qualitative, not quantitative
  questions, to reduce equation-matching behavior
  and promote deeper thinking

48
Interactive Question Sequence

• Set of closely related questions addressing
  diverse aspects of single concept
• Progression from easy to hard questions
• Use multiple representations (diagrams, words,
  equations, graphs, etc.)
• Emphasis on qualitative, not quantitative
  questions, to reduce equation-matching behavior
  and promote deeper thinking

49
Flash-Card Questions
50
Flash-Card Questions
51
(No Transcript)
52
(No Transcript)
53
Examples of student response data(Algebra-based
general physics at Iowa State University)

• Intended to assist instructors at other
  institutions in selection and planning in use of
  item sequences

54
1 A 0 B 7 C 93 D 0 E 0
1 A 0 B 7 C 93 D 0 E 0
1 A 0 B 7 C 93 D 0 E 0
1 A 0 B 7 C 93 D 0 E 0
55
1 A 0 B 7 C 93 D 0 E 0
2 A 10 B 8 C 77 D 2 E 5
1 A 0 B 7 C 93 D 0 E 0
1 A 0 B 7 C 93 D 0 E 0
1 A 0 B 7 C 93 D 0 E 0
1 A 0 B 7 C 93 D 0 E 0
56
7 A 2 B 3 C 3 D 83 E 9
8 A 0 B 2 C 8 D 87 E 3
57
9 A 0 B 13 C 7 D 53 E 22
10 A 67 B 20 C 9 D 2 E 0
58
Problem Dissection Technique

• Decompose complicated problem into conceptual
  elements
• Work through problem step by step, with continual
  feedback from and interaction with the students
• May be applied to both qualitative and
  quantitative problems

Example Electrostatic Forces
59
Problem Dissection Technique

• Decompose complicated problem into conceptual
  elements
• Work through problem step by step, with continual
  feedback from and interaction with the students
• May be applied to both qualitative and
  quantitative problems

Example Electrostatic Forces
60
Problem Dissection Technique

• Decompose complicated problem into conceptual
  elements
• Work through problem step by step, with continual
  feedback from and interaction with the students
• May be applied to both qualitative and
  quantitative problems

Example Electrostatic Forces
61
Problem Dissection Technique

• Decompose complicated problem into conceptual
  elements
• Work through problem step by step, with continual
  feedback from and interaction with the students
• May be applied to both qualitative and
  quantitative problems

Example Electrostatic Forces
62
Problem Dissection Technique

• Decompose complicated problem into conceptual
  elements
• Work through problem step by step, with continual
  feedback from and interaction with the students
• May be applied to both qualitative and
  quantitative problems

Example Electrostatic Forces
63
Four charges are arranged on a rectangle as shown
in Fig. 1. (q1 q3 10.0 ?C and q2 q4
-15.0 ?C a 30 cm and b 40 cm.) Find the
magnitude and direction of the resultant
electrostatic force on q1.
Question 1 How many forces (due to electrical
interactions) are acting on charge q1? (A) 0 (B)
1 (C) 2 (D) 3 (E) 4 (F) Not sure/dont know
ff
64
Four charges are arranged on a rectangle as shown
in Fig. 1. (q1 q3 10.0 ?C and q2 q4
-15.0 ?C a 30 cm and b 40 cm.) Find the
magnitude and direction of the resultant
electrostatic force on q1.
Question 1 How many forces (due to electrical
interactions) are acting on charge q1? (A) 0 (B)
1 (C) 2 (D) 3 (E) 4 (F) Not sure/dont know
ff
65
Four charges are arranged on a rectangle as shown
in Fig. 1. (q1 q3 10.0 ?C and q2 q4
-15.0 ?C a 30 cm and b 40 cm.) Find the
magnitude and direction of the resultant
electrostatic force on q1.
Question 1 How many forces (due to electrical
interactions) are acting on charge q1? (A) 0 (B)
1 (C) 2 (D) 3 (E) 4 (F) Not sure/dont know
ff
66
Four charges are arranged on a rectangle as shown
in Fig. 1. (q1 q3 10.0 ?C and q2 q4
-15.0 ?C a 30 cm and b 40 cm.) Find the
magnitude and direction of the resultant
electrostatic force on q1.
Question 1 How many forces (due to electrical
interactions) are acting on charge q1? (A) 0 (B)
1 (C) 2 (D) 3 (E) 4 (F) Not sure/dont know
ff
67
For questions 2-4 refer to Fig. 2 and pick a
direction from the choices A, B, C, D, E, and F.
Question 2 Direction of force on q1 due to
q2 Question 3 Direction of force on q1 due to
q3 Question 4 Direction of force on q1 due to
q4
68
For questions 2-4 refer to Fig. 2 and pick a
direction from the choices A, B, C, D, E, and F.
Question 2 Direction of force on q1 due to
q2 Question 3 Direction of force on q1 due to
q3 Question 4 Direction of force on q1 due to
q4
69
For questions 2-4 refer to Fig. 2 and pick a
direction from the choices A, B, C, D, E, and F.
Question 2 Direction of force on q1 due to
q2 Question 3 Direction of force on q1 due to
q3 Question 4 Direction of force on q1 due to
q4
70
For questions 2-4 refer to Fig. 2 and pick a
direction from the choices A, B, C, D, E, and F.
Question 2 Direction of force on q1 due to
q2 Question 3 Direction of force on q1 due to
q3 Question 4 Direction of force on q1 due to
q4
71
Let F2, F3, and F4 be the magnitudes of the force
on q1 due to q2, due to q3, and due to q4
respectively.
Question 5. F2 is given by (A)
kq1q2/a2 (B) kq1q2/b2 (C) kq1q2/(a2
b2) (D) kq1q2/?(a2 b2) (E) None of the
above (F) Not sure/Dont know Question 6. F3
is given by (A) kq1q3/a2 (B)
kq1q3/b2 (C) kq1q3/(a2 b2) (D) kq1q3/?(a2
b2) (E) None of the above (F) Not
sure/Dont know
72
Let F2, F3, and F4 be the magnitudes of the force
on q1 due to q2, due to q3, and due to q4
respectively.
Question 5. F2 is given by (A)
kq1q2/a2 (B) kq1q2/b2 (C) kq1q2/(a2
b2) (D) kq1q2/?(a2 b2) (E) None of the
above (F) Not sure/Dont know Question 6. F3
is given by (A) kq1q3/a2 (B)
kq1q3/b2 (C) kq1q3/(a2 b2) (D) kq1q3/?(a2
b2) (E) None of the above (F) Not
sure/Dont know
73
Let F2, F3, and F4 be the magnitudes of the force
on q1 due to q2, due to q3, and due to q4
respectively.
Question 5. F2 is given by (A)
kq1q2/a2 (B) kq1q2/b2 (C) kq1q2/(a2
b2) (D) kq1q2/?(a2 b2) (E) None of the
above (F) Not sure/Dont know Question 6. F3
is given by (A) kq1q3/a2 (B)
kq1q3/b2 (C) kq1q3/(a2 b2) (D) kq1q3/?(a2
b2) (E) None of the above (F) Not
sure/Dont know
74
Let F2, F3, and F4 be the magnitudes of the force
on q1 due to q2, due to q3, and due to q4
respectively.
Question 5. F2 is given by (A)
kq1q2/a2 (B) kq1q2/b2 (C) kq1q2/(a2
b2) (D) kq1q2/?(a2 b2) (E) None of the
above (F) Not sure/Dont know Question 6. F3
is given by (A) kq1q3/a2 (B)
kq1q3/b2 (C) kq1q3/(a2 b2) (D) kq1q3/?(a2
b2) (E) None of the above (F) Not
sure/Dont know
(etc.)
75
Ongoing Curricular Development(Projects starting
2003)

• Formative Assessment Materials for
  Large-Enrollment Physics Lecture Classes
• Funded through NSFs Assessment of Student
  Achievement program
• Active-Learning Curricular Materials for Fully
  Interactive Physics Lectures
• Funded through NSFs Course, Curriculum, and
  Laboratory Improvement Adaptation and
  Implementation program

76
Ongoing Curricular Development(Projects starting
2003)

• Formative Assessment Materials for
  Large-Enrollment Physics Lecture Classes
• Funded through NSFs Assessment of Student
  Achievement program
• Active-Learning Curricular Materials for Fully
  Interactive Physics Lectures
• Funded through NSFs Course, Curriculum, and
  Laboratory Improvement Adaptation and
  Implementation program

77
Project Phases

• Complete the development of question sequences
  for Chaps. 10-14 of Volume II of Workbook for
  Introductory Physics
• Acquire baseline data by administering questions
  in class with electronic student response system
• Begin work on question sequences for initial
  chapters of Volume I of Workbook

78
Materials Development

• Carried out by DEM and Ngoc-Loan Nguyen (Iowa
  State U. graduate student)
• Created question sequences for electrodynamics,
  optics, modern physics, thermodynamics, and
  mechanical forces
• Acquired baseline data (at Iowa State University)
  for questions on electrostatics
• Currently carrying out editing and additional
  data acquisition

79
Contact

• David E. Meltzer
• dem_at_physicseducation.net
• www.physicseducation.net