Physics Domain Analysis HALO PHASE 2

1
Physics Domain AnalysisHALO PHASE 2

• Mark Valderrama/Grant Coble ISX/Omniscience
• Christian Haag phi-t/Ontoprise
• Gordon Novak Univ. of Texas
• Anders Rosenquist SRI International

2
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

3
Introductory College Physics Courses

• Three categories
• A qualitative and conceptual with minimal math
• B more in depth than A and assumes
  algebra/trig background
• C deeper but less breadth as B, assumes
  calculus background

4
The AP Physics Tests

• AP tests for B and C levels only
• Tests comprised of
• 70 multiple choice questions (breadth)
• 6-8 free response questions (depth)
• Two 90 minute sections
• Single 1-5 score given
• 5 and 4 typically given college credit
• 3 occasionally given college credit
• Graded on curve, but in general
• 5 gt 75 correct, 4 gt 60, 3 gt 45

5
AP Physics B Test - Goals

• Designed to assess students abilities to
• Read, understand, and interpret physical
  information
• Describe/explain steps required to analyze/solve
  problem
• Use of basic mathematical reasoning (arithmetic,
  algebraic, geometric, trigonometric)
• Perform experiments and interpret results

6
AP Physics B Test - Topics
7
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

8
Initial Analysis

• All teams analyzed AP sample test
• 20 multiple choice questions
• 2 free-response questions
• Initial analysis consisted of three parts
• questions
• tasks
• topics
• All parts linked in Excel workbook

9
Initial Question Analysis

• Questions analyzed by deciding on the necessary
  concepts/procedures needed to solve the question
• These concepts/procedures were instantiated as
  tasks
• Bottom-up approach to analysis

10
Example Question/Tasks
11
Example Question/Tasks

• Tasks
• Solve for final velocity after collision
• Calculate sum of vectors
• Solve equation for certain variable

12
Example Question/Tasks
13
Example Question/Tasks

• Tasks
• Calculate current in a resistor
• Solve system of linear equations

14
Initial Question Table
15
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

16
Initial Question/Task Analysis

• Tasks are building blocks that can be used
  individually or combined to solve a given test
  question
• Tasks are independent they should not call on
  another task implicitly - this is handled by
  subtasks
• A subtask is a related, secondary component for
  solving a particular question. For example
• for the task calculate force of kinetic
  friction a subtask might be calculate weight
• for the task solve for frequency and wavelength
  of photon a subtask might be calculate speed of
  light through medium

17
Question Analysis Comparison

• Sample of an initial question analysis
• Team Task .
• Mark (ISX) Calculate flight time
• Anders (SRI) Calculate time for body to fall
  distance x
• Gordon (SRI) Uniform acceleration (special
  case falling)
• Chris (Onto) Calculate first and second
  derivatives
• Solve equation by certain variable
• Reduce dimension if possible

18
Initial Task Analysis

• After combining the initial analysis of all three
  teams we found different granularity in the
  formulation of tasks. Some of the tasks had a
  distinct math flavor. For example
• Calculate sum of vectors
• Solve equation for certain variable
• Solve system of linear equations

19
Initial Task Analysis

• It was decided to separate these math tasks
  from the physics tasks and put them in a separate
  math tasks table.

20
Math Task Table

• Math Task table contains 8 parent tasks 10
  subtasks that are sometimes helpful, and at times
  mandatory, to solve AP questions.
• They can be divided into the following categories
  (topics)
• Elementary Algebra
• Geometry, Trigonometry, and Vectors
• Calculus (not necessary for Physics B but
  sometimes helpful)

21
Math Topics Table

• An associated Math Topics table was created to
  link to the math tasks

22
Math Tasks and Topics

• The sources for the math tasks and topics are the
  mathematical appendix of the Serway textbook and
  the Princeton Review
• Math tasks are the basic building blocks in this
  analysis
• Physics comes into play when mathematical objects
  get physics related semantics, units, etc.

23
Initial Task Analysis

• Initial task table contained
• 74 physics tasks
• 17 math tasks

24
Initial Task Table
25
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

26
Initial Topic Table

• Topics were generated using the AP syllabus
• Additional topics were taken from a TOC of a
  textbook source (Physics by Halliday, Resnick,
  and Jearl)
• Top-down approach to analysis

27
Initial Topic Table

• Each task in task table was mapped related
  topic(s)
• Prerequisites established for Topics
• Eg. velocity has prerequisites of
  displacement and time interval
• Eg. buoyancy has prerequisites of density and
  static equilibrium

28
Topic Prerequisites

• Knowledge necessary for understanding of the
  topic
• Prerequisites are inherited transitively and from
  prerequisites of child topics
• Table does not explicitly list prerequisites if
  they are inherited

29
Direct Prerequisites
Velocity requires an understanding of time and
displacement
30
Transitive Prerequisites
Acceleration requires velocity and time, but time
is included with velocity
31
Task-Topic Link
Task Calculate flight time of a projectile.
Each task is linked to associated topic(s). This
can allows us to track prerequisites to define a
sub domain of relevant knowledge, and
32
Task-Topic Link
Task Calculate flight time of a projectile.
Each task is linked to associated topic(s). This
can allows us to track prerequisites to define a
sub domain of relevant knowledge, and map tasks
back to text sources for background information.
33
Initial Topic Table
34
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

35
Issues/Observations

• Common Terminology - compiled a list, but loosely
  adhered to it (time issue). For example
• Calculate numerical calculation
• Solve for symbolic algebra manipulation
• Determine qualitative decision making
• Analysis Schema - wanted to add columns to
  existing tables. For example
• Add topiccolumn to task table (implemented)
• Add question column to task table
  (not-implemented)

36
Issues/Observations

• Rephrasing of Questions - concern over how to
  rephrase test questions for the question table.
  Specific guidelines not established.
• Analysis Tool - issue over whether or not Excel
  was best tool for our purposes. After some
  discussion it was agreed (for the most part) that
  excel was good for synthesizing work from various
  teams, was familiar, and allowed for basic
  analysis.

37
Issues/Observations

• Task Definition - concern over what a "task"
  should look like - should it be at the level of
  "friction" or more operationalized - "calculate
  force of kinetic friction"?
• Question Types - Ontoprise established 5
  question types, but these were not used in
  analysis (time issue)

38
Issues/Observations

• Nature of Tasks - the initial tasks from
  Ontoprise were more mathematical in nature than
  from ISX and SRI - this spawned the creation of
  math specific task and topic tables
• Depth of topic and task trees - initial topic
  tree was 5 levels deep in some cases, whereas
  task tree was 2 levels deep max. Task tree uses
  subtasks to some extent to show depth. Tasks
  could be reorganized to better represent depth.

39
Issues/Observations

• Task/subtask relationships were not well defined
  - future inconsistencies in the use of subtasks
  will create problems
• Multiple solutions were not addressed - no
  convention was set for this other than to use a
  single, common solution path.

40
Issues/Observations

• Ontoprise proposed dividing the tasks into three
  different classes or levels of abstraction
• general, basic building blocks (mainly math
  tasks) like adding vectors, calculate
  derivatives, and compare objects
• more general physics-related tasks like Newtons
  laws and Bernoulli effect
• tasks with the highest level of specialization
  like calculate time for body to fall distance x
  and find current through resistor

41
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

42
Full Analysis

• After the three teams came to agreement on the
  question, task, and topic tables, we began our
  full analysis. This analysis was organized the
  following way
• Each team was responsible for analyzing a full AP
  test
• Each team was responsible for review the work of
  another team
• Each team was responsible for contributing to the
  task table

43
Topic Representation
44
Question/Task/Topic Counts

• 319 questions
• 215 physics tasks
• 210 physics topics
• 18 math tasks
• 11 math topics

45
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

46
Diagram Analysis

• Analysis of 257 test questions
• Is the diagram necessary, or just illustrative?
• What type of diagram is it?

47
Diagram Analysis Component Diagrams
1993 Physics B 45

• General level
• objects connected together
• Physics level
• masses acted upon by forces
• motion in two directions
• recognizable components

48
Diagram AnalysisNetwork Diagrams
1993 Physics B 15

• General level
• entities connected together
• Physics level
• schematic diagram of a network
• recognizable parts make up the network
• various parts automatically group together

49
Diagram AnalysisMetric Diagrams
1993 Physics B 57

• General level
• bar with weights on ends
• Physics level
• rigid object with multiple forces acting on it
• strength (length) of forces and their positions
  important

50
Diagram AnalysisR-rated Diagrams

• General level
• complex diagram
• complex motion of object
• Physics level
• multiple parts/regions
• each part/region can be solved independently
• system made up of smaller systems
• novel combination of familiar objects

1993 Physics B FR3
51
X-rated Diagrams

• General level
• Complex diagram
• Many parts
• Physics level
• diagram involves novel objects - not made up of
  familiar components
• novel combination of objects

1998 Physics B FR3
52
Diagram Statistics
53
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

54
Surprises

• Occurrence of separate math task and math topic
  tables
• Initial analyses of the 3 teams had greater
  variance than expected and there were differing
  opinions on how to conduct the analysis. However,
  it was a good idea to let all three teams work
  together - higher quality analysis this way

55
Surprises

• High number of qualitative questions
  expectation that most of the exam would revolve
  around analytical problem-solving and that
  question analysis and problem-solving techniques
  would be fairly straight forward

56
Remaining Work

• Remove duplicate tasks from task list
• Check for tasks that should be collapsed or
  expanded

57
Outline

• Background on Physics AP Test
• Initial Analysis
• Question
• Task, Math task
• Topics/Prerequisites
• Issues/Observations
• Full Analysis
• Statistics
• Diagrams
• General Comments/Observations
• Surprises/Remaining work
• Physics domain observations

58
Physics Domain Observations

• Areas such as mechanics have more numerical
  problems, sometimes ask for derivation of
  formulas.
• More modern areas such as nuclear physics have
  more qualitative problems (e.g. '93 37 and 38)
  that are knowledge base retrieval.
• Often the questions for more exotic kinds of
  physics are easy questions, just testing whether
  the student knows the basics of the more exotic
  area. (e.g. '93 13)

59
Physics Domain Observations

• Areas that have more difficult diagrams include
  electromagnetism, optics, waves, pV gas diagrams,
  some mechanics problems
• Most of the free-response problems have difficult
  diagrams
• Most mechanics problems have medium-difficulty
  diagrams
• Areas with easy diagrams electric circuits,
  point charges, electron energy states.

60
Physics Domain Observations

• Why Physics is intellectually interesting
• It involves physical laws that are the basis of
  much of science and engineering, and it is built
  on a set of basic principles and everything
  (theoretically) can be deduced from those
  principles.
• It involves spatial reasoning
• It involves combinations of principles, not just
  fact retrieval, e.g. '98 9. This requires
  building a combined analysis model from
  components
• It involves "what if?" reasoning, e.g. '93 5
  involves an experiment on another planet