Creating A Multiple Measures Placement System

1
Creating A Multiple Measures Placement System

• An Exercise
• With
• Ron Gordon Armand Brunhoeber

2
The Concept

• Although test scores may predict failure, they do
  not necessarily predict success.
• Research shows that students backgrounds,
  environments, and personal habits may have more
  influence on their potential academic success
  than their residual academic skills.
• Find a way to factor that information into the
  placement decision at testing time.

3
Decision Process

• Involve Faculty
• Select Questions
• How many
• What variety
• Select Weight Values
• Use negative weights?
• How much possible total weight

4
Select Questions

• Use enough questions that any one does not have
  undue influence
• Must be manageable
• Each weighted answer choice requires an
  additional line in the placement rule
• Weights must be repeated in each rule segment
• Questions must, in some way, relate to student
  success.

5
Limits

• Limit total weight so that background information
  does not allow students to skip a course level
• Limit number of questions to a manageable number
• More questions adds to testing time
• Placement rules can become unmanageable
• Answer choices must be mutually exclusive and all
  inclusive

6
Example

• (If Arithmetic, plus all weighted choices is gt
  75 OR
• Algebra, plus all weighted choices is gt 48)AND
• (Algebra, plus all weighted choices is lt 65 OR
  Algebra Not Taken) AND
• (CLM, plus all weighted choices is lt62 OR
• CLM Not Taken)
• Then Placement is Elementary Algebra

7
Example, Continued

• If this rule had 5 questions with 4 weighted
  answer choices each, there would be 80 lines just
  for weights.
• With too many questions, or too many choices per
  question, rules can become unmanageable

8
Conditional Weights

• High school accomplishments have limited shelf
  life
• How much does it matter that a 25-year-old
  student had 2 years of high school algebra?
• Does it matter that the same 25-year-old student
  works for a surveyor and uses algebra daily?

9
Conditional Weights

• Young students have not had time to build skill
  usage experience, but their high school
  accomplishments are relevant
• What older students accomplished in high school
  is less relevant than what they do now.

10
Example

• How long has it been since you were enrolled in
  high school or other formal educational process?
• Less than 2 years or still enrolled
• 2 to 5 years
• More than 5 but less than 7 years
• 7 years or more
• Use high school data for up to 5 years,
  experience for more than 5 years.

11
Assigning Weights

• Total possible weight should not move student
  more than one level in either direction
• Set maximum possible weight so a student who
  scores near or above the midpoint of a placement
  range could move up, but one who scores below the
  midpoint could not.
• Use faculty to select BGQ and assign weight
• Guide them

12
Multiple Measures Movement Model
13
Sample Question With Weights

• Which choice below best describes you when you
  read textbooks or other complex information?
• I usually need to read material several times
  before I understand it well -.01
• Sometimes I can understand what I read the first
  time, but often I must reread it .00
• I usually understand what I read if I take notes
  or highlight passages. .01
• I always understand what I read the first time
  through .02

14
Preparing to Build the System

• Assign numeric codes to course names
• Determine which tests will be used for each
  course in each discipline
• Create cut score Table
• Create a BGQ weight Matrix

15
Building the System

• Create Background questions
• Assign BGQ to groups
• Create branching profiles
• Create course groups
• Create courses and assign to groups List
• Create majors if used
• Create placement rules Edit1 Edit2

16
Verify

• Write most complex rule first
• Run verify function in branching profile
• Use several BGQ and score combinations to test
  the placement rule
• Compute weighted score for each run
• Try to hit cut scores to test for bad weight or
  answer choice selections

17
Common Errors

• Unequal weights between rules
• E.G. A response has .01 weight in one rule and
  -.01 weight in the next rule
• Misplaced Parentheses
• The multiple measures weights make the rule
  larger and more difficult to visualize
• Misuse of AND/OR
• Misuse of arithmetic operators
• Wrong answer choice in rule line

18
Troubleshooting

• From the score report, determine what the
  students weight should be from the BGQ responses
• Using the weight, compute the weighted score
• Determine what the placement should be
• Examine the appropriate rule for errors

19
Computing the Weight 1
Question Response Weight
High English A -.02
Grade English C 0
Goal Import. B .02
Understand Read D .03
Study Time A .01
Total Weight .04
20
Computing the Weight 2
Question Response Weight
High English A -.02
Grade English F -.02
Goal Import. A -.01
Understand Read D .03
Study Time C -.01
Total Weight -.03
21
Computation

• Score is multiplied by 1 plus the accumulated
  weight.
• 85 (1.04) 88.4
• Placement will be based on a score of 88.
• Example 2
• 85 (1 -.03) 82.45
• Placement will be based on a score of 82

22
Creating a Multiple Measures Placement System

• An exercise with
• Ron Gordon
• Armand Brunhoeber
• Thank you for not throwing things at the
  presenters