Assessment Using Online Assignment Management Systems

1
Assessment Using Online Assignment
ManagementSystems
Glenn Ledder Department of Mathematics University
of Nebraska-Lincoln gledder_at_math.unl.edu
2
Assessment Using Online Assignment
ManagementSystems

1. Defining the terms
2. OAM advantages and disadvantages
3. Low-maintenance administration
4. Accurate and reproducible results
5. Getting meaningful results

3
Defining the Terms
4
Assessment

• Conflicting meanings

5
Assessment

• Conflicting meanings
• General meaning measuring academic success in
  any context

6
Assessment

• Conflicting meanings
• General meaning measuring academic success in
  any context
• Specific meaning measuring academic success in a
  broad context

7
Assessment

• Conflicting meanings
• General meaning measuring academic success in
  any context
• Specific meaning measuring academic success in a
  broad context
• (larger than a single course)

8
Assessment vs Evaluation

• Dictionary definition
• assess to determine the importance, size, or
  value of
• evaluate to determine or fix the value of

9
Assessment vs Evaluation

• Dictionary definition
• assess to determine the importance, size, or
  value of
• evaluate to determine or fix the value of
• Connotation
• assessment usually emphasizes the making of
  summary judgments
• evaluation usually emphasizes the measurement
  of quantitative value

10
Assessment vs Evaluation

• Assessment
• measuring knowledge and understanding
• Evaluation
• measuring achievement

11
Assessment vs Evaluation

• Assessment
• measuring knowledge and understanding
• used for placement
• Evaluation
• measuring achievement
• used to determine course grades

12
Assessment vs Evaluation

• Assessment
• measuring knowledge and understanding
• used for placement
• Evaluation
• measuring achievement
• used to determine course grades
• Test an instrument used for assessment,
  evaluation, or both

13
OAM Advantages and Disadvantages
14
OAM Advantages/Disadvantages

• Immediate grading
• (no wait for grading or machine scoring)

15
OAM Advantages/Disadvantages

• Immediate grading
• General purpose testing center
• (no need for direct faculty involvement)

16
OAM Advantages/Disadvantages

• Immediate grading
• General purpose testing center
• Administrative difficulties
• (How do you maintain the database?)

17
OAM Advantages/Disadvantages

• Immediate grading
• General purpose testing center
• Administrative difficulties
• Algorithmic Qs and symbolic As
• (individual answers, no prompts)

18
OAM Advantages/Disadvantages

• Immediate grading
• General purpose testing center
• Administrative difficulties
• Algorithmic Qs and symbolic As
• Difficulties in writing questions
• (How to avoid tricky, picky, sticky?)

19
Low-Maintenance Administration
20
Low-Maintenance Administration

• Lots of instructors and proctors
• Should require minimal knowledge

21
Low-Maintenance Administration

• Lots of instructors and proctors
• Should require minimal knowledge
• One or two faculty manage the system
• Should require minimal time

22
Low-Maintenance Administration

• Lots of instructors and proctors
• Should require minimal knowledge
• One or two faculty manage the system
• Should require minimal time
• Questions are permanent.
• Student records are transient.

23
The Math 106 EDU folder structure

• 106 COURSE
• Question banks
• Gateway exam
• Practice assignments

The 106 COURSE data is permanent. The only
regular changes are to the Gateway exam
dates. Maintained by faculty manager.
24
The Math 106 EDU folder structure

• 106 COURSE
• Question banks
• Gateway exam
• Practice assignments

• 106 CLASS
• Question banks
• Gateway exam
• Practice assignments
• Student records

The 106 COURSE data is permanent. The only
regular changes are to the Gateway exam
dates. Maintained by faculty manager.
Students register for their own 106 CLASS. Tests
are inherited. Student records are
local. Maintained by individual
instructors/proctors.
25
Accurate and Reproducible Results
26
Reproducibility

• Tests should measure something.
• The test has to yield consistent results in
  multiple administrations for a single student.

27
Reproducibility

• Tests should measure something.
• The test has to yield consistent results in
  multiple administrations for a single student.
• But questions must change when a test is given at
  different times.
• This is especially important for evaluations.

28
Reproducibility

• Use template problems to get a great variety of
  answers.
• Use template problems to get uniformity of
  content and difficulty.
• Group problems into categories that are
  consistent in content and difficulty.

29
The Math 106 Gateway Exam
10 questions, 8 correct to pass

• Elementary functions
  xn, sin(ax), cos(ax),
  tan(ax), eax, ln x, nx
• 2. Products 3. Quotients 4. Compositions
• 5. Compositions of compositions
• 6. Products with a composite factor
• 7. Compositions of products
• 8. Quotients with an embedded composition
• 9. Quotients with an embedded product
• 10. Functions defined by equations

30
Category 4 - Compositions

• A, C, N gt 0 B ? 0 K ? 0, 1

The algorithm chooses 5 random integers
satisfying certain requirements.
31
Category 4 - Compositions

• A, C, N gt 0 B ? 0 K ? 0, 1
• X t, u, v, w, x, y, z

The algorithm randomly chooses an independent
variable.
32
Category 4 - Compositions

• X t, u, v, w, x, y, z A,C,Ngt0 B?0
  K?0,1
• P XNB, XNBX
• Q AXNB, AXNBX, sqrt(X)B
• S sin AX, cos AX, tan AX
• T e -CXB, eKXBX
• U Ae -CXB, AeKXBX, A ln X, ANX

The algorithm randomly creates functions P, Q, S,
T, U, using some of the chosen integers.
33
Category 4 - Compositions

• X t, u, v, w, x, y, z A,C,Ngt0 B?0
  K?0,1
• P XNB, XNBX
• Q AXNB, AXNBX, sqrt(X)B
• S sin AX, cos AX, tan AX
• T e -CXB, eKXBX
• U Ae -CXB, AeKXBX, A ln X, ANX
• F1 sqrt(P), sqrt(S), sqrt(T)
  
• F2 SN, TN
• F3 ln Q, ln CS
  
• F4 eQ, eCS
• F5 sin Q, cos Q, sin U, cos U

The algorithm chooses one of 5 functions based on
prior elements. There are 38 templates, each with
7 independent variables and at least one
parameter
34
Reproducibility

• Tests should measure something.
• The test has to yield consistent results in
  multiple administrations for a single student.

35
Accuracy

• Tests should measure what they were intended to
  measure.

36
Accuracy

• Tests should measure what they were intended to
  measure.
• Questions should have to be done right to be
  counted right.

37
Accuracy

• Tests should measure what they were intended to
  measure.
• Questions should have to be done right to be
  counted right.
• Students who can do what the test is supposed to
  measure should get a good score.

38
Counted right ? Done right

• Savvy test takers can do well on multiple choice
  tests even when they dont understand the
  material.

39
Counted right ? Done right

• Savvy test takers can do well on multiple choice
  tests even when they dont understand the
  material.
• For calculations, students should have to type in
  the answer.

40
Counted right ? Done right

• Savvy test takers can do well on multiple choice
  tests even when they dont understand the
  material.
• For calculations, students should have to type in
  the answer.
• For conceptual questions, use multiple selection.
  (Which of the following)

41
Can do ? Good score

• Tricky Misinterpretation or small mistakes
  account for many wrong answers.

42
Can do ? Good score

• Tricky Misinterpretation or small mistakes
  account for many wrong answers.

Find the derivative of cos e2x and Find the
derivative of e-2 cos x are tricky
43
Can do ? Good score

• Tricky Misinterpretation or small mistakes
  account for many wrong answers.

Find the derivative of cos e2x and Find the
derivative of e-2 cos x are tricky, compared
to Find the derivative of e2 cos x
44
Can do ? Good score

• Tricky Misinterpretation or small mistakes
  account for many wrong answers.

Find the derivative of cos e2x and Find the
derivative of e-2 cos x are tricky, compared
to Find the derivative of e2 cos x
The problem is meant to test the chain rule, not
elementary algebra.
45
Can do ? Good score

• Picky Too many details connect minor errors with
  wrong answers.

Find the derivative of cos2(2x3) 4 sin x.
Find the derivative of 4x5-2x cos (ex2).
46
Can do ? Good score

• Picky Too many details connect minor errors with
  wrong answers.

Find the derivative of cos2(2x3) 4 sin x.
Find the derivative of cos2(2x3).
Find the derivative of 4x5-2x cos (ex2).
Find the derivative of ex5 cos (ex2).
47
Can do ? Good score

• Picky Too many details connect minor errors with
  wrong answers.

Find the derivative of cos2(2x3) 4 sin x.
Find the derivative of cos2(2x3).
The simpler problems are hard enough.
Find the derivative of 4x5-2x cos (ex2).
Find the derivative of ex5 cos (ex2).
48
Can do ? Good score

• Sticky Difficulties in transferring answer from
  paper to computer.

2x2 x3
4x(x3)-2x2 (x3)2
Find the derivative of .
49
Can do ? Good score

• Sticky Difficulties in transferring answer from
  paper to computer.

2x2 x3
4x(x3)-2x2 (x3)2
Find the derivative of .
2x2 x3
32 25
Find the derivative of at x 2.
50
Can do ? Good score

• Sticky Difficulties in transferring answer from
  paper to computer.

2x2 x3
4x(x3)-2x2 (x3)2
Find the derivative of .
2x2 x3
32 25
Find the derivative of at x 2.
2x2 (x3)2
Type 4x(x3)-2x2 / (x3)2, get
4x(x3)-
51
Getting Meaningful Results
52
Accuracy

• Tests should measure what they were intended to
  measure.

53
Getting Meaningful Results

• Tests should measure what they were intended to
  measure.
• What should OAM evaluations and assessments be
  intended to measure?

54
Dont give up paper evaluations
55
Dont give up paper evaluations

• Use paper exams for questions that demand partial
  credit and questions where the answer is an
  integral, a graph, or an explanation.

56
Dont give up paper evaluations

• Use paper exams for questions that demand partial
  credit and questions where the answer is an
  integral, a graph, or an explanation.
• Use OAM evaluation for routine computations and
  basic concepts.

57
Dont give up paper evaluations

• Use paper exams for questions that demand partial
  credit and questions where the answer is an
  integral, a graph, or an explanation.
• Use OAM evaluation for routine computations and
  basic concepts.
• Testing takes time and effort. Decide what is
  important and then create the test to match.
• Set high standards and allow retakes.

58
Do give up paper assessments
59
Do give up paper assessments

• Assess routine computations and basic concepts.

60
Do give up paper assessments

• Assess routine computations and basic concepts.
• Expect very few retakes. Qualified students
  should pass on the first try.
• Provide necessary formulas online or as a printed
  supplement. Test conceptual understanding and
  knowledge, not memory for trivia.