What I Learned

1
What I Learned About Assessment From the AP
Program Dan Kennedy Baylor School Houston AP
Teachers Meeting November 11, 2006
2
True Confession of a Veteran Mathematics Teacher
For many years I never thought much about
assessment. I graded my students in what I
thought was an appropriate variety of ways Tests
Quizzes and Homework. This model had stood
the test of time.
3
In 1986 I was invited to become a member of the
AP Calculus Test Development Committee. From
1990 to 1994 I would serve as chair.
My experience with this group changed my views of
assessment forever.
4
I had already learned one important fact about
classroom assessment merely by teaching an AP
course It changes the entire classroom dynamic
when the teacher honestly does not know what will
be on the test.
The teacher has no other option but to teach the
students how to think for themselves!
5

• Why students dont think on tests
• Thinking takes time.
• Thinking is only necessary when you cannot
  do something without thinking.
• If you can do something without thinking, you
  can do it very well.
• Students who can do something very well have
  been well-prepared.
• Therefore, if you prepare them well, your
  students will proceed through your tests
  without thinking!

6
Were AP Calculus exams predictable? 1987 BC
Exam 1. Differential equation2. Implicit
Differentiation3. Area/volume4. Series5.
Particle problem6. Theory problem (stretch)
Of course, this was just one exam. But there
were others like it.
7
But if we tried to change anything, teachers
would notice. Then, in AP workshops all over
the country, teachers would find themselves
uttering to AP consultants the words they dreaded
most when spoken by their students
8

• And why should teachers NOT ask that question?
• It is how the game is played.
• We show the students how to do math.
• We let them practice at it for a while.
• Then we give them a test to see how well they
  can mimic what we did.
• The game is won and lost for BOTH of us on test
  day.

9
This was just another example of the educational
paradigm that was leading my student not to think
on tests!
But how can teachers change the game if we want
our students to succeed?
10
Teachers have one secret weapon We define what
it means to succeed. We control the grade!
11
Something I learned about assessment from the AP
program It is perfectly OK to scale grades!
75
5
12

• At our school, 75 is not a good grade. In fact,
  65 is a minimal pass.
• Is this reasonable? Think about it.
• The all-time NBA record for field goal
  percentage in a season is 72.7.
• The all-time record batting average for major
  league baseball is .440 (44).
• A salesperson who makes a sale on 75 of first
  contacts is a genius.
• So how can we expect 75 success from someone who
  is just learning?

13

• If the AP exam were constructed so that the
  low-to-average student could get 75 of the
  maximum points,
• it wouldnt be much of a test, and
• the distribution would be skewed rather than
  normal.

14
99


92

82
71


30
75
93
20
15
An Important Disclaimer Scaling grades is not
about building self-esteem. Scaling grades is
about teaching mathematics. Assessment should
support your efforts to teach your students
mathematics. It should not get in the way.
16
Scaling grades on the TI-84 Plus
17
(No Transcript)
18
(No Transcript)
19
(No Transcript)
20

• Some things that ETS worried about that I didnt
• r-biserial
• Content validity
• Speededness
• True score
• Grading rubrics

21
r-biserial (r-bis) A correlation coefficient
relating performance on a test question and
performance on the measure used as a criterion.
It is an index of discrimination measuring the
extent to which examinees who score high on the
measure used as the criterion tend to get the
question right and those who score low tend to
get it wrong.
22
1969 Multiple-choice question 26
The answer is (C).
23
AB Stats A 3 B 57 C 7 D 3 E 20 BC
Stats A 1 B 70 C 11 D 2 E
9 Projected Chimpanzee Stats A 20 B 20
C 20 D 20 E 20
24
Correct responses to problem 26
AB 7 BC 11 Chimps 20
25
Content Validity Validity is the extent to
which a test measures what it is supposed to
measure. The content validity of an (AP) test is
the extent to which the content of the test
represents a balanced and adequate sampling of
the universe of content in which the test is
intended to measure achievement.
26
The AP Calculator Experiment (1983-84)
In 1983 the AP Calculus Committee decided to
allow (but not require) the use of scientific
calculators on the AP Calculus examinations.
This was not to be a very happy debut for
technology on the AP stage.
27
AP readers found that students were losing points
on the free-response section because of
calculator misuse. The calculators affected the
scores. But calculators were not being tested!
This compromised the content validity. The
committee had two choices 1. Forbid calculators
and test as usual 2. Require calculators and
alter the test. They chose to forbid the
calculators.
28
One of my Precalculus tests from 1990
Note the emphasis on computation. Note that
there is nothing here to suggest that any of this
stuff is worth knowing!
29
A recent test on the same functions
Still not perfect, but a better test.
30
Speededness
The appropriateness of a test in terms of the
length of time allotted. For most purposes, a
good test will make full use of the examination
period but not be so speeded that an examinees
rate of work will have an undue influence on the
score he receives.
31
Allowing for speededness
Exam Format for AP Calculus AB Exam Format Exam Format for AP Calculus AB Exam Format Exam Format for AP Calculus AB Exam Format Exam Format for AP Calculus AB Exam Format

  
        

        

32
True Score A score entirely free of errors of
measurement. True scores are hypothetical values
never obtained in actual testing. A true score is
sometimes defined as the average score that would
result from an infinite series of measurements
with the same or exactly equivalent tests,
assuming no practice effect or change in the
examinee during the testings.
33
Why teachers dont need to worry about true
score We can assess our students all year
long! The more often the better. Sorry, kids.
Yessss!
34
AP Calculus Grading Rubrics
If the AP readers can give partial credit fairly
to 250,000 students, I ought to be able to do it
for my own students. In AP Calculus, I can even
use the AP rubrics to do it.
35

• AP Calculus Exams are
• Designed to test knowledge
• Designed to test cleverness
• Scaled reasonably
• Not made up by the teacher
• Open assessments
• Comprehensive assessments (valid)
• Honest about technology

36

• Two Fundamental Principles
• Assess what you value.
• Value what you assess.

37

• Some problems with traditional tests
• They assess only a fraction of what we value.
• They depend too much on luck.
• There is often no feedback (as with final exams).
• They are usually taken alone. (Is this what we
  value?)
• They are usually timed. (Is this a good model for
  quality work?)
• They are frequently taken under artificial,
  stressful conditions.
• They are dependent on teacher stimulus.
• They are often devoid of creativity (if students
  are prepared).
• They favor one narrow kind of student
  performance.
• Success is usually short-term and
  non-transferable.
• The emphasis in the end is what the student can
  NOT do.
• They can inhibit further learning.

38

• Some assessment strategies I like
• Assess what you value and value what you assess!
• Assess often, with different kinds of
  assessments.
• Give meaningful and prompt feedback.
• Give partial credit for partially correct work.
• Explain all your expectations to your students
  from the start.
• Test diligence, knowledge, and cleverness in
  focused ways.
• Encourage creativity through your assessments.
• Scale grades to control the standard deviation.
• Only fail students who are failures. Keep
  everyone in the game.
• Encourage collaboration in class and on homework.
• Assess diligence. Find a way to grade homework
  frequently.
• Try portfolios.
• Remember This is not about self-esteem. Its
  about teaching mathematics to all your students!

39
Rebecca Flakes Portfolio Entry
Rebecca Flake
40
Students need to hand in a portfolio of items of
their own choosing. The main point of this
assessment is that they are not responding to a
stimulus from me (as in a test or a quiz). My
primary directive for student portfolio entries
is this Give me evidence of your learning that I
otherwise would not have!
41
This was my first year to be a peer tutor, and I
enjoyed helping the girls in the dorm a lot. Last
night, though, I finally saw the importance of my
peer tutoring. My roommate came in at 1000
extremely upset over her Precalculus test that
was the next day. I calmed her down and told her
that I would help her if I could. Carrie, who had
been in the play, had gotten behind in her work,
so she didnt understand what they were doing.
She showed me the problem. I knew the answer, but
I wasnt sure how to explain it to her in a way
that was not confusing. I thought about it for a
while, and I ended up trying several approaches
(with Claras help) that I had learned in
Calculus, until I finally got through to her.
Then I made her work a few problems for me, and
she did them perfectly. She understood! I was so
happy to be able to help her that I had forgotten
I was supposed to be studying for my own Calculus
test. She was so happy she understood that she
began to cry. She really began to cry. Its great
to be able to use the things you have learned to
help other people learn too.
42
A happy footnote Carrie really did understand.
She scored 93 on the Precalculus test the
following day a personal best for her, and a
full 9 points above the class average.
Actress Carrie
43
E-mail me at
dkennedy_at_baylor.chattanooga.net
44
Or visit the Baylor School web site at
www.baylorschool.org. Click on me under Faculty
and link to my home page.