Statistical Moderation Decoded

1
Statistical ModerationDecoded

• Andrew Shortell
• 8th April 2006

2
Why are we here?

• Want the best outcomes for our students
• Want to understand what is being done to us

3
How Study score works

• Long line
• Fill from start
• Bell curve
• Certain number at each point
• French 1500 ? likely gt 1 at each point
• Italian 850 ? gaps at top end
• Very small LOTEs treated individually
• Why the line? Click

4
The line

• Uni Entrance ? ENTER
• Ranking
• Comparison
• Different position each year
• Why Bother?
• Isnt absolute better? Click

5
Absolute

• Do teachers know what they are doing?
• Can teachers be trusted?
• Yes, but
• State-wide comparison
• Nation-wide
• International? Note Bologna style University
  degrees

6
Your Job

• Rank students
• Based on subject content
• Prepare for the filter of the exam
• Based on subject content
• Maybe teach??

7
Rank students

• Exam already does that
• Curriculum driven or exam driven ?
• Oral does that
• On the day problems
• Stage fright
• Etc
• Can VCAA really do it? click

8
Who really knows students?

• Parents ?
• ROFL
• Teachers?
• IYD In your dreams
• INMJ
• Care factor

9
Objective .. Subjective

• Evidence Based reporting
• More than just a gut feeling
• Many other factors
• Priorities (ours, theirs, parents, clients.)
• Repeatable
• Between school comparison

10
SACs

• Internal
• Trust the teacher??
• Variable ?
• On the day etc
• Cross marked??
• With whom (multiple classes ?? ROFL again)

11
What to do?

• Old CAT days
• Called in,
• marked independently
• marked from across state
• Get a reasonable idea of state standard for this
  year
• Expensive
• Have to pay people .. (OMG)
• Alternative?? click

12
Statistical Moderation

• Easy to do
• Computerised
• Easy to design
• Easy to implement
• Easy to repeat
• Teacher factor removed (more or less)
• Cheaper (Is that really important?)

13
Who?

• Professor Tim Brown
• Uni Melb Statn
• Loves his job
• Understands stats

14
Reliable

• Statistics are reliable
• When there are no other issues
• Other statisticians can understand them
• Thus Unis rely on their statns
• They believe them
• Thus Do we have a choice?

15
What this means for us
16
Understand what is being done to you so that you
can work the system to your best advantage
17
Quick Explanation

• Exam reference score
• Adds all exam results for each student
• Creates a ranking
• GAT reference score
• Yes it is used (sometimes) (If it helps statl
  reliability means?)
• SAC scores (treated as a single item)

18
Then

• Zero on the exam removed
• Special provision (C.O.D.) removed
• Statistically anomalous removed
• E.g. outliers at the low end (2 SDs )
• N removed
• These are all put back afterwards

19
BUT..

• What if that takes your class to
• (Shock, Horror) Less than five
• Oh Dear, Oh Dear
• Unreliable
• Trouble

20
Detour

• Need five to be absolutely reliable
• Four is ok but not so reliable in every
  circumstance
• Remember stats work best with large groups
• ? Partner class
• More problems (I have them in Systems)
• Do I need to tell you ? (rhetorical Q) RQ

21
Then

• Statistically equated
• AS A GROUP-- more later
• Added together
• Thus the long line for the study score

22
Detail

• Top SAC score is changed to equal top exam ref
  score
• Does not matter who it is
• Upper quartiles made equal
• Mean
• Lower quartile
• But not the bottom

23
In between

• Attempt a line of best fit
• Else try a quadratic (Do you care?)

24
Consequences

25
E. g.
After removing CoD, zero etc
26
3 equal SAC at the top
27
3 equal exam

• Alternatively
• So this shows that separation is not
  disadvantageous
• This type of thing happened for me

28
Fair?

• It is your job to separate.
• In this scenario what is right?
• Who can we blame?

29
How it occurs

• Teacher sets work for SAC
• T teaches to SAC
• T marks it
• Students do well
• ? Parents happy
• ? School happy

30
Value.

• Everyone is happy
• Until the results come out
• Nothing worse than despondent students in class
  who think the teacher is too harsh
• Easily fixed in above scenario
• After the exam they are gone so no problem
• ??

31
Perhaps

• Need to review SAC results
• Review SAC ranking after each SAC
• Need to educate students about expectations
• Tell them about these scenarios (or similar)
• Build in idea that no-one gets full marks
• Ensure by putting in higher order questions
• Dont teach entirely to the SAC

32
Implications

• Student expectations are very high in some
  schools
• Big fish, small pond etc
• State-wide is bigger pond
• Remember the long line

33
Consider
Student expectations of top 3 here (And parents)
34
Consider

• Alternate scenario
• NB Same result
• Different start
• Expectations are very different
• Would you have got this from despondent students?

35
Add understanding

• Students need to be educated empowered
• Get them to understand the rank order is the
  important thing

36
Bigger example (smaller font)

• This is not quite precise (mathematically)
• Top end lowered
• Bottom end raised
• Mean, SD, UQ, LQ not precise in this e.g.)
• Rank order is Preserved

37
Practical consequences

• Students to fight to climb rank order
• Students co-operate to raise exam results, Oral
  results
• Share exam prep
• Secretive SAC prep

38
Reality

• Many will share SAC prep too
• Ensure work is different
• Set SACs that are slightly different for each
  student
• Authentication interviews

39
Practical

• Check rank order before last two SACs
• Set Questions that are harder Higher order
• Separate

40
VCAA believes

• that you know your students
• that you can accurately set rank order
• That if you make two students equal then they are
• That you do not have a state-wide picture

41
Partner Class

• VCAA believes
• Partner class teachers agree on every mark
• Can register disagreement on VASS
• Can unpartner struggle but can do
• Partner class teachers work together
• You need to
• That the rank order is across all of the group
• So you need to

42
What if

• Every student in your class obtained 100 in exam
  and oral
• 50 problem
• Teacher of the year material
• You make a mistake
• Fix it before exam results are published
• Or..

43
Questions

• Scenarios
• Copyleft

44
Thanks

• VCAA
• Sean Fitzmaurice
• Professor Tim Brown
• For creating this stat modn model
• Teachers for surviving
• To you for listening

45

• Andrew Shortell
• a.shortell_at_braemar.vic.edu.au
• VFTA