The Future of Educational Accountability: Notes for a Political Economy of Measurement

1
The Future of Educational AccountabilityNotes
for a Political Economy of Measurement

• Michael J. Feuer
• National Research Council
• The National Academies
• CRESST
• January 2007
• Celebrating Bob Linn

2
How do we sound...?

• A bit depressed?
• Is anything all right?
• A bit complicated?
• Famous Speeches - Video Gallery

3
Roberts rules

• 1- do rigorous science
• 2- convey findings simply, without simplifying
• 3- build public and policy maker confidence
• 4- keep smiling, even when the Denver airport is
  closed

4
Pretest

• For this section, solve each problem and decide
  which is the best of the choices given. Fill in
  the corresponding circle on the answer sheet. You
  may use any available space for scratchwork.
• (The use of a calculator is permitted.)

5
1. Scholarship

• Bob Linns contributions to measurement theory
  and practice cover which of the following topics
• Validity, reliability, fairness
• Standard setting
• Military, K-12, employment, personality testing
• Admissions and placement
• Intelligence
• Classroom based assessment
• Performance measurement
• ELLs and SWD
• Test linking and equating
• Accountability
• All of the above

6
2. Mentorship

• Choose the adjective that best characterizes Bob
  Linn in his work with students, colleagues, and
  freeloaders (like the NRC)
• Accessible
• Honest
• Friendly
• Supportive
• Patient
• Modest
• All of the above

7
3. Public service

• During his career Bob Linn has been involved in
  which of the following activities
• District level school reform consulting
• Statewide assessment design and evaluation
• National policy review and strategy development
• International comparative assessments of school
  and teacher quality
• Chairmanship or participation in national, state,
  local commissions
• All of the above

8
3. Statesmanship

• In answering the question, Bob, would you be
  willing to help us on a project ..., what
  phrases are missing from Bobs lexicon
• No
• How much are you paying?
• Do I get first authorship?
• Can I fly first class?
• What, that question AGAIN?
• All of the above

9
Sneak-peek at the answers...

• Sorry, that would be cheating...

10
Oh, what the heck. A little test coaching wont
kill anyone...

• If an item looks like it might have more than one
  correct answer, and if you have the option of
  choosing all, then you should. (Keep this
  advice handy for later...)
• Congratulations on scoring above basic!

with apologies to Dan Koretz
11
And now for a quick summary of my paper...

• Defining terms
• Reasoning by analogy
• An audacious theorem
• From optimization to optimism

12
Political Economy

• Markets
• Externalities and market failure
• Government, visible hands, collective action
• Rowhouses and outdoor lighting
• Accountability
• cause and remedy
• Some examples

13
An idea from the department of clever symmetries

• Political economy in general is about the
  measurement of externalities.
• In the context of educational accountability its
  about
• the externalities of measurement

14
Test based accountability externality
considerations

• Origins from Horace Mann to Bobby Kennedy
• Principles
• education provided through system of acceptable
  coercion, i.e., taxation
• Legitimate public demand for information about
  quality of schooling
• Concern for intended and unintended consequences
  of instruction
• Good news if you like democracy Educational
  accountability is part of the broader family of
  social arrangements designed to instill
  discipline in the quality and provision of public
  goods.
• Caution from Jim March the demand for
  accountability is a sign of pathology in the
  social system

15
In other words...

• Like most (all?) beneficial technologies, high
  stakes testing produces its own externalities
• narrowing of the curriculum, score inflation,
  excessive rote memorization at the expense of
  deeper learning, reduction of educational goals
  to low minimum standards, etc.
• incentives for opportunism, manipulation,
  cheating
• perpetuation of inequality, differential
  expectations
• Erosion of support for education due to
  persistent disappointment
• Etc.

16
Ideas possibly worth stealing from other policy
arenas

• Common approaches to management of externalities
• Regulation, enforcement, accountability
• Upstream and downstream remedies
• Examples

17
Testing in context recap

• Assumptions
• Multiple purposes
• Validity and reliability
• Consequences
• Freedom of information
• The power of parsimony
• Accountability and democracy (trust but verify)

18
Pop quiz

• IN A MORE PERFECT WORLD, accountability systems
  would have which of the following features
• Domain robustness authentic criterion-based
  representations of complex cognitive functioning
  based on efficient domain sampling
• Recursive stability immunity of scores to
  opportunistic behavioral responses and
  degradation of validity
• Causal neutrality description of population
  differences without risk of implied attribution
  of cause and/or implied acceptance of
  differential or discriminatory standards
• Policy precision incentives for improved
  performance in desired domains without unintended
  degradation of performance in other domains
• Instrument selectivity use of tests solely for
  the purposes for which they were designed and
  validated
• Diagnostic generality measures of aggregate
  performance (or growth) based on population
  sampling that yield valid inferences for high
  stakes individual-level decisions
• Minimal burden comprehensive information based
  on multiple measures within constraints on budget
  and testing time
• Rational exuberance standards set high enough to
  motivate public commitment to improvement without
  risk of cyclical disappointment and erosion of
  morale
• ALL OF THE ABOVE?

19
Sso much for coaching...
20
Some discouraging news

• Theorem
• There is no accountability system that satisfies
  all those conditions simultaneously.
• Proof left as an exercise to the reader...

21
But dont despair...

• Other impossibility theorems can be EVEN MORE
  depressing
• Like democracy and majority rule? Try this
• From the presentation to Kenneth Arrow of the
  Nobel Prize in economics, 1972
• ... Let us assume that in a society one has a
  number of alternative conditions to choose
  between and that each individual in the society
  can rank all these alternatives in order of
  desirability. Is it ... possible to find
  ethically acceptable, democratic rules, for
  making a collective (or social) ranking of the
  different alternatives in order of desirability?
  
• Arrows answer No.
• The conclusion rather discouraging as regards
  the dream of democracy...

Original paper Arrow, K., "A Difficulty in the
Concept of Social Welfare," Journal of Political
Economy, 58, 4, August, 1950.
22
From impossibility to reason

• Cognitive science applied to policy
• Procedural rationality
• Assessing the counterfactuals
• Suppose NCLB really does die...
• The predicament we face
• Measuring (!) externalities downstream
• Compensation or redress what to do with false
  positives?
• Anticipating externalities upstream
• CPA

see Feuer, Moderating the Debate Rationality
and the Promise of American Education, Harvard
Education Press, 2006
23

• Thank you.
• I have benefited from this opportunity, and hope
  the externality I produced was at least a little
  bit positive...
• Comments write me at
• mfeuer_at_nas.edu