Model Validation as an Integrated Social Process

1
Model Validation asan Integrated Social Process

• George P. Richardson
• Rockefeller College of Public Affairs and Policy
• University at Albany - State University of New
  York
• GPR_at_Albany.edu

2
What do we mean by validation?

• No model has ever been or ever will be thoroughly
  validated. Useful, illuminating, or
  inspiring confidence are more apt descriptors
  applying to models than valid (Greenberger et
  al. 1976).
• Validation is a process of establishing
  confidence in the soundness and usefulness of a
  model. (Forrester 1973, Forrester and Senge
  1980).

3
The classic questions

• Not Is the model valid, but
• Is the model suitable for its purposes and the
  problem it addresses?
• Is the model consistent with the slice of reality
  it tries to capture? (Richardson Pugh 1981)

4
The system dynamics modeling process
Adapted from Saeed 1992
5
Processes focusing on system structure
6
Processes focusing on system behavior
7
Two kinds of validating processes
8
The classic tests
Forrester 1973, Forrester Senge 1980,
Richardson and Pugh 1981
9
Validation is present at every step

• Conceptualizing
• Do we have the right people?
• The right dynamic problem definition?
• The right level of aggregation?
• Mapping Developing promising dynamic hypotheses
• Formulating Clarity, logic, and extremes
• Simulating Right behavior for right reasons
• Deciding Implementable conclusions
• Implementing Requires conviction!

10
Do we have the right people?
11
Problem frame stakeholder map
High
Opposition
Low
Problem Frame
Low
Support
High
Weak
Strong
Stakeholder Power
Bryson, Strategic Planning for Public and
Nonprofit Organizations
12
Power versus Interest grid
High
Interest
Low
Weak
Strong
Power
Eden Ackerman 1998
13
Pursuing validity in mapping

• Think causally, not correlationally
• Think stocks and flows, even if you dont draw
  them
• Use units to make the causal logic plausible,
  even if you dont write them down
• Be able to tell a story for every link and loop
• Move progressively from less precise to more
  precise -- from informal map to formal map

14
The standard cautions
15
These arrows mean and then

• We start with some understandings of the problem
  and its systemic context, and then we
  conceptualize (map) the system.
• Then we build the beginnings of a model, which we
  then test to understand it.
• Then we reformulate, or reconceptualize, or
  revise our understandings, or do some of all
  three, and then continue

16
Arrows here are flows of material
The words here represent stocks. This is not a
causal diagram.
17
Only this one is a causal loop
No explicit stocks or flows, no clear units, but
it tells a compelling story Its a good start.
18
Project modeling core structure
19
Identical structure without explicit stocks and
flows
20
Pursuing validity writing equations

• Recognizable parameters
• Robust equation forms
• Phase relations
• Richardsons Rule Every complicated, ugly,
  excessively mathematical equation and every
  equation flaw saps confidence in the model.

21
Modeling conflict within between nations
22
Complexity flaws destroy confidence

• P of int'l conflict
• DELAY FIXED ((Lateral pressure/10Military force
  effect/Trade and bargaining leverage
  International conflict)/Lateral conflict break
  point, 1 , 0)
• Flaws
• Complexity, discreteness, units confusion and
  disagreement, disembodied parameter, confusion of
  the effect of a concept leverage with the
  concept itself, and the wonder what keeps this
  probability between 0 and 1?

23
Robust equation forms
24
Causal mish-mash
25
Robust equation formulations
26
Robust equation formulations
27
Robust equation formulations
28
Robust equation formulations
29
Pursuing validity in equations Phasing
30
Phase relations
Constant Perceived Value suggests continually
rising Resources, but that doesnt seem correct
31
Phase relations
Here, the Perceived Value of Integrated
Information sets a planned level of resources
32
Pursuing validity fitting to data

• Generally, a weak test of model validity
• Whole-model procedures
• Optimization
• Partial-model procedures
• Reporting results
• Graphically
• Numerically Theil statistics

33
Example of weakness of fitting to data

• Logistic curve
• dx/dt ax - bx2
• Gompertz curve
• dx/dt ax - bx ln(x)

34
Fitting global petroleum with Logistic
35
Fitting global petroleum with Gompertz
36
Presenting model fit visually
37
Presenting model fit numerically

• Theil statistics, for example
• Based on a breakdown of the mean squared error
• 1 Bias Variation Covariation

38
Presenting model fit numerically
39
Learning from surprise model behavior

• Have clear a priori expectations
• Follow up all unanticipated behavior to
  appropriate resolution
• Confirm all behavioral hypotheses through
  appropriate model tests (Mass 1991/1981)

40
Tests to reveal and resolve surprise behavior

• Testing the symmetry of policy response (up and
  down)
• Testing large amplitude versus small amplitude
  response
• Testing policies entering at different points
• Testing different patterns of behavior
• Isolating uniqueness of equilibrium or steady
  state
• Understanding forces producing equilibrium
  positions (Mass 1991/1981)

41
Summary

• Modelers, stakeholders, problem experts, and
  others in the modeling process pursue validity at
  every step along the way.
• We have rigorous traditions guiding model
  creation, formulation, exploration, and
  implications.
• We have a powerful, intimidating battery of tests
  of model structure and behavior.
• Model-based conclusions that make it through all
  this deserve the confidence of everyone in the
  process.

42
Epilog

• Reason is itself a matter of faith. It is an act
  of faith to assert that our thoughts have any
  relation to reality. (G.K. Chesterton)
• I have no exquisite reason fort, but I have
  reason good enough. (Sir Andrew, Twelfth Night)

43
References

• Greenberger, Crensen and Crissy (1976). Models in
  the Policy Process. New York Russell Sage
  Foundation.
• Forrester, J. W. (1973). Confidence in Models of
  Social Behavior--With Emphasis on System Dynamics
  Models., M. I. T. System Dynamics Group.
• Forrester, J. W. and P. M. Senge (1980). Tests
  for Building Confidence in System Dynamics
  Models. System Dynamics. A. A. Legasto, Jr. et
  al., New York, North-Holland. 14 209-228.
• Richardson, G. P. and A. L. Pugh, III (1981).
  Introduction to System Dynamics Modeling with
  DYNAMO. Cambridge MA, Productivity Press.
  Reprinted by Pegasus Communications.
• Saeed, K. (1992). "Slicing a complex problem for
  systems dynamics modeling." System Dynamics
  Review 8(3) 251-262.
• Bryson, J. (199x). Strategic Planning for Public
  and Nonprofit Organizations, citing Eden and
  Ackerman, Making Strategy (1998) and Anderson,
  Bryson, and Crosby (1999).
• Eden, C. and F. Ackerman (1998). Making
  Strategy.
• Mass, N. J. (1991/1981). "Diagnosing surprise
  model behavior a tool for evolving behavioral
  and policy insights (1981)." System Dynamics
  Review 7(1) 68-86.