Human Systems Dynamics Theory Applied to Evaluation Practice

1
Human Systems Dynamics Theory Applied to
Evaluation Practice

• American Evaluation Association2008

• Beverly Parsons, Ph.D.InSites bparsons_at_insites.o
  rg

• Meg Hargreaves, Ph.D.Mathematica Policy
  Research, Inc. mhargreaves_at_mathematica-mpr.com

2
(No Transcript)
3
(No Transcript)
4
(No Transcript)
5
Introduction to a Systems Perspective In
Evaluation

• This section presents
• System definitions
• System features
• System characteristics
• Types of systems
• Examples of types

6
Systems Definitions

• Multiple definitions
• A group of interacting, interrelated, or
  interdependent parts forming a complex whole
• A configuration of parts joined together by a web
  of relationships
• The parts form a whole, which is greater than the
  sum of its parts

7
System Features
Systems are as much an idea about the real
world as a physical description of it

• Boundaries define who or what lies inside or
  outside the system
• Differences among the parts influence the
  systems dynamics
• Relationships among parts, between parts and
  whole, and between whole and its environment, are
  key focus of systems

8
System Characteristics

• Common patterns, behaviors, and properties
• Patterns unorganized, organized, or organic
  (self-organized)
• Behaviors random, simple, complicated, or
  complex adaptive linear or nonlinear
• Properties independent, interrelated, or
  interdependent relationships
• Scale small to large, self-similarity across
  levels (fractals)

9
System Types

• Systems can be grouped by their level of
  complexity or organization
• Random (no system) - unorganized
• Simple system - organized
• Complicated system organized
• Complex adaptive system organic

10
Random (Unorganized)

• Random, chaotic activity no pattern
• Independent, unconnected parts
• No cause-effect relationships constant chaos
  and surprise
• Turbulence - no equilibrium
• Random parts without a system
• No leadership - people react blindly
• Unknowable

11
Random System Examples

• War zone Civilians caught in crossfire, random
  flight to escape conflict
• Natural disaster At landfall or in the eye of
  the storm, residents react instinctively to
  events
• Leadership transitions During changes in
  administration old patterns are suspended before
  new patterns are established

12
Simple System (Organized)

• Stable, static pattern
• Parts connected in linear ways
• Predictable cause-effect relationships
• Set equilibrium
• System reducible to parts and replicated
• Directive leadership - designed change
• Known knowns answers are evident

13
Simple System Examples

• Baking a cake Follow a recipe to assemble and
  combine ingredients into a batter that is baked
  at a pre-set temperature with predictable results
  
• Flu shot clinics Nurses use consistent
  procedures to administer the same shots to each
  person, following a set protocol in assembly-line
  fashion

14
Complicated (Organized)

• Dynamic pattern of feedback loops
• Many interrelated parts across subsystems, levels
  
• Complex, nonlinear cause-effect relationships
• Feedback can stabilize equilibrium thermostat
• System can be reduced to parts and replicated
• Multiple answers investigate options
• Unknowns become known through expert analysis at
  multiple levels

15
Complicated System Examples

• Space Shuttle Challenger disintegrated (1986)
  when O-ring failure caused a rocket booster
  breach, creating flare that damaged external fuel
  tank, spilling fuel that exploded
• In large healthcare institutions, human behaviors
  are part of wider systems of causality, in which
  medical errors can occur in organizational and
  policy contexts that result in long (36-hour)
  shifts, large caseloads, and strained staff
  relations

16
Complex Adaptive System (Organic)

• Dynamical patterns parts adapting to each
  other and to environment as a whole
• Parts are massively entangled, interdependent
• Parts self-organize, learn, coevolve organically
• Equilibrium in flux - sensitive to initial
  conditions
• System not replicable, cant repeat past
• Emergent change manage conditions of organic
  development and experimentation
• Unknown unknowns trial and error

17
Complex Adaptive System Examples

• Economic system interactions of homeowners,
  mortgage lenders, stock market traders,
  investors, federal banking institutions, and
  worried consumers are coevolving into global
  crisis and recession, despite governments
  interventions
• User networks (Diabetes, AA) facilitate exchange
  of information and advice on care for chronic
  conditions among participants, learning from each
  other

18
Background about Systems Theories

• This section presents
• General systems theory
• Cybernetics systems dynamics
• Complex adaptive systems
• Implications for evaluation

19
General Systems Theory

• Holistic change ideas ancient Greeks
• General systems theory - von Bertalanffy
  (1930s) earliest work by Bogdanov (1910)
• Open systems nonrandom elements organized into
  interacting, interrelated components that seek to
  survive through interactions with environment
• Each system level nested in higher level (cells,
  organisms, families, organizations, communities,
  societies)

20
Implications for Evaluation

• The whole can enable/constrain parts and the
  parts can contribute to and/or challenge
  stability of the whole
• Because open systems are structured in
  hierarchies useful to look one level above and
  one level below the system in focus
• Evaluate system viability does system have both
  the parts and the information and decision flows
  among the parts that are needed to survive?

21
Cybernetics and System Dynamics

• System dynamics founded by Forrester at MIT
  (1950s) for electrical engineering
• Method for calculating and modeling how many
  circular, interlocking, sometimes time-delayed
  relationships among parts are important in
  shaping system-wide behavior
• Through negative feedback, adjustments made to
  keep system in balance positive feedback used to
  move system in same direction, moving out of
  balance

22
Implications for Evaluation

• Assess influence of feedback loops on behavior of
  systems parts and on whole
• Behavior of whole not only explained by behavior
  of parts (e.g. medical errors)
• Feedback loops undermine sustainability of
  public interventions (policy resistance)
• Evaluators cannot step outside social and
  ecological systems to observe (not
  value-neutral) self-reflection needed

23
Complex Adaptive Systems

• Key CAS writers Weaver (1948), Simon (1962),
  Prigogine (1987), Stacey (1997, 2007), Zimmerman
  et al (2001), Eoyang (2006)
• CAS many semi-independent and diverse agents,
  who are free to act in unpredictable ways,
  continually interact with each other, adapting to
  each other and to environment as a whole,
  creating system-wide patterns
• Key concepts emergence, organic
  self-organization, co-evolution, simple rules

24
Implications for Evaluation

• Currently relevant evaluation criteria and
  measures may need to be updated as new
  conditions emerge
• Measure frequently for emerging patterns
• Avoid grand modeling projects for prediction use
  smaller projects for ongoing experimentation and
  learning
• Also visualize system interactions as networks
  look outside nested levels for system patterns,
  drivers, and constraints
• Ask what, so what, now what?

25
Three Dynamics of a Social System and its Context
Unorganized     dynamics (random
unpatterned seemingly
chaotic)
far from agreement
organic dynamics     (emerging patterns
    coherent but not predictable)
Agreement
Organized  dynamics    
(predictable    orderly
controlled)
close to agreement
far from certainty
close to certainty
Certainty
26
Match of Evaluation Designs to Dynamics of
Social Systems and Their Context
N
T
E
O
X
T
C
Exploratory Design
far from agreement
unorganized dynamic
Organic Design
Agreement
Initiative Renewal Design
organic dynamic
Predictive Design
close to agreement
organized dynamic
far from certainty
close to certainty
Certainty
27
Understanding Organic Dynamics (Activity)

• Divide into triads
• Selects one other triad member (doesnt tell) and
  uninvolved person in refreshment area
• Stay at least two feet apart and equidistant from
  the other two
• Do this for about 1-2 minutes while trying to
  reach refreshments
• Reflect on experience

28
Case Study Introduction
Do the preconference professional development
offerings contribute to effective
evaluation-related work of association members?
If so, how?
29
Unorganized System Dynamics
What is happening? What boundaries, differences,
similarities, and relationships might shape how
the offerings contribute to participants
evaluation-related work?
30
Organized System Dynamics
Do participants receive high-quality content that
is relevant to their evaluation-related work and
is delivered through high-quality instructional
methods?
31
Organized System Dynamics
How do the format and content of the session
support or hinder participants in understanding
and using the session to apply the learning from
the session to their evaluation work?
32
Organic System Dynamics
What patterns among participants (including the
session facilitators) before and during the
session are likely to affect the participants
understanding and application of the learning to
their evaluation-related work?
33
Patterns
34
Patterns
35
Patterns
36
Patterns
37
Patterns
Centers for Medicare Medicaid Services. (2008).
System and Impact Research and Technical
Assistance for CMS FY2005, FY2006, and FY2007
RCSC Grants (2008). Annual Report. Cambridge,
MA Abt Associates, Inc. (p. 10)
38
Patterns
Centers for Medicare Medicaid Services. (2008).
System and Impact Research and Technical
Assistance for CMS FY2005, FY2006, and FY2007
RCSC Grants (2008). Annual Report. Cambridge,
MA Abt Associates, Inc. (p. 27)
39
Patterns
Centers for Medicare Medicaid Services. (2008).
System and Impact Research and Technical
Assistance for CMS FY2005, FY2006, and FY2007
RCSC Grants (2008). Annual Report. Cambridge,
MA Abt Associates, Inc. (p.42)
40
Patterns
Centers for Medicare Medicaid Services. (2008).
System and Impact Research and Technical
Assistance for CMS FY2005, FY2006, and FY2007
RCSC Grants (2008). Annual Report. Cambridge,
MA Abt Associates, Inc. (p. 78)
41
Patterns
42
FractalsPatterns, Patterns Everywhere

• In nature . . .
• Mathematical construct of iterating nonlinear
  equation and plotting on complex number
  planeMandelbrot Set
• Similar shapes at all scalesBroccoli
• Biological scaling gives coherence in widely
  diverse entitiesOak tree
• Scale-free networks

43
FractalsPatterns, Patterns Everywhere

• Recognizing patterns is criticalsimilarities,
  differences, and relationships that have meaning
  across space and time
• Basic values or simple rules generate diverse,
  but self-similar behavior across scales
• Naming and telling stories about dynamics in a
  system help reinforce and shape fractal patterns

44
Fractals
45
Looking at the Dynamics as a Whole

• What is the overall picture of system dynamics
  affecting how the preconference professional
  development offerings contribute to effective
  evaluation-related activities of AEA members?
• Given the findings from the three system dynamics
  within the preconference session, how might the
  preconference professional development process
  and offerings be modified to contribute more
  substantially to the quality of AEA members
  evaluation-related work?