P1249598116vpEjn - PowerPoint PPT Presentation

Title: P1249598116vpEjn


1
(No Transcript)
2
Performance Decisions

• Civil Engineering Systems
• University of Bristol

3
Recognise these questions?

• Have we really taken into account all the factors
  affecting our decision?
• Tell me again about the assumptions?
• Just what are our levels of risk?
• But where did these numbers come from?
• Have we got any evidence to prove that?
• Where are we vulnerable?
• Are we really ready to make that decision yet?
• What is the most important factor?

4
Specialist analysis hard to understand /
interrogate / test Not enough analysis
Clash of personal / departmental cultures
Input numbers are shaky Assumptions not
clearly identified / tested Decision factors /
influences poorly / incompletely framed
Uncertainty, risks and unknowns hidden or
forgotten
5
Yet why?

• Breadth depth balance of issues not fully
  grasped
• Undue trust in quantitative analysis
• Different influencers see only part of the big
  picture
• Delays due to lack of consensus
• Poor knowledge management or corporate learning
• Qualitative judgement and quantitative analysis
  done in isolation of each other
• Lack of framework

6
What is the PeriMeta Approach?

• A means to
• Enhance decision making in the context of
  incomplete, sparse and conflicting information
• Communicate complex systems simply
• Manage uncertainty explicitly
• Integrate hard and soft influences
• By
• Modelling systems as hierarchies of processes
• Recording the attributes of the processes
• Embedding a rich uncertainty calculus
• Using all available evidence in whatever form

7
Added Value

• Shared understanding of the state of the asset
• across teams
• up and down the organisation
• with all stakeholders ..simply
• Meta level system - health overview
• Identify success,failure and vulnerability
• Sensitivity and value of information
• Vehicle for testing intervention strategies
• Decision recording
• corporate memory, transparency, auditability

8
Programme Development
Civil Engineering Systems Bristol
9
Political risk City vuln 3.02 CMAM
10
At its heart

• A way to communicate uncertainty and its converse
  dependability

Sn(A) Evidence that A is successful 1
- Sp(A) Evidence that A is not
successful Sp(A) - Sn(A) Lack of evidence
11
Interval Probability Theory

• Coin Toss

0 1
classically
heads
open world

• Life elsewhere ?

12
(No Transcript)
13
Key principles 1/2

• Simple only one kind of blob and link!
• The blobs are processes with objectives
• The degree to which a process meets its
  objectives is expressed explicitly
• Its performance or dependability
• The asset or system of interest is described
  hierarchically
• Each hierarchy layer represents a fairly complete
  description of the system of interest

14
Key principles 2/2

• Layers in the hierarchy can be related to
  different levels of decision-making to the
  contingency planning process
• Evidence of performance is assembled from all
  available sources
• from expert judgement, visual inspection reports,
  instrumentation model analysis etc
• Best built as a challenged group activity

15
Generating specific views

• Multi-attribute weights can be changed to
  emphasise specific points of view on system
  performance, e.g. safety or economics.
• If any performance indicator is irrelevant to
  that aspect of performance is can be set a weight
  of zero.

16
Key features

• Rapid prototyping of models for exploring
  decision scenarios
• Comprehensive model building process for
  operational models
• It is not just a software package
• Rapid communication, exploration and modification
  with the built model

17
Presidential safety
18
The Human Decision Maker

• - Creative
• - Responsible
• - Operating in an Open World
• - Achieving Satisfaction Self
  esteem
• Therefore enhance rather than replace

19
Evidence for a Decision
Determining the Objectives
Generating the Options
Assembling the Evidence
Modelling Option Performance
Comparing Options with Objectives and states of
nature
Making the Choice
Assessing Risks Values
Taking Action to maximise value and mitigate risk
20
Evidence for a Decision
Generating the Options
Determining the Objectives
Assembling the Evidence
PeriMeta
Modelling Option Performance
Comparing Options with Objectives and states of
nature
Making the Choice
Assessing Risks Values
Taking Action to maximise value and mitigate risk
21
Philosophical problems

• Seventeenth century natural scientists dreamed
  of uniting the ideas of rationality, necessity
  and certainty into a single mathematical package,
  and the effect of that dream was to inflict on
  Human Reason a wound that remained unhealed for
  three hundred years a wound from which we are
  only recently beginning to recover

Stephen Toulmin 2001 Return to Reason Harvard
University Press, p13
22
Culture informs process which defines tools
Why before How
23
Building the model

• Model the Process
• Why before how
• Assemble the evidence
• All sources mapped to common expression
• Record the Attributes
• Why Who What When Where How?
• Use a Rich Uncertainty Calculus
• Gives a powerful handle on the dependability of
  each of the processes
• The process of constructing the model encourages
  creative collective reflection on how the asset
  system performs

24
Evidence from sub-processes

• Propagation of uncertainty requires input of a
  set of conditional probabilities

H
E3
E1
E2

• PeriMeta maps the conditional probabilities to
  linguistic variables

25
Summary of Judgements Required

• Evidence - for and against separated
• Sufficiency - How much of the evidence is
  directly relevant to the parent process?
• 1 it is sufficient on its own to fully
  determine the success of the parent
• Dependency - How much overlap of evidence is
  there between the sub-processes?
• Issues of bias and redundancy
• Necessity - Will the parent fail if the
  sub-process fails?
• Higher necessity puts more weight on the red

26
The Essence of Risky Decisions
27
Examples of Use

• Oil industry
• Expert interpretation of sparse data managing the
  oilfield asset
• Political risk
• Water sector
• Assessing sustainability of supply
• Assessing safety of contract
• Civil Engineering
• Flood defence decisions
• Measuring Egan performance
• Highways Agency PFI MAC
• Assessment of Terrorist Risk

28
Reservoir estimate dependability
Things coming out of the woodwork
29
Field Development Success?
30
Spend?
31
Value of InformationTesting strategies
32
(No Transcript)
33
Key Drivers for the Highways Agency

• Procurement have asked for a generic performance
  specification to apply to new MAC contracts
• The modernising government initiative
  (OGC/Treasury/NAO) requires outcome based
  processes to improve efficiency and effectiveness
• SSR are responding to HAs business needs
• Mapping of rate of progress showed that standard
  approach would be too slow

34
Proposition

• To develop a performance regime that connects
  outcomes to what people do (i.e. process) through
  consultation and learning to improve
• Provide a framework for high level performance
  specification and decision support
• Motivating people by helping them to understand
  where they fit and their contribution to outcomes
  
• Provide confidence and competence through
  systematic rigour and recognition of uncertainty

35
Asset Knowledge
36
Alignment
Cabinet Office and OGC are promoting the
recognition of uncertainty
WOOs Work on Outputs and Outcomes Purpose To
match route based outputs to network level
outcomes/targets.
OD Quality Management (OD Process Mapping)
Corporate Planning Team Lisa Scott / Dick
Tyson PPDG - Performance Planning Development
Group HA Performance Man. Framework for HA
PRIDe Performance Measurement Group Purpose To
develop better metrics for Area Performance
Indicators to facilitate benchmarking across the
network. To develop CCC compliance indicators.
Keith Shaw
Simon Smith
Glynn Harrison
John Fitch
Performance Regime Purpose To develop
performance regime for PFMAC Output performance
specification to allow greater freedom to
innovate and improve
Nick Harding
Keith Shaw
SUNS Setting up Network Strategy Purpose To
identify business improvements and best working
practices. Output Desk instructions John Bagley
(Leeds)
Maintenance Contractors Individual Providers have
various systems for demonstrating good
performance.
Halcrow
Keith Shaw
?
Supply Chain Management Integrated teams and
continuous improvement David Parker
Network Strategy KPIs Purpose Publish KPIs
against which to measure performance of key
Agency asset management and service delivery
activities.
?
Keith Shaw
We are not re-inventing the wheel!
37
Stakeholder Diagram
Freight
Commuters
Many Others
Road Users
Road Users
Consultants
MAC - Internal
Contractors
HA
Government
Suppliers
Others
Other Stakeholders
Other Stakeholders
Local Authorities
Many Others
Emergency Services
38
Measurement Boundaries
Service Delivery
Service Boundary
HA
HAMAC
MAC
Contract Boundary
Delivery Slice
Demonstrate
Detail

• Measurement needs to be at the Contract Boundary

39
Outcomes
4 D Model
Contract Boundary
Measured Process Indicators used to measure
delivery of outcomes
Deliver
Demonstrate
Process Demonstrating competence
Process Demonstrating competence
Develop Continuous Improvement over time
40
Where are we going?

• No right answer, so how can you be sure you
  have the right answer!?
• We are searching for a model that
• is robust, and with which the owners are
  comfortable.
• enables exploration of the performance regime by
  all sorts of stakeholders
• Will drive improvement in outcomes
• Enables alignment to payment

41
Journey to Process Model
Why How
When ?Reflecting process cycles

• Who? Process owners a useful guide

42
Attributes of Process
HIGHWAYS AGENCY KEY OBJECTIVES

• Sustainability

• Safety

• Process
• Winter Service
• Owner
• Network Manager
• Purpose
• - Safe mobility in winter
• Relevant outcome measures
• - Journey times
• Accident rates
• Material used (proxy sustainability)

Staff (A Good Employer)

• Customer perspective

• Best Value (Finance)

• Delivering in Partnership

• Performance
• Measurement

Innovation Learning
NOTE HA Balanced Scorecard Perspectives in Blue
43
Types of Evidence

• Types of evidence considered
• Measured
• Performance Measurement Group (PMG) measures
• Existing contract Area Performance Indicators
• HA High Level KPIs (e.g. Balanced Scorecard,
  Government Objectives, etc.)
• Instrumentation, analytical and computer models
• Linguistic
• Expert judgement from form reports or interviews

44
Value Functions

• Performance targets are expressed as value
  functions
• Translate PIs onto a non-dimensional 0-1 scale
• 0 failure
• 1 total success
• Allows different types of evidence to be brought
  together

45
Mapping Value Functions
Value
Performance Indicator
46
Highways Agency measurement

• Library of Value Functions HA KPIs Targets
  2003/04

Effective Maintenance
Maintain at least 85 of the network in good
condition.
47
Winter service

• Worked example

48
Contractors process model for a MAC
HA objectives
Functional processes
Core processes
Procedures
Tasks
Winter service
49
Winter service process
Propagated
WHY
HOW
WHEN
50
Outcome measures of success

• Current Total number 29
• 8 Key performance measures
• 12 Covered in quality plan
• 9 Other measures
• Reduced to 3 key measures
• Accident rate
• Average journey time through system
• Quantity of grit used (environmental constraint)
• Plus conformance with plan
• to fulfil public duty

51
Benefit of outcome measures

• Simple measures common for many processes
• Moderated to be relevant to process context eg
  MOORI (Met office open road indicator).
• Benchmarked to achieve continuous improvement
• Empowers those doing the job to improve
  performance by
• Better measurement of their process
• Continuous improvement
• Innovating

52
Conclusions

• Progress is on programme
• Strong evidence of need to improve measurement
• Key generic measures to be tested
• Safer travel KSI reduction
• Mobility hours of congestion
• Customer Satisfaction local surveys
• Minimise adverse environmental impact
• Robust process model to enable context based
  measurement at MAC boundary
• Next stage wider engagement - validation

53
Overall PeriMeta Conclusions

• Shared understanding of the state of the asset
• across teams
• up and down the organisation
• with all stakeholders ..simply
• Performance Regime
• health vulnerability overview
• Sensitivity and value of information
• Decision recording
• corporate memory, transparency, auditability

54
(No Transcript)
55
Terminology
Risk is the likelihood of an uncertain event or
behaviour, and its consequences for our intended
purpose or objectives, set in a context that
needs to be understood
Uncertainty is a property of information
fuzziness, incompleteness and randomness
Vulnerability susceptibility to
disproportionate damage from an event or behaviour
Surprise - an unexpected event an unrecognised
risk
Hazard - set of incubating preconditions for
failure
56
The Nature of Uncertainty
Fuzziness - Imprecision of
definition Incompleteness
- That which we do not
know, choose not to
include or cannot
afford to include Randomness -
Lack of a specific
pattern
57
The Nature of Uncertainty

• Fuzziness Incompleteness Randomness

58
Tools for Uncertainty Management
Analogue Studies Case Based Reasoning
Parametric Studies Safety Factors
Monte Carlo Simulation Bayesian
Reasoning Fuzzy Methods
Neural Nets Genetic
Algorithms Evidential
Reasoning
Process models
59
Just because...
... past futures have resembled past pasts, it
does not follow that future futures will
resemble future pasts. Bryan Magee Popper
1973
60
Understanding risks
frequent
Predict from history
Priority action
occurence
Understanding what we do not know
Look out for change
infrequent
consequence
high
low
61
Vulnerability

• Now done in many areas manually
• Complexity of systems means that is now
  unreliable to depend on unaided human
  identification
• Need systems approach as opposed to a
  reductionist paradigm

62
Juniper

• There is a need for generating new processes
  for imagining imaginative outcomes .
• Schneider The future of climate potential for
  interaction and surprises In Downing Climate
  Change and World Food Security. Springer Berlin
  (1996 p79)

63
Risk Management Plans

• ( KnowRisk Australian RM software )

64
QRA

• Limited scope applicable in tightly constrained
  physical environments
• Needs frequency database
• Makes bold assumptions
• Difficult to bring in soft systems
• Yet we need to mix hard and soft..

65
HA Balanced Scorecard
HA Vision Safe Roads Reliable Journeys Informed
Travellers Travelling with Confidence
Staff Perspective How effective are we at
managing, developing and motivating our
workforce?
7 Oct 02