Challenges With Building, Using and Interpreting Risk Models

1
Challenges With Building, Using and Interpreting
Risk Models

• Ian Branagan
• 21st June 2007

2
IntroductionThere is no Magic Bullet

• Model
•    noun 1 a three-dimensional representation of
  a person or thing, typically on a smaller scale.
  2 (in sculpture) a figure made in clay or wax
  which is then reproduced in a more durable
  material. 3 something used as an example.
• 4 a simplified mathematical description of a
  system or process, used to assist calculations
  and predictions.
• 5 an excellent example of a quality. 6 a person
  employed to display clothes by wearing them. 7 a
  person employed to pose for an artist. 8 a
  particular design or version of a product.

Source Oxford English Dictionary
At the end of 2005 many users and recipients of
Cat models and their output questioned their
value based on the comparison of actual vs
modelled losses in The 2004 and 2005 North
Atlantic Hurricanes. Underlying this belief many
lost site of the fact that Cat models models are
simplifications Of complex, often poorly
understood, physical systems and should be seen
as tools to assist our process rather than be
its sole driver.
3
Risk Modelling Culture

• Effective pragmatic and holistic implementation
  of Risk modelling tools can be a powerful
  component of a business, from pricing to exposure
  management and capital alocation. However this
  begins with some cultural fundamentals including
• An understanding that all models are wrong
• An actual outcome will never match the expected
  case
• But it should be with in your distribution !
• If it looks too good to be true it probably is
• Comfort with uncertainty
• There is no single answer, merely a range of
  potential outcomes
• Multiple views of risk
• Actual vs Expected
• Learn from mistakes and confront them openly
• Questions and feedback loops
• Just because a particular problem is hard dont
  ignore it make it up
• You can only improve it over time

4
All Models are Wrong - 5 Key Drivers of Model
Error
An attempt at examining the main common sources
of misunderstanding or error, it is by no means
exhaustive however

• Fundamental difficulties
• Un modeled perils and risks
• Modelling malpractice
• Portfolio error
• Be Careful

5
Fundamental Difficulties

• Building a mathematical model to represent
  natural phenomenon is just hard
• Hazard
• What
• Where
• How Often
• How big
• Vulnerability
• How bad
• Financial Loss
• How much
• Models will always be wrong to some degree in
  actual individual events every event provides
  learning
• Unknown EQ Faults ?
• Appropriate historical period to estimate
  frequency and severity ?
• Wind from the wrong direction ?
• Demand Surge on Pool Enclosures?
• Hopefully an appropriate stochastic risk model
  should see actual losses appear somewhere in the
  modelled distribution
• We just may have got the frequency of loss size
  wrong !
• Important to use multiple models and views on the
  same problem
• More Physics Less Statistics !

6
Fundamental Difficulties

• Hazard Uncertainty - Earthquake
• Tokai NanKai
• Two opposing theories regarding the seismic
  characteristics
• One theory that the Tokai segment of the Nankai
  trough cannot rupture on its own it can only
  rupture in association with the other segments of
  the Nankai trough.
• The rival theory is that the Tokai segment can
  rupture on its own and, is overdue for rupture.
• Using Model A as an index, these four models
  shown intend to represent the spread of opinion
  in terms of relative loss size at different
  return periods based on different views described
  above.
• If each view currently has reasonable credibility
  then relying on only one of them could be
  problematic

7
Fundamental Difficulties Importance of Multiple
Models

• Single model users invariably Optimise into the
  model
• By design or accident will migrate towards the
  most optimistic representation of risk over time
• All models have biases, some serious. By
  offering different biases, alternative model
  views can mitigate single bias error.
• Dealing with model change
• Given the uncertianty surrounding all underlying
  components of Cat models, they may be subject to
  significant change from version to version.
• Hazard
• Vulnerability
• Financial

7
8
Unmodeled Perils

• Significant losses historically from unmodeled
  perils, some known and unmodelled and some
  unknown. The table gives some background on
  estimated losses sustained in recent years this
  is by no means an exhaustive list
• Memory from region to region seems to be poor
• Did we learn from the 1999 events in France and
  apply that learning to Skandinavia ?

9
Model MalpracticePoor Data, Data Misuse and
Inappropriate Model use

• Poor data
• Incomplete
• Underestimated values
• Do the characteristics and quantum of data
  provided change dramatically every year ?
• Inappropriate (optimistic ?) values and risk
  characteristic coding.
• Examples in Katrina well known
• Safe to assume that we should not be surprised if
  similar problems highlighted following a large
  Japan Typhoon ?
• Confusing Precision with Accuracy
• More Data and higher resolution does not
  necessarily equal more accurate loss estimetes
• Hazard
• Does our ability to model the hazard support
  increasing precision as a deliverer of increasing
  accuracy ?
• Detailed vs. aggregate modeling
• Cross check detailed model runs against higher
  level analyses, explain the differences

10
Being Careful

• Absence of any judgment or reasonability testing
  on model outputs
• Over reliance on modelled results
• Apply learning from one region to another
• Think you have learned somethgn about NA
  hurricane vulnerability ?
• Where else might an application of this learning
  provide insight or change your view ?
• Japan Australia
• Tail reasonability
• Black Swans

11
Stress TestingLook for the Black Swan

• Look for the unusual. Just because you have
  never seen a Black Swan does not mean they dont
  exist.
• Stress tails with extreme hypothetical events
  look out side of the box
• Look for correlations
• Levy failure and storm surge in Katrina was not
  Black Swan event
• From a risk perspective it was well understood
  and predicted
• The industry loss (50bn) is a relatively high
  probability event for US Hurricane (15 to 20yrs)
  ?
• WTC was a Black Swan event
• Unexpected risk
• Although Industry loss was not a low probability
  event for Property
• Wcomp Loss is low probability

12
Portfolio ErrorInter and Intra Portfolio
Correlation

• Errors from the previous slides compound in a non
  linear fashion, then if you over lay an
  underestimation of intra and inter portfolio
  correlation things can get even worse.
• Intra-portfolio correlation
• Multi locations under blanket policy
• Inter-portfolio correlation
• Correlation in the tails often underestimated
• Treaty/Fac/Energy/Assets/Liability etc
• Deals/classes where no cat risk was assumed
  covered
• Communications between different risk entities in
  the same company
• The roll up of all of these errors has
  contributed to the wide gap between modelled and
  actual losses in recent large industry events
• (WTC, Katrina, Enron)

13
Inter Portfolio Correlation Kyrill 18th to
21st Jan 2007

• Kyril represented a lt10yr event for Europe
• Meteorologically
• Industry loss
• Property losses expected
• What surprises were there ?

Image courtesy UKMO
14
Kyrill - M/V MSC Napoli In Trouble21st June 2007

• Unexpected Correlated Losses In a High Frequency
  Event
• On shore property
• Marine
• Cargo
• Environmental Liability

15
NapoliWashed Up Cargo 22nd Jan 2007
16
summary

• Probabilistic modelling of risk (Cat and other)
  can be extremely powerful
• Needs to be used with care
• Understand limitations
• Use multiple views
• Get comfortable with uncertainty
• Constantly challenge

17
Further Information
RenaissanceRe Renaissance House 8 20 East
Broadway Pembroke, HM 19 Bermuda Tel (441)
295-4513 www.renre.com