Making Sense out of Metrics

1
Making Sense out of Metrics

• ISDA / PRMIA
• 17th August 2004

2
In 2002

• A global survey of 76 banks on the existence of
  formal KRI programs

Acknowledgement Raft International Limited
3
What is a KRI?
The number of fails has increased by 2 Is this
a KRI?
4
Some definitions

• Metric something observed or calculated that is
  used to show the presence or state of a condition
  or trend an instrument or gauge that measures
  something and registers the measurement
  something such as a light, sign, or pointer that
  gives information, for example about which
  direction to follow
• KRI Key Risk Indicator
• KPI Key Performance Indicator
• KCI Key Control Indicator
• KMI Key Management Indicator
• Overall, prefer the general term indicator

5
What is a Risk Indicator?

• Risk indicators are usually monitored over time
• Late trade processing in Bank X in May 2004
• London 9, New York 10, Singapore 9
• In which branch
• do we have a
• problem?

6
What is a Risk Indicator?

• So, Singapore is the problem!
• Is it?

Data without context may not expose the entire
problem!
7
The Choice from the Multitude

• A typical operation can identify hundreds of
  indicators
• Some are risk, others performance indicators
• Indicators relevance and weight change over time
• Some indicators are meaningless on their own

Graphics adapted from Reason, J. Managing the
Risks of Organizational Accidents", Aldershot
Ashgate, 1997
8
Other factors

• If a ratings agency is to rate operational
  exposure, how will they compare different
  organisations?
• How will regulators evaluate the effectiveness of
  different organisations operational risk
  capabilities?
• How does a business unit provide senior
  management quality information?
• Can the organisation use operational metrics to
  provide stakeholder information?

9
But, KRIs have been disappointing

• No means of consistently relate the occurrence of
  loss events and the location of
  problems/situations
• No means of classifying types of KRIs
• Plenty of data but no idea of its relevance
• No way to determine relevance
• No observable best practice
• No means of comparison, either internally or
  externally

10
The solution?

• An organisation needs a common language or
  framework which identifies areas of exposure and
  then allows..
• Metrics to be identified to measure, monitor and
  manage those exposures
• Data on losses, near misses and control failures
  to be recorded
• Ongoing assessment of the exposure
• Performance measurement around the exposure,
  including use of capital

11
KRI Study - Background

• Recognising the need, RiskBusiness developed a
  strawman framework for identifying risk points
  within an institution
• RMA and RiskBusiness co-sponsored Part I of the
  Study to ascertain the feasibility of using the
  framework as a KRI framework
• Seven institutions tested the framework in a risk
  mapping exercise

12
Initial Participants

• 7 banks participated in Part I
• Citigroup
• Deutsche Bank
• Dresdner Kleinwort Wasserstein
• JP Morgan Chase
• KeyCorp
• Royal Bank of Canada
• State Street
• ANZ and Abbey then joined to form the Study
  Steering Group for Part II

13
The Study

• Part I Proof of Concept
• Part II 3 primary activities
• Broaden participation, transfer experience, build
  industry risk profile
• Define KRI Library
• Develop detailed specifications
• Future
• Industry benchmarking
• Extend participation further

14
The KRI Framework
15
Validation

• Initial risk maps were evaluated to establish
  most risky risk categories and business
  functions
• Compared these to a similar evaluation of the QIS
  3 data and to the complete Fitch Risk First
  database
• Broad correlation, taking into account the nature
  of the two data sources

16
Participant risk map deviations

• Based on risk category

17
Participant risk map deviations

• Based on business function

18
Participants in Part II

• Abbey
• ABN Amro
• ABSA
• Acleda Bank
• Alliance and Leicester
• ANZ
• Bank Austria Creditanstaldt
• Bank Julius Bäer
• Bank of America
• Bank Rakvat Indonesia
• Bank Vontobel
• BNP Paribas
• Byblos Bank
• Capital One
• Citigroup
• Commerzbank
• De Lage Landen (sub of Rabobank)
• Deutsche Bank
• Dresdner Kleinwort Wasserstein

• Huntington National Bank
• Investec
• JP Morgan Chase
• KeyCorp
• Kookmin Bank
• Macquarie Bank
• Mizuho International
• National Australia Group
• National Bank of Canada
• Nomura International
• Northern Trust
• Peoples Bank
• Royal Bank of Canada
• Royal Bank of Scotland
• RZB
• San Paolo IMI
• SE Banken
• Southwest Bank of Texas
• Standard Bank of South Africa

Lead Participant
19
Defining Indicators

• Identify candidates for each risk point
• Evaluate candidates against qualifying criteria
  (effectiveness, comparability, ease of
  use/collection)
• Agree descriptions for each qualifier
• Prioritise nominated indicators
• Participant comment and review
• Generate detailed specifications, stored in KRI
  Library at www.KRIeX.org

20
Considering each risk point..
Before the problem has been generated
Look for Early Warning Flags
After the problem has been generated
Identify Problems in Progress
Potential loss events, include events that could
lead this or another institution to loss
Near misses, loss 0
Research Historical Events
0 lt Losses lt threshold
Losses gt threshold
21
Issues for consideration

• What is a KRI...KPIKCIor MIS?
• An indicator can perform multiple roles depending
  on who is using it
• What about scaling and aggregation?
• Do we scale then aggregate, or vice versa?
• How many indicators should a firm be monitoring?
• The quest for the Magic 10 the KRI Library
  has 1,600 indicators

22
Issues for consideration

• Top-down versus Bottom-Up
• Operations develop metrics for ongoing use,
  Management want information
• Combinations and clusters of indicators
• Experience has demonstrated that in many cases,
  it is groups of indicators which will provide the
  best management information
• Staff Turnover and Transaction Volumes and Error
  Rates and

23
Next Steps - Benchmarking

• Selected indicators will be driven totally by
  broad participant agreement
• Consists of participants delivering KRI data to
  centralised function
• Data will be anonymous
• Data will be collated, analysed and benchmark
  values calculated
• Participants have access to benchmarks for
  comparative purposes
• First submission expected in Q2, 2005

24
Next Steps KRI Library

• Next intake of participants into KRI Library
  starts in October 2004 currently have 68
  additional firms wishing to join
• Insurance KRI Study starts during Q4 2004
• Ongoing maintenance and extension to the Library

25
In Summary..

• A well-developed and structured indicator program
  can deliver quality management information and
  could possibly be used as an adjustor to
  capitalor at least as a measure of efficient and
  effective use
• Common language and standardisation is imperative
• The indicator program must deliver value at all
  levels

26
Contact details

• RiskBusiness International Limited
• URL www.riskbusiness.com
• Study URL www.kriex.org
• Mike Finlay, Managing Director Europe, Asia,
  Australia and Africa
• Telephone [phone]
• E-mail mike.finlay_at_riskbusiness.com