Credit Model Performance Monitoring System

1
Credit Model Performance Monitoring System

• Oliver Brunke
• Kamyar K.Moud
• Credit Risk Analytics, CIRM, CIBC
• CSRSA Conference 2006, Niagara Falls

2
Credit Risk Analytics

• Modeling and Implementation of Basel for Retail
  division
• Development of Scorecards
• Consulting to business units on strategies

3
Agenda

• Background
• Review of Scorecards
• Requirements
• Architecture
• Analytics

4

• Credit Models gt Scorecards used for
• automated decisioning
• Performance gt Comparing outcomes with
• previously generated
• scores
• Monitoring gt On-going process

5
Motivation

• Assist business units in managing scoring models
  and support decisions
• Governance provide independent monitoring
• Regulatory compliance
• Basel II CP3 clause 349 section H
• OCC 2000 16
• OSFI

6
Objective

• Early warning of scorecard deterioration
• Regulatory Compliance
• Support decisions on scoring models
• Review usage of scoring models
• Reduce losses on business lines

7
Scope

• About 40 scorecards across the banks retail
  division
• A number of external scores
• Origin of Scorecards
• In-house developed
• Externally developed (Fair Issacc, etc.)
• Industry Standard scores (Equifax Beacon Score,
  TransUnion Empirica Score, etc.)

8
Usage of Scorecards
Scorecards
Score Banding Approach
Dual Matrix Approach
Vector Approach
Scorecard Score
High Score
Scorecard Score
Low Score
Bureau Score
High Score
Accept
High Score
Decision Rules 1
Band A
Reject
Reject
Accept
Accept
Decision Rules 2
Band B
Reject
Cut-off Score
Scorecard Score
Accept
Decision Rules 3
Band C
Reject
Accept
Reject
Accept
Decision Rules 4
Band D
High Score
Reject
Low Score
Low Score
9
Types of Scorecards

• Application scorecards
• Behavior scorecards
• Industry Standard Credit scores
• Specialized scores BNI, Fraud detection models

10
Inputs to Scorecards

• In general we need the following types of data
  for classifying accounts
• Customer Data
• Exclusion Data
• Credit Bureau Data
• Performance Data

11
Customer Data

• This refers to data that is typically supplied by
  the customer as part of an application for a
  particular product and that is used as input to
  the score. This data typically includes items
  such as
• Income
• Residential Status
• Assets/Liabilities
• Net Worth

12
Exclusion Data

• Data that is used to identify scorecard
  exclusions. Exclusions are accounts that were
  deliberately excluded from the scorecard
  population during development because the
  accounts are expected to display non-typical
  behaviour as a result of special circumstances
  such as
• Fraud (fraudulent applications, stolen cards,
  illegal activities)
• Death of customer
• Legal Action
• Special Treatment customers (VIP, bank employees)
• Administrative Issues
• Disputes

13
Performance Data

• On-going data allowing us to classify the
  performance of each account at that point in
  time. In general we have the following possible
  classifications
• Goods accounts that are in good standing
• Bads charge-offs, dwo, delinquents
• Indeterminate accounts that are slightly
  delinquent but may still be able to recover
• Insufficient accounts that do not have enough
  history to make a clear classification

14
Miscellaneous Data

• Processing credit facility utilization, final
  adjudication outcome
• Sub-population Identification
• Financial account balances, payment amounts

15
Current Baseline Populations

• Current Population a snapshot of the account
  population at the time of reporting, including
  all the scorecard data discussed previously as
  well as account status information
• Baseline Population a fixed dataset that is
  being used as a benchmark to measure against.
• Ideally get the account population that was used
  to develop or validate the scorecard as well as
  the corresponding performance data
• Alternatively use a defined period of live
  scorecard data and corresponding outcomes as
  proxy for development data

16
Behavior Scorecards
Performance Data
Score
Outcome
Characteristics Data
Financial Data
Q3 2004
Q4 2004
Q1 2005
Q2 2005
Q3 2005
Q4 2005
Q1 2006
Q2 2006
Future Batches
June
May
June
April
12 Months time lag between scoring date and
outcome
12 Months performance period to be used for next
reporting cycle
17
Application Scorecard
Performance Data
Score
Outcome
Characteristics Data
Ancillary Data
Q3 2004
Q4 2004
Q1 2005
Q2 2005
Q3 2005
Q4 2005
Q1 2006
Q2 2006
Future Batches
Initial Exclusion Period (6 months)
Monitoring Period (18 months)
Monitoring Period to be used for next reporting
cycle
Initial Exclusion to be used for next reporting
cycle
18
Architecture
Automated processing with minimal manual
intervention
manual interpretation
Flat Files from LOB source systems
Raw Monitoring Reports
Interpreted Reports
Generic Analytics Module
LOB1
Target Database
Data Validation
LOB2
LOB3
Preprocessed classified datasets for each
scorecard
Feedback to Business Unit
19
Generic Classification Scheme
20
Generic Classification Tree
Total Population
Inclusions
Exclusions
Accept
Declined
Actual Accept
Actual Declined
Actual Accept
Actual Declined
Booked
Un-booked
Booked
Un-booked
Bads
Bads
Indeterminate
Indeterminate
Insufficient
Insufficient
Goods
Goods
21
Analytics

• Performance
• Characteristic
• Group

22
Population Stability

• Population Stability Index (PSI)
• Quantifies the difference by measuring the
  distance (distributional shift) between two
  sample distributions current and baseline
• Thresholds
• A PSI of 0.1 or less indicates little or no
  difference between two score distributions
• An index from 0.1 to 0.25 indicates that some
  change has taken place, but it is too small to
  determine whether it is an isolated incident or a
  part of a longer trend
• An index above 0.25 signifies a large shift. The
  population should be looked on characteristic-by-c
  haracteristic basis to ascertain potential causes

23
Model Predictability Power
Gini coefficient defined as G 2 (Area Under
the Curve) 1. This has the useful property that
the perfect model will have G1, and the random
model will have G0.

24
Goodness-of-fit

• Kolmogorov-Smirnov (K-S) Statistic measures
  the maximum difference between two distributions.
  We measure K-S for the cumulative score
  distributions of
• Baseline and current samples
• Goods and Bads

25
Rank Ordering Power
Log Odds vs. Score graph shows the rank ordering
power of scorecard.
26
Trending of Performance
Gini coefficient trend graph
Bad rate trend graph
27
Characteristic Analysis

• Detailed insight into the population dynamics
• Characteristics contribution to the average score
  change

28
Scorecard Group Analysis
Transition Matrix Groups of behavior scorecards
designed to work together
Current period
Previous period
29
Scorecard Group Analysis (cont.)

• Sub-population Gini coefficients
• Rank order the scorecards using bad rate
• Calculate the Gini coefficient for each
  sub-population

A
New
B
Above
C
Same
D
D
E
Below
F
G
30
Challenges

• Availability of Scorecard Information
• Documentation on models
• Development data
• Proprietary information
• Availability of Data
• Time Lag to get Performance Data
• Volume of Data
• Interpretation of Results

31
QA
Questions?