Critical phenomena in portfolio selection

1
Critical phenomena in portfolio selection

• Imre Kondor
• Collegium Budapest and Eötvös
• University
• European Conference on Complex Systems
• Paris, November 14-18, 2005

2
Contents

• I. Preliminaries portfolio selection, risk
  measures, the problem of noise
• II. Noise sensitivity of risk measures
  (variance, mean absolute deviation,
  expected shortfall, maximal loss)
• III. The feasibility problem, phase
  transition- like phenomena in portfolio
  selection

3
Coworkers

• Szilárd Pafka (Eötvös University, Budapest, now
  at CIB Bank, Budapest)
• Gábor Nagy (Debrecen University and CIB Bank,
  Budapest)

4
I. PRELIMINARIES
5
Portfolios

• A portfolio is a linear combination
• of random variables (returns on financial
  assets) with (not necessarily positive)
  weights that add up to unity
  
• The portfolios expected return is
  

6
Risk measures

• A risk measure is a quantitative characterization
  of our intuitive concept of risk (fear of
  uncertainty and loss).
• Risk is related to the stochastic nature of
  returns. It is (or should be) a convex functional
  on the joint probability distribution of returns.

7
Portfolio selection

• Rational portfolio selection is seeking a
  tradeoff between risk and expected return, by
  optimizing the risk functional over the weights,
  given the expected return and possibly other
  costraints.

8

• This task would be relatively straightforward if
  we knew the functional to be minimized.
• The appropriate choice may depend on the nature
  of data (e.g. on their asymptotics), and on the
  context (investment, risk management,
  benchmarking, index tracking, regulation, capital
  allocation).
• Whichever risk measure we choose, we will need
  observed data to construct it.

9
The problem of noise

• Even if returns formed a clean, stationary
  stochastic process, we could only observe finite
  time segments, therefore we never have sufficient
  information to completely reconstruct the
  underlying process. Our estimates will always be
  noisy.
• The problem of noise is more severe for large
  portfolios (size N) and relatively short time
  series (length T) of observations, and different
  risk measures are sensitive to noise to a
  different degree.
• We have to know how the decision error depends on
  N and T for a given risk measure

10
II. NOISE SENSITIVITY OF RISK MEASURES
11
To test the noise sensitivity of various risk
measures we use simulated data

• The rationale behind this is that in order to be
  able to compare the sensitivity of risk measures
  to noise, we better get rid of other sources of
  uncertainty, like non-stationarity. This can be
  achieved by using artificial data where we have
  total control over the underlying stochastic
  process. For simplicity, we use iid normal
  variables in the following.

12

• For such simple underlying processes the exact
  risk measure can be calculated in all four cases.
• To construct the empirical risk measure, we
  generate long time series, and cut out segments
  of length T from them, as if making observations
  on the market.
• From these observations we construct the
  empirical risk measure and optimize our portfolio
  under it.
• The ratio qo of the empirical and the exact risk
  measure is a measure of the estimation error due
  to noise.

13
The four risk measures considered here

• Variance
• Mean absolute deviation (MAD)
• Expected shortfall (ES)
• Maximal loss

14
Variance

• According to Markowitz classical theory the
  tradeoff between risk and reward can be realized
  by minimizing the variance
• over the weights, for a given expected return
• and budget

15
The minimal risk portfolio

• Expected returns are hardly possible (on
  efficient markets, impossible) to determine with
  any precision.
• In order to get rid of the uncertainties in the
  returns, we confine ourselves to considering the
  minimal risk portfolio only, that is, for the
  sake of simplicity, we drop the return
  constraint.

16
The weights of the minimal risk portfolio

• The minimal variance portfolio corresponds to the
  weights for which
• is minimal, given .
• The solutions is

17
Empirical covariance matrices

• The covariance matrix has to be determined from
  measurements on the market. From the returns
  observed at time t we get the estimator
• For a portfolio of N assets the covariance matrix
  has O(N²) elements. The time series of length T
  for N assets contain NT data. In order for the
  measurement be precise, we need N ltltT. Bank
  portfolios may contain hundreds of assets, and it
  is hardly meaningful to use time series longer
  than 4 years (T1000). Therefore, N/T ltlt 1 rarely
  holds in practice. As a result, there will be a
  lot of noise in the estimate, and the error will
  scale in N/T.

18
A measure of the effect of noise

• Assume we know the true covariance matrix and
• the noisy one . Then a natural, though not
  unique,
• measure of the impact of noise is
• where w are the optimal weights corresponding
• to and , respectively.

19
The true covariance matrix corresponding to iid
normal variables is the unit matrix

• Spectrum
• ? 1, N-fold degenerate
  
• Noise will split this
• into band

1
0
C
20
The corresponding empirical covariance matrix is
the Wishart matrix

• If N,T ?8 with N/T fixed, lt 1, then the
  spectrum of this empirical covariance matrix is
  the Wishart or Marchenko-Pastur spectrum
  (eigenvalue distribution)
• 
  where

21
Numerical measurements

• The relative error of the optimal portfolio
  is a random variable, fluctuating from sample to
  sample.
• The weights of the optimal portfolio also
  fluctuate.

22
The distribution of qo over the samples
23
The expectation value of qo as a function of N/T
24
The critical point N T

• The rank of the covariance matrix is minN,T
• For TltN the covariance matrix is positive
  semidefinite, the optimization is meaningless
• Approaching the critical point the relative error
  of the portfolio diverges
• The expectation value of the error can be shown
  to be

25
Instability of the weigthsThe weights of a
portfolio of N10 iid normal variables for a
given sample, T500
26
Instability of the weigthsThe weights of a
portfolio of N100 iid normal variables for a
given sample, T500
27
The distribution of weights in a given sample

• The optimization hardly determines the weights
  even far from the critical point.
• The standard deviation of the weights relative to
  their exact average value

28

• qw and qo are directly related in each sample
• Therefore, at the critical point the variance of
  the distribution of weights diverges with qo

29
Fluctuations of a given weight from sample to
sample, non-overlapping time-windows, N100, T500
30
Fluctuations of a given weight from sample to
sample, time-windows shifted by one step at a
time, N100, T500
31
All is not lost filtering procedures can save
the day, and allow us to penetrate even into the
region below the critical point TltN
32
Similar studies under mean absolute deviation,
expected shortfall and maximal loss
33
Mean absolute deviation (MAD)
Some methodologies (e.g. Algorithmics) use the
mean absolute deviation rather than the standard
deviation to characterize the fluctuation of
portfolios. The objective function to minimize is
then
instead of
MAD can be optimized by linear programming
34
Expected shortfall (ES)

• ES is the mean loss beyond a high threshold ß
  defined in probability (not in money). For
  continuous pdfs it is the same as the
  conditional expectation beyond the VaR quantile.
  ES is coherent (in the sense of Artzner et al.)
  and as such it is strongly promoted by a group of
  academics. In addition, Uryasev and Rockafellar
  have shown that its optimizaton can be reduced to
  linear programming for which extremely fast
  algorithms exist.

35
Maximal loss

• Maximal loss is ß 100 limit of ES, i.e. we
  select the worst return in time and minimize this
  over the weights
• subject to
• This risk measure is coherent, one of Acerbis
  spectral measures.

36
Compare the noise sensitivity of the following
risk measures standard deviation, absolute
deviation, expected shortfall (the latter at ß
70), and its extreme version, maximal loss (ß
100). For the sake of comparison we use the same
(Gaussian) input data of length T for each,
determine the minimal risk portfolio under these
risk measures and compare the error due to noise.
37
Relative error in portfolios optimized under
various risk measures
38
Instability of portfolio weights

• Similar trends can be observed if we look into
  the weights of the optimal portfolio the weights
  display a high degree of instability already for
  variance optimized portfolios, but this
  instability is even stronger for mean absolute
  deviation, expected shortfall, and maximal loss.

39
Instability of weights for various risk measures,
non-overlapping windows
40
Instability of weights for various risk measures,
overlapping weights
41
III. THE FEASIBILITY PROBLEM
42
The feasibility problem

• For T lt N, there is no solution to the portfolio
  optimization problem under any of the risk
  measures considered here.
• For T gt N, there always is a solution under the
  variance and MAD, even if it is of low quality
• for T not large enough.
• In contrast, under ES (and ML to be considered
  later), there may or may not be a solution for T
  gt N, depending on the sample. The probability of
  the existence of a solution goes to 1 only for
  T/N going to infinity.
• The problem does not appear if short selling is
  banned

43
Why is the existence of an optimum a random event?

• To get a feeling, consider maximal loss for
  NT2.
• The two planes
• intersect the plane of the budget constraint in
  two straight lines. If the slopes of these two
  straight lines are of opposite sign, then there
  is a solution, if they are of the same sign,
  there is not. It is easy to see that for
  elliptical distributions the probability of there
  being a solution is ½.

44
Maximal loss is always convex, but for a given
sample it may be unbounded
45
Probability of the feasibility of the minimax
problem

• For TgtN the probability of a solution (for an
  elliptical underlying pdf) is
• (The problem is isomorphic to some problems in
  operations research and random geometry Todd,
  M.J. (1991), Probabilistic models for linear
  programming, Math. Oper. Res. 16, 671-693. )
• For N and T large, p goes over into the error
  function.
• For N,T? 8, the transition becomes sharp at
  N/T1/2.

46
Probability of finding a solution for the minimax
problem
47
(No Transcript)
48
(No Transcript)
49
(No Transcript)
50
Feasibility of optimization under ES
Probability of the existence of an optimum under
CVaR. F is the standard normal distribution. Note
the scaling in N/vT.
51
For ES the critical value of N/T depends on the
threshold ß
52
(No Transcript)
53
With increasing N, T ( N/T fixed) the transition
becomes sharper and sharper
54
until in the limit N, T ?8 with N/T fixed we
get a phase boundary
55
As we approach this phase boundary, the relative
error in the portfolio diverges
56
The relative error as a function of the threshold
ß for fixed N/T
57
A wider context

• The feasibility problem is analogous to the
  family of algorithmic phase transitions
  discovered in some combinatorial optimization,
  random assignment, graph partitioning, and
  satisfiability problems recently. There the
  character of the algorithm changes (e.g. from
  polynomial to NP-complete) at a critical value of
  some parameter characterizing the problem, and
  the transition is accompanied by divergent
  fluctuations, run times, etc.

58
Concluding remarks

• Due to the large number of assets in typical bank
  portfolios and the limited amount of data, noise
  is an all pervasive problem in portfolio theory.
• It can be efficiently filtered by a variety of
  techniques from portfolios optimized under
  variance.
• For alternative risk measures filtering methods
  are less developed.
• The two coherent measures (ES, ML) we have
  studied display large sample-to-sample
  fluctuations and feasibility problems under
  noise.
• Portfolio optimization exhibits critical
  phenomena around a critical ratio of the size of
  the portfolio and the length of the time series