Upper Limits and Discovery in Search for Exotic Physics

1
Upper Limits and Discovery in Search for Exotic
Physics
Jan Conrad Royal Institute of Technology
(KTH) Stockholm
2
Outline

• Discovery
• Confidence Intervals
• The problem of nuisance parameters (systematic
  uncertainties)
• Averaging
• Profiling
• Analysis optimization
• Summary

3
General approach to claiming discovery
(hypothesis testing)

• Assume an alleged physics process characterized
  by a signal parameter s (flux of WIMPS, Micro
  Blackholes .... etc.)
• One can claim discovery of this process if the
  observed data is very unlikely to come from the
  null hypothesis , H0, being defined as
  non-existence of this process (s0). Very
  unlikely is hereby quantified as the
  signifcance probability asign, taken to be a
  small number (often 5 s 10-7).
• Mathematically this is done by comparing the
  p-value with asign and reject H0 if p value lt
  asign

Actually observed value of the test statistics
test statistics, T, could be for example ?2
4
P-values and the Neyman Pearson lemma

• Uniformly most powerful test statistic is the
  likelihood ratio
• For p-values, we need to know the
  null-distribution of T.
• Therefore it comes handy that asymptotically
• Often it is simply assumed that the
  null-distribution is ?2 but be careful !
• see e.g. J.C. , presented at NuFACT06,
  Irvine, USA, Aug. 2006
• L. Demortier, presented at
  BIRS, Banff, Canada, July 2006

5
Type I, type II error and power

• Type I error Reject H0, though it is
  true.
• Prob(Type I error) a
• Type II error Accept H0, though it is
  false
• Power 1 - ß 1 Prob(Type II error)
• In words given H1, what is the probability
  that we will reject H0 at given
• significance a ? In other words what is
  the probability that we detect H1 ?
• In designing a test, you want correct Type I
  error rate (this controls the number of false
  detections) and as large power as possible .

6
Why 5 s?

• traditional we have seen 3 s significances
  disappear (.we also have seen 5 s signficances
  disappear on the other hand .)
• Principal reasoning (here done for the LHC)
• LHC searches 500 searches each of which has 100
  resolution elements (mass, angle bins, etc.) ? 5
  x 104 chances to find something.
• One experiment False positive rate at 5 s ? (5
  x 104) (3 x 10-7) 0.015. OK !
• Two experiments
• Assume we want to produce lt 100 unneccessary
  theory papers
• ? allowable false positive rate 10.
• ? 2 (5 x 104) (1 x 10-4) 10 ? 3.7 s required.
  
• Required other experiment verification (1 x
  10-3)(10) 0.01 ? 3.1 s required.

It seems that the same reasoning would lead to
smaller required signficance probabilities for EP
searches in NT.
7
Confidence Intervals (CI)

• Instead of doing a hypothesis test, we might want
  to do a interval estimate on the parameter s with
  confidence level 100(1 a) (e.g. 90 )
• Bayesian
• Frequentist
• Invert by e.g. Neyman construction of confidence
  intervals (no time to explain)
• - special case 1 n 2 ? ? upper limit
• - special case 2 two sided/one sided limits
  depending
• on observation ? Feldman Cousins
• Confidence intervals are often used for
  hypothesis testing.

G. Feldman R. Cousins, Phys. Rev D573873-3889
See e.g. J.C. presented at NuFACT06, Irvine, USA,
Aug. 2006
K. S. Cranmer, PhyStat 2005, Oxford, Sept. 2005
8
Nuisance parameters1)

• Nuisance parameters are parameters which enter
  the data model, but which are not of prime
  interest (expected background, estimated
  signal/background efficiencies etc. pp., often
  called systematic uncertainties)
• You dont want to give CIs (or p-values)
  dependent on nuisance parameters ? need a way to
  get rid of them

1) Applies to both confidence intervals and
nuisance parameters
9
How to get rid of the nuisance parameters ?

• There is a wealth of approaches to dealing with
  nuisance parameters. Two are particularly common
• Averaging (either the likelihood or the PDF)
• Profiling (either the likelihood or the PDF)
• ... less common, but correct per construction
  fully frequentist, see e.g

Bayesian
G. Punzi, PHYSTAT 2005, Oxford, Sept. 2005
10
NT searches for EP why things are bad ..... and
good.

• Bad
• Low statistics makes the use of asymptotic
  methods doubtful
• systematic uncertainties are large.
• Good
• Many NT analyses are single channel searches with
  relatively few nuisance parameters
• ? rigorous methods are computationally feasible
  (even fully frequentist)

11
Coverage

• A method is said to have coverage (1-a) if, in
  infinitely many repeated experiments the
  resulting CIs include (cover) the true value in a
  fraction (1-a) of all cases (irrespective of what
  the true value is).
• Coverage is a necessary and sufficient condition
  for a valid CI calculation method

12
Averaging hybrid Bayesian confidence intervals

• Example PDF
• Perform Neyman-Construction with this new PDF (we
  will assume Feldman Cousins in the remainder of
  this talk)
• Treats nuisance parameters Bayesian, but performs
  a frequentist construction.

Integral is performed in true variables ? Bayesian
J.C, O. Botner, A. Hallgren, C. de los Heros
Phys. Rev D67012002,2003
R. Cousins V. Highland Nucl. Inst. Meth.
A320331-335,1992
13
Coverage of hybrid method.
Use Log-normal if large uncertainties !!!!!
(1- a)MC
true s
true s
F.Tegenfeldt J.C. Nucl. Instr.
Meth.A539407-413, 2005
J.C F. Tegenfeldt , PhyStat 05, Oxford, Sept.
2005, physics/0511055
14
Commercial break pole

• Bayesian treatment in FC ordering Neyman
  construction
• treats P(nes b)
• Consists of C classes
• Pole calculate limits
• Coverage coverage studies
• Combine combine experiments
• Nuisance parameters
• supports flat, log-normal and Gaussian
  uncertainties in efficiency and background
• Correlations (multi-variate distributions and
  uncorrelated case)
• Code and documentation available from
• http//cern.ch/tegen/statistics.html

J.C F. Tegenfeldt , Proceedings PhyStat 05,
physics/0511055
15
Example hybrid Bayesian in NTs

• From Daan Huberts talk (this conference)

with systematicswithout systematics
16
Profiling Profile Likelihood confidence
intervals
meas n, meas. b
MLE of b given s
MLE of b and s given observations
2.706
To extract limits
Lower limit
Upper Limit
17
From MINUIT manual

• See F. James, MINUIT Reference Manual, CERN
  Library Long Write-up D506, p.5
• The MINOS error for a given parameter is
  defined as the change in the value of the
  parameter that causes the F to increase by the
  amount UP, where F is the minimum w.r.t to all
  other free parameters.

Confidence Interval
Profile Likelihood
??2 2.71 (90), ??2 1.07 (70 )
18
Coverage of profile likelihood
Available as TRolke in ROOT ! Should be able to
treat common NT cases
(1- a)MC
W. Rolke, A. Lopez, J.C. Nucl. Inst.Meth A 551
(2005) 493-503
true s
19
Profile likelihood goes LHC.

• Basic idea calculate 5 s confidence interval and
  claim discovery if s 0 is not included.
• Straw-man model
• Typical b 100, ? 1 (? 10 sys. Uncertainty
  on b)

Size of side band region
- 35 events!!
- 17 events!!
K. S. Cranmer, PHYSTAT 2005, Oxford, Sept. 2005
20
Analysis optimisation

• Consider some cut-value t. Analysis is optimised
  defining a figure of merit (FOM). Very common
• Alternatively, optimize for most stringent upper
  limit. The corresponding figure of merit is the
  model rejection factor, MRF

Mean upper limit (only bg)
G. Hill K. Rawlins, Astropart. Phys.
19393-402,2003
21
In case of systematics ?

• Simplest generalizations one could think of
• In general, I do not think it makes a difference
  unless

NO !
Yes !
22
Optimisation for discovery and upper limit at the
same time ?

• Fix significance (e.g asign 5 s) and confidence
  level (e.g. 1-aCL 99 ). Then define
  sensitivity region in s by
• The FOM can be defined to optimize this quantity
  (e.g simple counting experiment)

Signal efficiency
Number of s (here assumed asign 1 aCL)
G. Punzi, PHYSTAT 2003, SLAC, Aug. 2003
23
Conclusions/Final Remarks

• Two methods to calculate CI and claim discovery
  in presence of systematic uncertainties have
  been discussed.
• The methods presented here are certainly suitable
  for searches for Exotic Physics with Neutrino
  Telescopes and code exists which works out of
  the box
• Remark the simplicity of the problem (single
  channel, small number of nuisance parameters)
  make even rigorous methods applicable
• Remark 2 the LHC example shows that for large
  signficances (discovery) hybrid Bayesian might be
  problematic.
• I discussed briefly the issue of sensitivity and
  analysis optimisation.

24

• Backup Slides

25
B0s ? µµ-
J.C F. Tegenfeldt , Proceedings PhyStat 05,
physics/0511055
26
Neyman construction
Exp 3
Exp 2
Exp 1
One additional degree of freedom ORDER in which
you inlcude the n into the belt
J. Neyman, Phil. Trans. Roy. Soc. London A, 333,
(1937)
27
Projection method with appropriate ordering.
Ordering function (Punzi, PhyStat05)
Poisson signal, Gauss eff. Unc (10 )
Can be any ordering in prime observable
sub-space, in this case Likelihood ratio (Feldman
Cousins) FC Profile
Average coverage
Max/Min coverage
s
28
FC ordering coverage
(1- a)MC
Calculated by Pseudo-experiments
Nominal coverage
true s
29
Some methods for p-value calculation

• Conditioning
• Prior-predictive
• Posterior-predictive
• Plug-In
• Likelihood Ratio
• Confidence Interval
• Generalized frequentist

30
Some methods for confidence interval calculation
(the Banff list)

• Bayesian
• Feldman Cousins with Bayesian treatment of
  nuisance parameters (Hybrid Bayesian)
• Profile Likelihood
• Modified Likelihood
• Feldman Cousins with Profile Likelihood
• Fully frequentist
• Empirical Bayes