G' Cowan

1
Input from Statistics Forum for Exotics
ATLAS Exotics Meeting CERN/phone, 22 January, 2009
Glen Cowan Physics Department Royal Holloway,
University of London g.cowan_at_rhul.ac.uk www.pp.rhu
l.ac.uk/cowan Input from Eilam Gross, Samir
Ferrag
2
Intro
Contributions to Statistics Forum from Exotics
group over last year have raised questions in
several areas methods for setting limits,
establishing discovery, methods for
incorporating systematic uncertainties, approval
of software, methods, Purpose of this talk is to
address some of these issues as part of an
ongoing discussion (not yet definitive
answers). Some pointers to info -- StatForum
Webpage twiki.cern.ch/twiki/bin/viewauth/AtlasP
rotected/StatisticsTools including notes in
Statistics FAQ and also 1st half of the Higgs
Combination chapter of CSC Book (p 1480).
3
Statistics Forum Website FAQ
Some general items PDG Chapters, Pedestrian's
guide, Glossary, ...
4
Statistics Forum FAQ Notes
This is a living document
5
Statistics Forum FAQ Notes
The FAQ consists of a collection of notes on
specific questions use cases, examples,
... Bayesian methods for ATLAS Higgs search
(GC) Comparison of significance from profile and
integrated likelihoods (GC, EG) Discovery
significance with statistical uncertainty in the
background estimate (EG, OV, GC) Error
analysis for efficiency (GC) How to measure
efficiency (DC) MC statistical errors in ML fits
(GC) Covariance matrix for histogram made using
seed events (GC) If you have a note which you
think should be included here, or if you are
interested to write such a note or comment on a
note or request a note on a specific subject
please let us know.
6
Some statistics issues in searches

• Define appropriate test variable(s).
• Cut-based
• Multivariate method (Fisher, NN, BDT, SVM,...)
• (2) Determine its (their) distribution(s) under
  hypothesis of
• background only, background (parametrized)
  signal, ...
• Data-driven or MC, parametric or histogram, ...
• Quantify systematic uncertainties.
• (3) Measure the distribution in data quantify
  level of
• agreement between data and predictions (results
• in limits, discovery significance).
• Exclusion limits (Neyman, CLs, Bayesian)
• Discovery significance (frequentist, Bayesian)

7
Multivariate methods brief comment
Most searches in the CSC book use physically
motivated cut-based selection analysis easy to
understand and easy to spot anomalous behaviour.
But by a nonlinear decision boundary between
signal and background leads in general to higher
sensitivity.
Many new tools on market (see e.g. TMVA manual)
Boosted Decision Trees, K-Nearest
Neighbour/Kernel-based Density
Estimation, Support Vector Machines,.. Multivariat
e analysis suffers some loss of transparency
but... 5s from MVA plus e.g. 4s from
cuts could win the race.
8
Search formalism
Define a test variable whose distribution is
sensitive to whether hypothesis is
background-only or signal background. E.g.
count n events in signal region
expected signal
expected background
strength parameter m s s/ ss,nominal
events found
9
Search formalism with multiple bins (channels)
Bin i of a given channel has ni events,
expectation value is
m is global strength parameter, common to all
channels. m 0 means background only, m 1 is
nominal signal hypothesis.
Expected signal and background are
btot, qs, qb are nuisance parameters
10
Subsidiary measurements for background
One may have a subsidiary measurement to
constrain the background based on a control
region where one expects no signal. In bin i of
control histogram find mi events expectation
value is
where the ui can be found from MC and q includes
parameters related to the background (mainly
rate, sometimes also shape). In some measurements
there may be no explicit subsidiary measurement
but the sidebands around a signal peak
effectively play the same role in constraining
the background.
11
Likelihood function
For an individual search channel, ni
Poisson(msibi), mi Poisson(ui). The
likelihood is
Here q represents all nuisance parameters
Parameter of interest
For multiple independent channels there is a
likelihood Li(m,qi) for each. The full
likelihood function is
12
Systematics "built in" as long as some point in
q-space "truth"
13
p-values
Quantify level of agreement between data and
hypothesis H with p-value Prob(data
with compatibility with H when
compared to the data we got H )
probability, under assumption of H, to obtain
data as bizarre as the data we got (or more
so) ? probability that H is true (!!!)
14
Significance from p-value
Define significance Z as the number of standard
deviations that a Gaussian variable would
fluctuate in one direction to give the same
p-value.
TMathProb
TMathNormQuantile
15
When to publish
HEP folklore is to claim discovery when p 2.9
10-7, corresponding to a significance Z 5. This
is very subjective and really should depend on
the prior probability of the phenomenon in
question, e.g., phenomenon
reasonable p-value for discovery D0D0
mixing 0.05 Higgs 10-7 (?) Life on
Mars 10-10 Astrology 10-20
Note some groups have defined 5s to refer to a
two-sided fluctuation, i.e., p 5.7 10-7
16
Distribution of qm
So to find the p-value we need f(qmm) . Method
1 generate toy MC experiments with hypothesis m,
obtain at distribution of qm. OK for e.g. 103
or 104 experiments, 95 CL limits. But for
discovery usually want 5s, p-value 2.8 10-7,
so need to generate 108 toy experiments (for
every point in param. space). Method 2 Wilk's
theorem says that for large enough
sample, f(qmm) chi-square(1 dof) This is the
approach used in the Higgs Combination
exercise not yet validated to 5s level. If/when
we are fortunate enough to see a signal, then
focus MC resources on that point in parameter
space.
17
Significance from qm
If we take f(qmm) c2 for 1dof, then the
significance is (see Higgs combo note)
For n Poisson (msb) with b known, testing m0
gives
To quantify sensitivity give e.g. expected Z
under sb hypothesis
18
Likelihood ratio Lsb/Lb
An alternative (in simple cases equivalent) test
variable is
Fast Fourier Transform method to find
distribution derives n-event distribution from
that of single event with FFT. Hu and Nielson,
physics/9906010 Solves "5-sigma problem". Used at
LEP -- systematics treated by averaging the
likelihoods by sampling new values of nuisance
parameters for each simulated experiment
(integrated rather than profile likelihood).
19
Determining distributions systematics
E.g. Mll distribution from Z'?dilepton search
(CSC Book p 1709), uses 4-parameter function for
signal. Sidebands provide estimate of
background. So nothing in real analysis from MC,
but...
Still should consider some systematic due to fact
that assumed parametric functions not
perfect. General approach include more
parameters making the model more flexible, so
that for some point in the enlarged parameter
space, model Nature (or difference negligible).
20
A general strategy (see attached note)
Suppose one needs to know the shape of a
distribution. Initial model (e.g. MC) is
available, but known to be imperfect. Q How can
one incorporate the systematic error arising
from use of the incorrect model? A Improve the
model. That is, introduce more adjustable
parameters into the model so that for some point
in the enlarged parameter space it is very close
to the truth. Then use profile the likelihood
with respect to the additional (nuisance)
parameters. The correlations with the nuisance
parameters will inflate the errors in the
parameters of interest. Difficulty is deciding
how to introduce the additional parameters.
21
A simple example
True model (Nature)
Data
0th order model
The naive model (a) could have been e.g. from MC
(here statistical errors suppressed point is to
illustrate how to incorporate systematics.)
22
Comparison with the 0th order model
The 0th order model gives qn 258.8, p 6
10-30
23
Enlarging the model
Here try to enlarge the model by multiplying the
0th order distribution by a function s
where s(x) is a linear superposition of Bernstein
basis polynomials of order m
24
Bernstein basis polynomials
25
Enlarging the parameter space
Using increasingly high order for the basis
polynomials gives an increasingly flexible
function. At each stage compare the p-value to
some threshold, e.g., 0.1 or 0.2, to decide
whether to include the additional parameter. Now
iterate this procedure, and stop when the data do
not require addition of further parameters based
on the likelihood ratio test. Once the enlarged
model has been found, simply include it in any
further statistical procedures, and the
statistical errors from the additional parameters
will account for the systematic uncertainty in
the original model.
26
Fits using increasing numbers of parameters
Stop here
27
Setting limits
Method outlined in the CSC Higgs Combo "CLsb
method", i.e., for the hypothesized m (e.g. 1)
compute the p-value
m is excluded at CL0.95 if p lt a 0.05, and if
m 1 is excluded, the corresponding point in
parameter space for the signal model is
excluded. E.g. present expected limit on m vs
mass parameter.
28
Setting limits CLs
Alternative method (from Alex Read at LEP)
exclude m 1 if
where
This cures the problematic case where the one
excludes parameter point where one has no
sensitivity (e.g. large mass scale) because of a
downwards fluctuation of the background. But
there are perhaps other ways to get around this
problem, e.g., only exclude if both observed and
expected p-value lt a.
29
Comment on validation procedures for methods
Ongoing discussions on methodology Ideal is to
use several methods (profile likelihood,
Bayesian, CLs,...) for each result. Formal
procedures still evolving, but if you are
going to use a novel statistical technique,
please come give a talk about it at the
Statistics Forum.
30
Comment on software tools
Summer 08 agree to develop RooStats as common
framework. Keep eye on ability to carry out
independent validation. Key players Kyle
Cranmer (ATLAS) Gregory Schott (CMS) Wouter
Verkerke (RooFit) Lorenzo Moneta (Root) Work
currently very active (and help needed).
31
Summary
Current areas of activity include Development
of profile likelihood, CLs, Bayesian methods
for searches (including systematics) Combination
tools (e.g. Higgs combination) RooStats
software effort, Multivariate methods,
... Statistics forum wants to increase active
dialogue with the physics groups. If you are
using a novel procedure or want to discuss a
statistical method, please contact us.
32
Extra slides
33
Physics Group / StatForum interaction
Eilam Gross, 8.12.08
34
Questions from Luis Flores, 24 September, 2008