PHYSTAT05

1
PHYSTAT05 Statistical Problems in Particle
Physics, Astrophysics and Cosmology
Louis Lyons
CDF
Oxford, November 2005
2
(No Transcript)
3

• History of Conferences
• Overview of PHYSTAT05
• Specific Items
• Bayes and Frequentism
• Software
• Goodness of Fit
• Dos and Donts with Likelihoods
• Systematics
• Signal Significance
• Where are we now ?

PHYSTAT05
4
HISTORY
Where CERN Fermilab Durham SLAC, Oxford
When Jan 2000 March 2000 March 2002 Sept 2003 Sept 2005
Issues Limits Limits Wider range of topics Wider range of topics
Physicists Particles Particles 3 astrophysicists Particles 3 astrophysicists Particles Astro Cosmo
Statisticians 3 3 2 Many
5
Issues

• Bayes versus Frequentism
• Limits, Significance, Future Experiments
• Blind Analyses
• Likelihood and Goodness of Fit
• Multivariate Analysis
• Unfolding
• At the pit-face
• Systematics and Frequentism

6
Talks at PHYSTAT05

• 7 Invited talks by Statisticians
• 9 Invited talks by Physicists
• 38 Contributed talks
• 8 Posters
• Panel Discussion
• 3 Conference Summaries
• Underlying much of the discussion
• Bayes and Frequentism
• (Bletchley Park, Holywell Concert)
• 90 participants

7
Invited Talks by Statisticians
David Cox Keynote Address Bayesian,
Frequentists
Physicists Steffen Lautitzen Goodness of
Fit Jerry Friedman Machine Learning Susan Holmes
Visualisation Peter Clifford Time
Series Mike Titterington Deconvolution Nancy
Reid Conference Summary (1/3)
8
Invited Talks by Physicists
Bob Cousins Nuisance Parameters for Limits Kyle
Cranmer LHC discovery Alex Szalay
Astro Terabytes Jean-Luc Starck Mutiscale
geometry Jim Linnemann Statistical Software for
Particle Physics Bob Nichol Statistical Software
for Astro Stephen Johnson Historical Transits of
Venus Andrew Jaffe Conference Summary
(Astro) Gary Feldman Conference Summary
(Particles)
9
Bayes versus Frequentism
Old controversy Bayes 1763
Frequentism 1937 Both analyse data (x) ?
statement about parameters ( ) e.g. Prob (
) 90 but
very different interpretation Both use Prob (x
)
10
We need to make a statement about
Parameters, given Data
The basic difference between the two Bayesian
Probability (parameter, given data)
Anathema to a Frequentist! Frequentist
Probability (data, given parameter)
(likelihood function)
11
PROBABILITY
MATHEMATICAL Formal Based on Axioms
FREQUENTIST Ratio of frequencies as n?
infinity Repeated identical trials Not
applicable to single event or physical
constant BAYESIAN Degree of belief Can be
applied to single event or physical
constant (even though these have unique
truth) Varies from person to person
Quantified by fair bet
12
Bayesian
Bayes Theorem
posterior
likelihood
prior
Problems P(param) True or False
Degree of belief
Prior What functional form?
Flat? Which
variable? Unimportant when data
overshadows prior Important for limits
13
Data overshadows Prior
Prior L
MZ 91188 2
MeV
14
Data? upper limit on signal s

• Choice of prior affects limit
• 
  prior1/s
• L prior L
• s?
  s?

15
Mass squared of ?e
L
M2?? Prior
N.B. Posterior

16
P (DataTheory) P (TheoryData)
HIGGS SEARCH at CERN
Is data consistent with Standard Model?
or with Standard Model Higgs?
End of Sept 2000 Data not very consistent with
S.M. Prob (Data S.M.) lt 1 valid
frequentist statement Turned by the press into
Prob (S.M. Data) lt 1 and therefore Prob
(Higgs Data) gt 99 i.e. It is almost certain
that the Higgs has been seen
17
P (DataTheory) P (TheoryData)
Theory male or female Data pregnant or not
pregnant
P (pregnant female) 3 but P (female
pregnant) gtgtgt3
18
Frequentist Neyman Construction

• µ
• 
  
• 
  x
• 
  x0
• µ Theoretical parameter
• x Observation NO PRIOR
  INVOLVED

19
at 90 confidence
and known, but random unknown, but fixed Probability statement about and
Frequentist
and known, and fixed unknown, and random Probability/credible statement about
Bayesian
20
Bayes versus Frequentism
Bayesian Frequentist
Basis of method Bayes Theorem ? Posterior probability distribution Uses pdf for data, for fixed parameters
Meaning of probability Degree of belief Frequentist definition
Prob of parameters? Yes Anathema
Needs prior? Yes, Yes, Yes No
Choice of interval? Yes Yes (except FC)
Data considered Only data you have . other possible data
Likelihood principle? Yes No
21
Bayes versus Frequentism
Bayesian Frequentist
Ensemble of experiment No Yes (but often not explicit)
Final statement Posterior probability distribution Parameter values ? Data is likely
Unphysical/ empty ranges Excluded by prior Can occur
Systematics Integrate over prior Extend dimensionality of frequentist construction
Coverage Unimportant Built-in
Decision making Yes (uses cost function) Not useful


22
Bayesianism versus Frequentism
Bayesians address the question everyone is
interested in, by using assumptions no-one
believes
Frequentists use impeccable logic to deal with
an issue of no interest to anyone
23
Statistical Software

• Linnemann Software for Particles
• Nichol Software for Astro
• Le Diberder sPlot
• Paterno R
• Kreschuk ROOT
• Verkerke RooFit
• Pia Goodness of Fit
• Buckley CEDAR
• Narsky StatPatternRecognition
• Recommendation of Statistical Software Repository
  at FNAL

24
Goodness of Fit

• Basic problem
• Very general applicability,
  but
• Requires binning, with gt 5..20 events per
  bin. Prohibitive with sparse data in several
  dimensions.
• Not sensitive to signs of deviations

X ?
K-S and related tests overcome these, but work in
1-D So, need something else.
25
Goodness of Fit Talks
Lauritzen Invited talk Yabsley GOF and sparse
multi-D data Ianni GOF and sparse multi-D
data Raja GOF and L Gagunashili and
weighting Pia Software Toolkit for Data
Analysis Block Rejecting outliers Bruckman
Alignment Blobel Tracking
26
Goodness of Fit
Gunter Zech Multivariate 2-sample test based
on logarithmic
distance function See also Aslan Zech,
Durham Conf., Comparison of different goodness
of fit tests R.B. DAgostino M.A. Stephens,
Goodness of fit techniques, Dekker (1986)
27
(No Transcript)
28
DOS AND DONTS WITH L

• NORMALISATION OF L
• JUST QUOTE UPPER LIMIT
• ?(ln L) 0.5 RULE
• Lmax AND GOODNESS OF FIT
• BAYESIAN SMEARING OF L
• USE CORRECT L (PUNZI EFFECT)

29
NORMALISATION OF L
MUST be independent of m
data param e.g. Lifetime fit to t1,
t2,..tn
INCORRECT
t
30
2) QUOTING UPPER LIMIT We observed no
significant signal, and our 90 conf upper limit
is .. Need to specify method e.g. L
Chi-squared (data or theory error)
Frequentist (Central or upper limit)
Feldman-Cousins Bayes with prior const,
Show your L 1) Not always practical
2) Not sufficient for frequentist methods
31
90 C.L. Upper Limits
m
x
x0
32
?lnL -1/2 rule

• If L(µ) is Gaussian, following definitions of s
  are equivalent
• 1) RMS of L(µ)
• 2) v(-d2L/dµ2)
• 3) ln(L(µs) ln(L(µ0)) -1/2
• If L(µ) is non-Gaussian, these are no longer the
  same
• Procedure 3) above still gives interval that
  contains the true value of parameter µ with 68
  probability
• Heinrich CDF note 6438 (see CDF Statistics
  Committee
• 
  Web-page)
• Barlow Phystat05

33
COVERAGE How
often does quoted range for parameter include
params true value? N.B. Coverage is a property
of METHOD, not of a particular exptl
result Coverage can vary with Study coverage
of different methods of Poisson parameter
, from observation of number of events n Hope
for
100
Nominal value
34
COVERAGE If true for all
correct coverage
Plt for some undercoverage

(this is serious !)
Pgt for some overcoverage
Conservative Loss of rejection power
35
Coverage L approach (Not frequentist)
P(n,µ) e-µµn/n! (Joel Heinrich CDF note
6438) -2 ln?lt 1 ? P(n,µ)/P(n,µbest)
UNDERCOVERS
36
Frequentist central intervals, NEVER
undercovers(Conservative at both ends)
37
Feldman-Cousins Unified intervalsFrequentist,
so NEVER undercovers
38
Probability orderingFrequentist, so NEVER
undercovers
39

• (n-µ)2/µ ? 0.1 24.8
  coverage?
• NOT frequentist Coverage 0 ? 100

40
Lmax and Goodness of Fit?
Find params by maximising L So larger L better
than smaller L So Lmax gives Goodness of Fit??
Great?
Good?
Bad
Monte Carlo distribution of unbinned Lmax
Frequency
Lmax
41

• Not necessarily
  pdf
• L(data,params)
• 
  
• fixed vary
  L
• Contrast pdf(data,params) param
• vary fixed
• 
  
• 
  
  data
• e.g.
• Max at t 0
  
  Max at
• p
  L
• t
  ?
  

42
Example 1 Fit exponential to times t1, t2 ,t3
. Joel Heinrich, CDF 5639 L
lnLmax -N(1 ln tav) i.e. Depends only on
AVERAGE t, but is INDEPENDENT OF DISTRIBUTION OF
t (except for..) (Average t is a
sufficient statistic) Variation of Lmax in Monte
Carlo is due to variations in samples average t
, but NOT TO BETTER OR WORSE FIT

pdf Same average t
same Lmax

t


43

Example 2 L

cos ? pdf (and likelihood) depends
only on cos2?i Insensitive to sign of cos?i So
data can be in very bad agreement with expected
distribution e.g. all data with cos? lt 0 and
Lmax does not know about it. Example of general
principle
44
Example 3 Fit to Gaussian with variable µ, fixed
s lnLmax N(-0.5 ln2p lns) 0.5 S(xi
xav)2 /s2 constant
variance(x) i.e. Lmax depends only on
variance(x), which is not relevant for fitting µ
(µest xav) Smaller than expected
variance(x) results in larger Lmax
x

x Worse fit,
larger Lmax Better
fit, lower Lmax
45
Lmax and Goodness of
Fit? Conclusion L has sensible properties with
respect to parameters
NOT with respect to data Lmax within Monte
Carlo peak is NECESSARY
not SUFFICIENT (Necessary
doesnt mean that you have to do it!)
46
Binned data and Goodness of Fit using L-ratio
ni
L µi
Lbest
x lnL-ratio lnL/Lbest
large µi -0.5c2 i.e.
Goodness of Fit µbest is independent of
parameters of fit, and so same parameter values
from L or L-ratio
Baker and Cousins, NIM A221 (1984)
437
47
L and pdf

• Example 1 Poisson
• pdf Probability density function for observing
  n, given µ
• P(nµ) e -µ µn/n!
• From this, construct L as
• L(µn) e -µ µn/n!
• i.e. use same function of µ and n, but
  . . . . . . . . . . pdf
• for pdf, µ is fixed, but
• for L, n is fixed
  µ L
• 
  
• 
  n
• N.B. P(nµ) exists only at integer non-negative n
• L(µn) exists only as continuous fn of
  non-negative µ

48
Example 2 Lifetime distribution pdf
p(t?) ? e -?t So L(?t) ? e ?t
(single observed t) Here both t and ? are
continuous pdf maximises at t 0 L maximises at
? t N.B. Functional form of P(t) and L(?) are
different Fixed ?

Fixed t p
L
t
?
49
Example 3 Gaussian N.B. In this case,
same functional form for pdf and L So if you
consider just Gaussians, can be confused between
pdf and L So examples 1 and 2 are useful
50
Transformation properties of pdf and L

• Lifetime example dn/dt ? e ?t
• Change observable from t to y vt
• So (a) pdf changes, BUT
• (b)
• i.e. corresponding integrals of pdf are INVARIANT

51
Now for Likelihood When parameter changes from ?
to t 1/? (a) L does not change dn/dt 1/t
exp-t/t and so L(tt) L(?1/tt) because
identical numbers occur in evaluations of the two
Ls BUT (b) So it is NOT meaningful to
integrate L (However,)
52

• CONCLUSION
• NOT recognised
  statistical procedure
• Metric dependent
• t range agrees with tpred
• ? range inconsistent
  with 1/tpred
• BUT
• Could regard as black box
• Make respectable by L Bayes
  posterior
• Posterior(?) L(?) Prior(?)
  and Prior(?) can be constant

53
pdf(t?) L(?t)
Value of function Changes when observable is transformed INVARIANT wrt transformation of parameter
Integral of function INVARIANT wrt transformation of observable Changes when param is transformed
Conclusion Max prob density not very sensible Integrating L not very sensible
54
(No Transcript)
55
Getting L wrong Punzi effect

• Giovanni Punzi _at_ PHYSTAT2003
• Comments on L fits with variable resolution
• Separate two close signals, when resolution s
  varies event by event, and is different for 2
  signals
• e.g. 1) Signal 1 1cos2?
• Signal 2 Isotropic
• and different parts of detector give
  different s
• 2) M (or t)
• Different numbers of tracks ?
  different sM (or st)

56
Events characterised by xi and si A events
centred on x 0 B events centred on x
1 L(f)wrong ? f G(xi,0,si) (1-f)
G(xi,1,si) L(f)right ? fp(xi,siA) (1-f)
p(xi,siB)
p(S,T) p(ST) p(T)
p(xi,siA) p(xisi,A) p(siA)
G(xi,0,si)
p(siA) So L(f)right ?f G(xi,0,si) p(siA)
(1-f) G(xi,1,si) p(siB) If p(sA)
p(sB), Lright Lwrong but NOT otherwise
57

• Giovannis Monte Carlo for A G(x,0, sA)
• 
  B G(x,1, sB)
• 
  fA 1/3
• 
  Lwrong
  Lright
• sA sB
  fA sf
  fA sf
• 1.0 1.0
  0.336(3) 0.08 Same
• 1.0 1.1 0.374(4)
  0.08 0. 333(0) 0
• 1.0 2.0 0.645(6)
  0.12 0.333(0) 0
• 1 ? 2 1.5 ?3
  0.514(7) 0.14 0.335(2) 0.03
• 1.0 1 ? 2
  0.482(9) 0.09 0.333(0) 0
• 1) Lwrong OK for p(sA) p(sB) , but
  otherwise BIASSED
• 2) Lright unbiassed, but Lwrong biassed
  (enormously)!
• 3) Lright gives smaller sf than Lwrong

58
Explanation of Punzi bias
sA 1 sB 2
A events with s 1
B events with s
2 x ?

x ? ACTUAL DISTRIBUTION
FITTING FUNCTION

NA/NB variable, but same for A
and B events Fit gives upward bias for NA/NB
because (i) that is much better for A events
and (ii) it does not hurt too much for B events
59
Another scenario for Punzi problem PID
A B
p K M

TOF Originally Positions of peaks constant
K-peak ? p-peak at large momentum si
variable, (si)A (si)B si
constant, pK pp COMMON FEATURE
Separation/Error Constant
Where else?? MORAL Beware of
event-by-event variables whose pdfs do not
appear in L
60
Avoiding Punzi Bias

• Include p(sA) and p(sB) in fit
• (But then, for example, particle
  identification may be determined more by momentum
  distribution than by PID)
• OR
• Fit each range of si separately, and add (NA)i ?
  (NA)total, and similarly for B
• Incorrect method using Lwrong uses weighted
  average of (fA)j, assumed to be independent of j
• Talk by Catastini at PHYSTAT05

61
SYSTEMATICS
For example
we need to know these, probably from
other measurements (and/or theory) Uncertainties
?error in
Physics parameter
Observed
for statistical errors
Some are arguably statistical errors
Shift Central Value Bayesian Frequentist Mixed
62
PHYSTAT05 Nuisance params/Limits/Discovery

• Cousins Limits and Nuisance Params
• Reid Respondent
• Punzi Frequentist multi-dimensional
  ordering rule
• Tegenfeldt Feldman-Cousins Cousins-Highland
• Rolke Limits
• Heinrich Bayes limits
• Bityukov Poisson situations
• Hill Limits v Discovery (see Punzi _at_
  PHYSTAT2003)
• Cranmer LHC discovery and nuisance parameters

63
Shift Nuisance Parameters
Simplest Method Evaluate using
and Move nuisance parameters (one at a time)
by their errors ? If nuisance parameters are
uncorrelated, combine these contributions in
quadrature ? total systematic
64
Bayesian
Without systematics
prior
With systematics
Then integrate over LA and b
65
If constant and
truncated Gaussian TROUBLE!
Upper limit on from
Significance from likelihood ratio for
and
66
Frequentist
Full Method Imagine just 2 parameters
and LA and 2 measurements N
and M
Physics
Nuisance
Do Neyman construction in 4-D Use observed N and
M, to give Confidence Region for LA and
68
LA
67
Then project onto axis This results in
OVERCOVERAGE
Aim to get better shaped region, by suitable
choice of ordering rule
Example Profile likelihood ordering
68
Full frequentist method hard to apply in several
dimensions Used in 3 parameters For
example Neutrino oscillations (CHOOZ)
Normalisation of data
Use approximate frequentist methods that reduce
dimensions to just physics parameters e.g.
Profile pdf i.e.
Contrast Bayes marginalisation Distinguish
profile ordering
Properties being studied by Giovanni Punzi
69
Talks at FNAL CONFIDENCE LIMITS WORKSHOP
(March 2000) by Gary Feldman Wolfgang Rolk
p-ph/0005187
version 2
Acceptance uncertainty worse than Background
uncertainty
Limit of C.L. as
Upper limit
Need to check Coverage
Uncertainty
70
Method Mixed Frequentist - Bayesian
Bayesian for nuisance parameters and Frequentist
to extract range Philosophical/aesthetic
problems? Highland and Cousins NIM A320 (1992)
331 (Motivation was paradoxical behavior of
Poisson limit when LA not known exactly)
71
Systematics Tests

• Do test (e.g. does result depend on day of week?)
• Barlow Are you (a) estimating effect, or (b)
  just checking?
• If (a), correct and add error
• If (b), ignore if OK, worry if not OK
• BUT
• Quantify OK
• What if still not OK after worrying?
• My solution
• Contribution to systematics variance is
• even if negative!

72
Barlow Asymmetric Errors
e.g. Either statistical or systematic How to
combine errors ( Combine upper errors in
quadrature is clearly wrong) How to calculate How
to combine results
73
Significance

• Significance ?
• Potential Problems
• Uncertainty in B
• Non-Gaussian behavior of Poisson
• Number of bins in histogram, no. of other
  histograms FDR
• Choice of cuts (Blind analyses
• Choice of bins Roodman and Knuteson)
• For future experiments
• Optimising could give S 0.1, B
  10-6

74
Multivariate Analysis
Friedman Machine learning Prosper Respondent Nar
sky Bagging Roe Boosting
(Miniboone) Gray Bayes
optimal classification Bhat
Bayesian networks Sarda
Signal enhancement
75
Blind Analyses

• Potential problem Experimenters bias
• Original suggestion?
• Luis Alvarez concerning Fairbanks
  discovery of quarks
• Aaron Roodmans talk
• Methods of blinding
• Keep signal region box closed
• Add random numbers to data
• Keep Monte Carlo parameters blind
• Use part of data to define procedure
• Dont modify result after unblinding, unless.
• Select between different analyses in pre-defined
  way
• See also Bruce Knuteson
• QUAERO, SLEUTH, Optimal binning

76
Where are we?

• Things that we learn from ourselves
• Having to present our statistical analyses
• Learn from each other
• Likelihood not pdf for parameter
• Dont integrate L
• Conf int not Prob(true value in interval data)
• Bayes theorem needs prior
• Flat prior in m or in are different
• Max prob density is metric dependent
• Prob (DataTheory) not same as Prob(TheoryData)
• Difference of Frequentist and Bayes (and other)
  intervals wrt Coverage
• Max Like not usually suitable for Goodness of Fit

77
Where are we?

• Learn from Statisticians
• Update of Current Statistical Techniques
• Bayes Sensitivity to prior
• Multivariate analysis
• Neural nets
• Kernel methods
• Support vector machines
• Boosting decision trees
• Hypothesis Testing False discovery rate
• Goodness of Fit Friedman at PHYSTAT2003 Panel
  Discussion
• Nuisance Parameters Several suggestions

78
Conclusions
Very useful physicists/statisticians interaction
e.g. Confidence intervals with nuisance
params Multivariate techniques
etc. Software repositories For
programme, transparencies, papers, etc. see
http//www.physics.ox.ac.uk/phystat05 Proceedings
to be published by IC Press (Spring 06) Future
79
FUTURE No Conference
in summer 2007 Workshops/schools e.g.
Manchester Nov 2005 SAMSI Duke, April
2006 Banff, July 2006 Spanish
Summer School, July 2006 Israel, Sept
2006? CERN LHC issues, early
2008? Repository of statistical software (Jim
Linnemann) Suggestions to
l.lyons_at_physics.ox.ac.uk
80
Thank you!

• Many people
• Local organising Committee
• Andy Carslaw
• John Cobb
• Sue Geddes!!!
• Emily Downs Talks on computer
• Muge Unel Editting the Proceedings
• Those of you who attended the Conference,
  especially..