Sleuth deficits search

1
Sleuth deficits searchVista significance
calculation
CDF Exotics meeting 2007/03/01 (5th special talk,
2.5 months since Full Status) Georgios
Choudalakis MIT
The MIT HighPt group
BruceKnuteson
ConorHenderson
Ray Culbertson
Georgios Choudalakis
2
Outline

• Vista significiance calculation in population
  comparison
• I expect b db and observe d. What is the
  significance of this effect?
• Sleuth deficits search
• Definition of the statistic
• Result
• Discussion

Georgios Choudalakis MIT 2
3
I expect b db and observe d. What is the
significance of this effect?

• It is NOT d-b / db
• It is NOT d-b / sqrt(db2b)
• It is a sum of p(n b,db)

p(n b10)
p(n b10, db2)
n
Georgios Choudalakis MIT 3
4
I expect b db and observe d. What is the
significance of this effect?

• It is NOT d-b / db
• It is NOT d-b / sqrt(db2b)
• It is a sum of p(n b,db)

p(n b10)
p(n b10, db2)
n
d
Georgios Choudalakis MIT 4
5
I expect b db and observe d. What is the
significance of this effect?

• It is NOT d-b / db
• It is NOT d-b / sqrt(db2b)
• It is a sum of p(n b,db)

p(n b10)
p(n b10, db2)
n
d
Georgios Choudalakis MIT 5
6
I expect b db and observe d. What is the
significance of this effect?

• d b - db probability
  s
• 2 5 - 0 0.124645
  -1.15208
• 3 5 - 0 0.265029
  -0.627918
• 5 5 - 0 0.384041
  0.294886
• 8 5 - 0 0.133376
  1.11057
• 12 5 - 0 0.00545294
  2.5457
• 18 5 - 0 5.41574e-06
  4.39987
• 2 5 - 0.5 0.130993
  -1.12171
• 3 5 - 0.5 0.271908
  -0.607052
• 5 5 - 0.5 0.384143
  0.294619
• 8 5 - 0.5 0.138468
  1.08723
• 12 5 - 0.5 0.00670938
  2.47246
• 18 5 - 0.5 1.08632e-05
  4.24637

Georgios Choudalakis MIT 6
7
I expect b db and observe d. What is the
significance of this effect?

• d b - db probability
  s
• 90 80 - 0 0.144488
  1.06037
• 135 80 - 0 1.32666e-08
  5.5629
• 202 80 - 0 7.64337e-24
  gt9.99969
• 90 80 - 16 0.296512
  0.534458
• 135 80 - 16 0.00240307
  2.81975
• 202 80 - 16 1.73732e-09
  5.90744
• 90 80 - 24 0.349893
  0.385609
• 135 80 - 24 0.0192294
  2.06993
• 202 80 - 24 4.78368e-06
  4.42673

Georgios Choudalakis MIT 7
8
Trials factor

• In Vista, the probability is diluted by a trials
  factor ( N number of final states ) and then
  converted into s.
• pdiluted1 (1-p)N
• Example 1mu1mu- final state
• d10648 b10846.4 db96
• p 0.08 , pdiluted1 , no need to worry about s

Georgios Choudalakis MIT 8
9
Sleuth deficit search
10
Definition of the statistic

• When looking for excesses
• Pvalue of each tail is the Poisson probability
  that given we expect b, we would observe d or
  more.
• When looking for deficits
• Pvalue of each tail is the Poisson probability
  that given we expect b, we would observe d or
  less.
• The definitions of scriptP and tildeScriptP
  remain the same, but the meaning of
  interestingness is twisted.

Georgios Choudalakis MIT 10
11
Interesting deficit 1
Georgios Choudalakis MIT 11
12
Seek the reason in Vista
Georgios Choudalakis MIT 12
13
All pT's look reasonable
Georgios Choudalakis MIT 13
14
All pT's look reasonable
Georgios Choudalakis MIT 14
15
All pT's look reasonable
Georgios Choudalakis MIT 15
16
But (mostly) this causes the deficit
Pythia jj 120 lt pT lt 150 10.5, Pythia jj 200 lt
pT lt 300 10.2, Pythia jj 60 lt pT lt 90 9.3,
Pythia bj 150 lt pT lt 200 8.3, Pythia bj 90 lt pT
lt 120 7.3, Pythia bj 120 lt pT lt 150 6.9,
Pythia gamma j 22 lt pT lt 45 6.4, Pythia bj 60 lt
pT lt 90 4.9, Pythia jj 40 lt pT lt 60 4.1,
MadEvent gamma gamma jj 3.1, Pythia bj 40 lt pT
lt 60 3.1, Pythia bj 200 lt pT lt 300 3, Herwig
ttbar 1.8, Pythia jj 18 lt pT lt 40 1.6, Pythia
jj 300 lt pT lt 400 0.8, Overlaid events 0.5
Georgios Choudalakis MIT 16
17
How does that affect the search for excesses?
Georgios Choudalakis MIT 17
18
What is the consequence when searching for
excesses?

• The tail starting at 250 GeV is less significant
  (an excess) than the one tagged at 604 GeV,
  possibly because of overestimating the background
  in the tail.
• This overestimation shows in the search for
  deficits.
• BUT it also shows in Vista, as a shape
  discrepancy!
• So, we knew it was there, butSleuth is not
  designed to findsuch things, because that's not
  what new physics is expected to look like.

Georgios Choudalakis MIT 18
19
Significant deficit 2
Spikes
Georgios Choudalakis MIT 19
20
Same final state, looking for excess
The is no actual excess in any tail.
Georgios Choudalakis MIT 20
21
Another deficit case, not spikessearching for
deficit
Georgios Choudalakis MIT 21
22
Another deficit case, not spikessearching for
excess
Georgios Choudalakis MIT 22
23
The underlying problem shows in Vista shapes
Georgios Choudalakis MIT 23
24
summary of deficit kinds

• Overestimation of background locally.
• due to spikes
• or mis-modeling of some object's pT in a final
  state.
• Overestimation of background globally.
• due to spikes
• or mis-modeling of some object's pT in a final
  state.

25
The smiley distribution of scriptP (for
excesses of course)
There are both underestimations (left) and
overestimations (right) of the background.
Georgios Choudalakis MIT 25
26
What is the scriptP distribution for deficits?
1) Many final states have significant
deficits. 2) Many final states have insignificant
excesses (right side of smile). 3) There are no
discovery-level excesses. 4) There are
discovery-level deficits. We don't debug
deficits as much as excesses. That probably
causes the difference. The reason is that we
wouldn't be able to discover anything with a
deficit. On the other hand, having lurking
deficits reduces sensitivity (to excesses of
course). On the third hand, we have estimated
our sensitivity in many ways, even thus.
Georgios Choudalakis MIT 26
27
Georgios Choudalakis MIT 27
28
2007-02-18 comparison
Each point is a Sleuth final state
The less significant the excess, the more
significant the deficit (overestimation of
background). For more significant excesses,
deficits are not significant, which might be the
reason the excesses were not rendered too
insignificant.
Georgios Choudalakis MIT 28
29
Grand Summary Conclusion

• Sleuth deficits Sleuth excesses Vista shapes
• We know there are discrepant shapes (please, look
  at them)
• There used to be more, many of which looked like
  potential discoveries, but they didn't survive
  debugging.
• No Vista population discrepancies No
  significant Sleuth excesses.
• So the discrepant shapes left are of the
  non-discovery-like kind
• They reduce our sensitivity. The sensitivity has
  been assessed.
• We can't prove there is no new physics, because
  it's impossible to debug ad infinitum
• There is no evidence that ideal debugging would
  reveal a genuine discovery
• Our prejudice is that a discovery would appear as
  a Sleuth excess. We see no such excess.
• What and how Sleuth seeks is clearly stated. The
  result is also clear.

30
backup section
31
What about b' lt 0 ?

• Poisson is not defined for b'0 or b'lt0.
• b' takes values in the (0,infinity) domain, and
  the gaussian part is renormalized by 1/f, where f
  is the fraction of the gaussian that is positive,
  so that it still has area1.

Georgios Choudalakis MIT 31
32
related final state not discrepant.
33
Significant deficit 3
A big spike
Georgios Choudalakis MIT 33
34
The same, looking for excess
This time the overestimation of background
(spike) was not hiding any excess, because there
is no excess in this final state. The whole
background is overestimated by spikes.
Georgios Choudalakis MIT 34
35
We don't overly overestimate or underestimate the
backgrounds