A Monte Carlo Model of Tevatron Collider Operations

1
A Monte Carlo Model of Tevatron Collider
Operations

• Elliott McCrory, BD/(Proton Integration)
• November 13, 2003
• A phenomenological model of Tevatron
  Collider operations has been created.  Key
  elements of the operation of the facility have
  been randomized in this model to reflect actual
  Run II performance. In particular, failures and
  downtimes occur randomly, in agreement with the
  rates observed in reality. Similarly,
  performances are randomized, also in agreement
  with the range of possibilities in reality.  Some
  of the performance elements that have been
  randomized include PBar transmission and
  emittance growth from the Accumulator to Low
  Beta, Shot Setup time, the Luminosity Lifetime,
  etc.  A primary motivation for this model is to
  guide the Run Coordinator on how to manage the
  operation of the Collider.  In particular, this
  model answers the question of how a particular
  criterion for ending stores affects the
  integrated luminosity.

2
Thanks To

• 1994
• Vinod Bhardawaj, Allan Hahn, Gerry Jackson and
  Peter Lucas
• 2003
• Dave McGinnis
• Jean Slaughter
• Paul Lebrun
• Larry Allen and the Linac Group

3
Outline

• Model overview
• Example
• Program Structure/Classes
• Random numbers
• Collider Operational Performance
• Data from Controls from Operations Weekly
  Summaries
• Matching Model to Reality
• Developing Intuition/Model Predictions
• Interesting Observations
• When should we start and end stores?
• Web mccrory.fnal.gov/montecarlo
• Conclusions

66 to 72 slides
4
1. Model Overview

• Phenomenological, non-analytic model of Tevatron
  Collider Operations
• Complexity ? Randomness
• Downtime
• For the Tevatron, stacking and the PBar Source
• Variations in all realistic parameters
• E.g., transmissions during a shot, lifetimes,
  uncertainty in exactly how many PBars we extract,
  shot setup time, downtime, etc.
• Shot data is used Match Model to Reality
• Appropriate range of values for important
  parameters
• Correlations among the parameters
• Develop intuition/guidance for controlling stores
• Goal For the Model to reflect accurately present
  and future Tevatron Operations
• Many already have this intuition, but not all.

5
Assumptions

• Performance does not improve
• Random fluctuations around a specific set of
  parameters
• As performance changes, Ill modify the Model
• No shutdown periods
• Existing shot data is accurate
• Supplemented and supported by other sources
• Luminosity
• L (t0) K ? Np(0) ? NPBar(0) / ?p (0)
  ?PBar (0)
• L (t) L (0) e -t/t(t)
• More on this later

6
Example Tevatron Failure Rate
Time Between Tevatron Failures Real Data
Model data for Tevatron Failures
f(t) ? e -? t s lt t gt 1/?
? e -? t
R 1 - ??t ?t 42 hours
? 0.975 / hour
7
Failure Rate Interpretation

• ? is measured directly from real data
• lt t gt s 1/ ?
• Probability of having stores of
• 1 hour 0.975
• 2 hours (0.975)2 0.951
• 10 hours (0.975)10 0.776
• 20 hours 0.603
• 30 hours 0.459
• Failure Rate is Independent of Time
• This is a random process!!

8
2. Program Structure

• How does this work?
• Step size 0.1 hours
• Diminish the luminosity
• Stack
• Has anything failed?
• Stacking stops?
• Stacking slows down?
• Lose a store?
• Lose a stack?
• End-of-store criterion? Start shot setup.
• Shot Setup over? Generate luminosity
• Shot process Heavily randomized
• Based on Reality
• Stack or store lost? Stack to reasonable stack
  size shoot
• Reasonable 100 mA (But see below!)
• If a stack is lost, we could keep the store in
  for a long time!
• Repeat for N weeks, dumping lots of relevant data.

9
Example

• linux simulate_week -f paramGold.dat -n10 -c 2
  -T 180 -M 24 -z 9 -y 10000 -H
• Wk Integ_Lum StrHrs Pbars stkHrs suHrs
  Setps TevDn PbarDn DStore DStk
• 0 6726.813 70.10 1028.334 138.60 13.60
  7 7.10 12.20 5 0
• 1 10669.255 126.70 893.990 123.70 11.70
  5 1.20 12.50 1 0
• 2 8805.500 106.00 864.362 132.60 16.10
  5 2.60 9.60 0 0
• 3 6884.789 85.70 881.393 111.80 22.00
  6 7.70 15.60 5 2
• 4 6028.576 82.80 1055.254 134.70 14.00
  7 6.70 16.20 4 0
• 5 6522.033 73.40 935.862 134.60 15.00
  7 4.20 12.00 5 1
• 6 7165.688 90.60 816.992 110.80 12.90
  6 10.10 10.70 4 0
• 7 9236.213 116.80 868.210 111.20 22.80
  6 0.00 15.50 0 1
• 8 4265.170 54.40 707.972 92.90 22.00
  6 3.30 14.40 4 0
• 9 9860.740 115.50 1021.286 143.40 14.90
  6 6.80 8.70 2 0
• Downtime stacking 207.073 (Total stacking time
  1664.6)
• Tevatron 1427.77
• 61 total stores, 30 Stores lost
• 4 Stacks lost

10
Model data Weeks 7, 8 and 9
11
Pictorially Weeks 7, 8, and 9
10000 1/nb
100 E30
200 mA
Week 7
Week 8
Week 9
7 9236.213 116.80 868.210 111.20 22.80 6
0.00 15.50 0 1 8 4265.170
54.40 707.972 92.90 22.00 6 3.30
14.40 4 0 9 9860.740 115.50 1021.286
143.40 14.90 6 6.80 8.70 2 0
12
Program Internals

• Written in C
• No external package dependencies
• Some object oriented features
• Random Numbers End-Store mechanism
• Run on FRH Linux 9
• Celeron PC at 2.2 GHz
• 5000 weeks (100 years) of operations 40 seconds
• Over 100 parameters
• Parameter definition file
• Command-line arguments
• Run from the Unix command line
• Shell/perl scripts control parametric scans
• A fledgling web interface is under development

13
2a. Random Numbers

• Linux drand48( )

RandomLikely(-2, 12, 8)
Product of these two distributions
RandomLikely(0, 5, 2)
14
3. Collider Performance

• Measurements of Operation Performance
• From our control system
• Shot Data Acquisition (SDA)
• D44 data logger
• D18 downtime logger
• From Weekly Operations Data Sheets
• Weekly hour usage
• Weekly integrated luminosity
• How well do we understand operations?
• Put it together Matching Model to Reality
• Match to good weeks of running in 2003
• Note Only 35 weeks in 2003 so far

15
Performance Measures from SDA

• Mostly from the Super Table
• Stacking rate
• Shot Setup duration
• Emittance of the PBars from the core
• PBars from the Accumulator to Low Beta
• Efficiency
• Emittance Growth
• Protons at Low Beta
• Stack Size vs. Initial Luminosity
• Initial Luminosity Lifetime
• As calculated in the Super Table
• From D44/Datalogger
• Time in store
• Time between stores

16
Stacking Rate 18-25 Aug 03
13.4(1-size/280.0)
Matched to Reality, see below
17
Stacking Rate Fluctuations
RandomLikely(0.7, 1.02, 0.95)
Number records, per bin, August 18-25, 2003
Ratio of Actual Rate to Expected Rate
18
Randomizations Shot Setup
RandomLikely(1.2, 3.5, 2.3)
Data from SDA Super Table
Setup Duration, Hours
19
Correlation Stack vs. Core ?V
Upper Limit 1E-4 x2 8E-3 x
Best Fit 5E-5 x2 8E-3 x
Lower Limit 1E-5 x2 8E-3 x
Vertical Emittance in the Core
Stack Size, mA
20
Shot PBar Effic vs. Stack Size
Transmission from Accumulator to Collisions
Subtle Stack Size dependence added
Stack size before first transfer
21
Emit Blow-up vs. Stack Size
PBar Emittance change, additive (p mm mr)
Stack size before first transfer
22
Protons at Low Beta Intensity
For all of 2003
In August, 2003
RandomLikely(140, 260, 170)
RandomLikely(230, 270, 250)
Number of shots, per bin, since Aug 2002
Protons per bunch at low beta, E9
23
Correlations NP (Low ß) vs. ?ltxygt
Longitundal Emittance at low Beta
24
Correlations NP (Low ß) vs. ?z
Extra
Transverse Emittance at low Beta
25
Initial Luminosity vs. Stack Size
Initial Luminosity (E30)
Stack size before first transfer
26
Luminosity Lifetime

• Luminosity lifetime imposed on the Real Data
• L (t) L (0) e t /t
• From t0 to 2 hours
• Lebrun Algorithm for fit
• Terrible fit to Real Data
• Model assumption
• L (t) L (0) e t /t(t)
• t(t) t(0) C1 t C2
• t(0) depends on L (0) and is adjusted to fit
  Real Data
• C1 1.8 0.2
• C2 0.595 0.005
• Excellent fit to Real Data

C11.7, c20.595
27
Initial Luminosity Lifetime
2003 Stores
Luminosity Lifetime (first 2 hours), hours
Simulated Stores
Initial Luminosity, E30 cm-2 sec-1
28
D44 Store Duration
Reality in 2003
RandomLikely(16.5, 25.5, 21.5)
Model
Failures
29
D44 Time Between Stores
30
Weekly BD/Ops Dept. Data Sheets

• Presented at the All-Experimenters Meeting.
• Data obtained Ops Staff from the E-Log
• Ops crews How many hours during their shift were
  spent in each category?
• Down Time Logger?
• Minimal correlation

31
Operations Sheets from 2003
32
Understanding Column Headings
Extra

• Store
• Time in which there is luminosity at the
  experiments
• Shot Setup
• Some problems with these numbers in SDA
• Resolved, somewhat, by OSDA/D44
• Tevatron Studies
• How to classify parsing the squeeze?
• Shutdown
• Tevatron is in access, Including overhead
• E.g, racking out and racking in.

• Failure
• Tevatron down
• For example, a quench and its recovery
• Turn Around
• Time between failure and useful Tevatron time
• Useful Shot Setup or Studies
• Operations and/or Tevatron staff wants to do more
  investigation before proceeding.
• Standby

33
Turn Around vs. Failure?

• The Real World
• Somewhat subjective.
• Operations crews differ on which hours go where
• Is not consistent with Down Time Logger
• The Model
• Much simpler!!
• Failure ? Recovery ? (Optional Studies/Access) ?
  Shot Setup
• Natural end ? (Studies/Access) ? Turn Around ?
  Shot Setup
• Different time spectrum for Recovery and Turn
  Around.
• For comparison
• Sum all time without Tevatron activity for both

34
Removing Non-Productive Weeks

• Only consider weeks when we are actually running
• Remove weeks in which
• Shutdown time gt 35 hrs
• Removes 6/35 weeks
• Study time gt 40 hrs
• 6/35 weeks
• Store Time lt 60 hrs
• 13/35 weeks
• Total eliminated
• 15 out of 35 weeks
• Only 20 weeks remain!
• Apply these cuts to the Model.

35
Averages for FY03 Ops Sheets

• From the Operations Summary Sheets
• With Cuts described previously

Note Average of 6 stores per week in 2003
36
Average Hour Usage per Week
37
3b. Matching Model to Ops Sheets

• Simultaneously match
• Average hours per week.
• Average weekly integrated luminosity
• PBars stacked
• Etc.
• Histograms of
• Store hours per week
• Stacking hours per week
• (More plots examined but omitted here)

38
Store Hours per Week
Extra
39
Stacking Hours per Week
Extra
40
Best Match of Model to Data

• The parameters of the model have been adjusted to
  get this match.

41
Params that Match to Reality
42
4. Developing Intuition/Predictions

• Interesting observations (Optional)
• Error bars from Model
• What is important?
• When should we end a store and why?
• End-of-store criteria Which is best?
• How best to recover from failures
• Dealing with interruptions
• Tevatron studies, access

43
Error Bars
0.5 error on centroids
21 RMS spread over 5000 weeks of running
Uncertainty on mean
44
Whats Important? Integ Luminosity!

• First and foremost
• Hours per week in collisions
• Secondarily
• Stack size from which we shoot
• Maximize the effectiveness of the antiprotons
• Time spent at low luminosity vs. high
• Minimize the effects of
• Failures
• Interruptions in the program Studies, accesses

45
4b. When to Start End Stores?

• Describe various End-Store criteria
• Defining a Best Criterion
• What is the Best Criterion?
• Which criteria work well and which dont?
• Extensive search over some of the available
  parameter space
• Recovery from failure
• When should we start a store after a failure?
• Dealing with Tevatron studies/access

46
Some End-Store Criteria

• Stack Size
• Minimum Luminosity
• Store Duration
• Integrated Luminosity
• Run Coordinator 2003
• Luminosity Potential
• Make assumptions on L ( Stack Size )
• Ratio, difference
• Simplify (Stack Size) / Luminosity

47
Defining a Best Criterion

• Integrates lots of luminosity
• Insensitive to many/most performance changes
• Some performance changes may be unnoticed
• Random fluctuations or improvements?!
• Simple
• Have rejected complicated schema
• Easy to define clear criteria for
• Running flat out
• Running that has intentional interruptions

48
Target Stack Size
Average Weekly Luminosity, E30
Note Shift!
No Studies or Post-Store Accesses
Green TevUp 0.99
Green/Magenta TevUp 0.99
Red TevUp 0.975
Red/Blue TevUp 0.975
Shoot when stack reaches this value, mA
49
Search Parameter Space
Caution Lots of data hidden here!

• Odd numbered sets TevUp 0.975/hour
• Even numbered sets TevUp 0.99/hour
• Each Set is run without (NSA) and with
• Tevatron Studies
• Experiment access
• Each Set is run for each End-Store criterion
• Search for optimum for each criterion, each set
• Goal Which criterion works best

50
Minimum Luminosity
Optimum shifts higher with better performance
Sets 5, 6 Enhanced Stacking, Improved Lum Life
Sets B, C More Protons, Enhanced stacking.
Sets D, E More P, PBar Reduced recovery
Sets 1, 2 2003 Running
Sets 7, 8 Reduced Recovery time
Sets 3, 4 Enhanced Stacking
Sets 9, A 25 more protons
51
Store Duration, Hours
Extra
Broad Optima NSA Shorter stores
52
Luminosity Potential

• Use chart, here
• Turn existing stack size into a likely initial
  luminosity
• Two choices for how to end store
• When difference gt L.
• When the ratio gt V.
• Problem?
• Assumption on the potential curve

Luminosity Potential
Initial Luminosity, E30
50 recent stores
Stack Size, mA
53
Luminosity Potential Ratio
Best End store when ratio 5
Average Weekly Integrated Luminosity, nb-1
(Likely Initial Luminosity) / (Actual Luminosity
Now)
54
Luminosity Potential Ratio
Average Weekly Integrated Luminosity, nb-1
Optimum stays between 3 and 4
(Likely Initial Luminosity) / (Actual Luminosity
Now)
55
Simplified Luminosity Potential
(400, 70)
(200 mA, 50 E30)
Initial Luminosity, E30
Stack Size, mA
56
Simplified Ratio
Optimum also stays between 3 and 4
57
Real Simple Ratio Method

• Ratio (Stack Size) / (Luminosity Now)
• For example 180 mA, 12 E30 1/cm2 1/sec,
• Ratio 180/12 15.
• No assumptions on Luminosity Potential
• But is very closely related to this method
• Model determines target ratio
• And Model does not need to enter non-linear
  region
• Assumptions on luminosity potential are built
  in
• Not so simple Requires knowledge of Np

58
A week running with Simple Ratio
Ratio 16
Stack Size, 10 mA Luminosity, E30 (cm-2 sec-1)
200 mA
10E30
Time, hours
59
Simple Ratio
Optimum shifts left (16 ?12) with more protons
60
Luminosity Potential Wins, but

• Default assumption is complex.
• Also, the target curve was derived from the Model
• What assumption for Stack?Luminosity is right?
• Use recent performance to determine luminosity
  potential?
• Handicapping?
• Simplified
• Luminosity at 200 mA could be changed as
  performance improves
• Real Simple works well
• If Np is factored in
• More work needed!

61
Recovery from a Failure

• What is the best way to maximize luminosity after
  a fault?
• Store lost
• Stack lost
• Shoot ASAP?
• Wait for a reasonable stack?

62
Reasonable Stack Size
Average Integrated Weekly Luminosity, nb-1
Choices from 60 to 120 mA 1.2 100 mA is a
good choice
Uncertainty on mean 25 nb-1, 0.3
Simple Ratio, mA/(E30 cm-2 sec-1)
63
Dealing with Tevatron Studies
Extra

• Change of strategy if you know that
  studies/access follows a store
• Will create an end-store criterion to simulate
  this
• More work needed

64
Time Spent in Store Studies
Extra
In Tevatron Stores
Hours
Using parameters for normal running
In Tevatron Studies
Target Simplified Ratio
Minimum Luminosity
65
Study Hours vs. Store Hours
Extra
Study Hours per Week
Store Hours per Week
Using MinLum
66
Where do we integrate Luminosity?
Extra
Target Stack Size
Store Duration
Run Coordinator 2003
Simplified Ratio
Minimum Luminosity
67
5. Web Pages

• http//mccrory.fnal.gov/montecarlo
• http//mccrory.fnal.gov/testForm.html

68
6. Conclusions

• Review specific findings
• Whats next?
• General Conclusions

69
Review of Specific Findings

• Tevatron failures are independent of time
• Up time 0.975/hour
• Stacking rate for 2003 has been
• R 13.4 / ( 1 S/280 ) mA/hour
• 20 weeks of good running in 2003
• Luminosity
• Averaged 6810 nb-1 4 ( 272 nb-1)
• Could have gotten about 7100 0.5
• Luminosity Lifetime
• L (t) L (0) e t /t(t)
• t(t) t(0) C1 t C2
• C1 1.8 0.2, C2 0.595 0.005
• After failure Build to 60 lt Stack Size lt 120 mA
• Allows time for Tevatron studies
• May need different End-Store criteria for flat
  out versus normal running.

70
Whats Next?

• Real Run Coordinator 2003 End-Store criterion
• Via Ron Moore, Doug Moehs, etc.
• 10 hours into store
• Estimate time to 10E30 at CDF
• Set end of store to not span shift change
• nb-1/hr measure
• Recycler/SY tax
• Incorporate Recycler
• ACNET variables for the new End-Store criteria
• E.g., Luminosity potential Difference Ratio

71
General Conclusions

• Model matches Reality well
• Hours per week
• Luminosity, luminosity lifetime, etc.
• Interruptions to integrating luminosity are
  important.
• Failures
• Studies, experiment access, recovery, etc
• Some End-Store criteria work well
• Ratio or Difference with luminosity potential
• Store Duration, minimum luminosity, target stack
  size
• Simple ratio of stack size to instantaneous
  luminosity
• Needs work
• Some criteria do not work well
• Work Continues