Project Organization and Budget Issues U'S' CMS Software and Computing Project Lothar A T Bauerdick,

1
Project Organization and Budget IssuesU.S. CMS
Software and Computing ProjectLothar A T
Bauerdick, FermilabProject Manager

• DOE/NSF Review Nov 27-30, 2001

2
Project Organization

JOG LHC Program Office LHC Project Office
DOE/NSF Review Chair Glen Crawford
Project Management Group (PMG) Chair Ken
Stanfield PMG for SC Chair Mike Shaevitz
Fermilab Project Oversight
Software and Computing Project L1 Project
Manager LATBauerdick/Fermilab L.Taylor/NEU deputy
SCOP Chair Ed Blucher
U.S. CMS Advisory Software and Computing
Board (USASCB)chair I.Gaines/Fermilab
Core Applications Software Project L2
Manager I.Fisk/UCSD
User FacilitiesProject L2 Manager V.ODell/Fermi
lab
Physics Reconstruction and Selection Detector
Software Groups
3
Project Organization

• All the Managers are on board
• latest addition Project Engineer Bakul Banerjee
• All the committees are working
• ASCB for contact with CMS collaboration entities
• PMG for project oversight
• We are reporting regularly on progress and status
• successful metric reporting effort in FTE vs the
  WBS
• more work on milestones to even more closely
  track the project
• We are going for the first MOU/SOW to
  sub-contract CAS effort to Universities
• Drafts for SOW exist

4
Managerial Issues w/ NSF Proposal

• NSF Funds are directed at Project(s)
• U.S. CMS SC Project and U.S. CMS Construction
  Project
• Project Management allocates Funds
• Funds being administered by a NSF Lead
  Institution (PI)
• U.S. CMS recommends that Northeastern University
  should do that
• Northeastern has considerable NSF experience
• Do already have a working administration for NSF
  funds for U.S. CMS
• handles all NSF funds for the Construction
  Project
• handles NSF software (supplementary) funds (in
  SC project)
• a lot of accumulated group experience
• 1 FTE of administrative support (on-project)
• NEU already has key managerial and technical
  roles in SC
• Will use Northeastern administrative service for
  new NSF proposal
• MO and SC Projects have very similar
  administrative needs to Construction Project
• Procedure can be fine-tuned as required

5
Existing Northeastern Admin.
FNAL US-CMS project office PMG oversight
US-CMS Institutes
DOE
SOWs
North- eastern
NSF

• All funds allocated by US-CMS project
• Only then are they released by NEU
• Tracking data to US-CMS project office
• Valuable cross-check for NSF (NEU and FNAL)
• N.B. a lot of bureaucratic details in all these
  connections

6
Administration of NSF Funds

• NSF Funds will be received by lead institution
  of PI
• NSF Funds are allocated to U.S. CMS institutions
  by project management
• following the procedures as outlined in the PMP
• including SOW to be signed by the U.S. CMS
  groups
• NSF Funds are being released by subcontracting,
  via invoicing
• invoices need signature by project management
• accounting is done by WBS,
• The administrative organization to support this
  is part of the Project Office(s)
• Separate dedicated account for project
  fundsespecially with part of the funds being
  transferred to CERN
• Administrative Assistant at lead inst. to support
  accounting, invoicing, reporting
• There is also a role for the PI in the
  administration of fundse.g. negotiations with
  Universities, interaction with NSF,
  support for reports and reviews

7
Scope Preparation for the LHC Research Program

• Defined Scope and Worked Out Budget ? WBS
• Build the software systems for CMS
• Fulfill commitments towards CMS support U.S.
  specific needs
• Build a strong Tier-1 center at Fermilab RD,
  Equipment, Staff
• As part of our obligation towards building the
  CMS computing infrastructure
• To enable full U.S. access to LHC physics
  dataand to provide sufficient resources to do
  physics analysis in the U.S.
• Empower Universities to be full participants in
  the LHC Research Program
• Tier-2 centers equipment and operations
• Support Distributed Infrastructure Grid Services
  and related RD
• Education and Outreach as part of the NSF
  proposal
• Project Office, Management Reserve
• Defined Schedule ? Major Milestones
• DAQ TDR and HLT studies/PRS groups now to 2002
• 5 Data Challenge and CCS TDR end of 2003,
  Physics TDR end of 2004
• 20 Data Challenge and start of facility
  deployment end of 2004
• Start of Physics in 2006

8
Installed Capacity Tier-1 Facility
5 DC RD Tier-1 System
20 DC Prototype Tier-1 System
Fully Functional Facilities At 40 Capacity
for Physics
9
Tier-1/2 RD Systems SC TDR

• Milestones CCS TDR U.S. CMS UF RD Systems
  (FY2002/3)

10
Prototype Tier-1/2 20 Data Challenge

• Milestone 20 DC, U.S. CMS UF pTier-1/2 System
  (5 system)

11
40 Capacity Tier1/2 LHC Pilot/Physics Run

• U.S. CMS UF Milestones fully functional Tier-1/2
  Systems

12
NSF Proposal

• CMS and ATLAS have submitted a proposal to the
  NSF
• Empowering Universities Preparation of LHC
  Research Program
• Means to get NSF funding of SC (and MO) to CMS
• Scope SC and MO/upgrade RD, 8M in 2006
• Unclear to which program we will be submitting
• Time table unclear (start of funding supposed to
  be in 2002)
• DOE/NSF review in November part of the reviewing
  process (of the SC part)
• Split CMS/ATLAS assumed to be 50/50, also for MO
• Unclear funding profile, but use existing advise,
  scaled to 8M in 2006
• Putting SC and MO into one proposal gives has
  two projects addressing
• Includes Education and Outreach, Broad impact
• This is a big step forward for CMS, thanks to
  the people at the NSF

13
NSF Funding through Grid Projects

• Budget for the GriPhyN and iVDGL projects funded
  by the NSF. Shown is the total project budget and
  the part that is going directly to Universities
  involved in CMS.
• GriPhyN (NSF-ITR) and iVDGL (NSF-MPS)

14
Tier-2 Equipment Costs

• Deployment cost of U.S. CMS Tier-2 centers over
  time. The RD phase occurs until 2003, and the
  pilot starts in 2004. The underlined figures show
  the beginning of real Tier-2 operations, which
  occur in different years as the Tier-2 centers
  are phased in.

15
Tier-2 Labor Effort

• Cost of support personnel (in FY2002k) for the
  Tier-2 centers over time. Part of the personnel
  costs for the RD and prototyping phase (two
  prototype centers) is funded out of iVDGL. The
  cost for central staff is also shown.

16
CAS Labor Effort

• Estimated personnel requirements for the U.S. CMS
  Core Application Software efforts, including
  U.S.-specific support. Shown is the total U.S.
  CMS SC effort, the part requested from the NSF,
  and the corresponding cost for the NSF part,
  including personnel cost and associated support,
  such as travel, desktops, training and software
  licenses

17
NSF Proposal Budget

• All numbers are given in thousand FY2002 dollars,
  except the grand total, which contains an
  escalation of 3 per year. Total costs are 22.8M

18
NSF Proposal Funding Profile

• Numbers for each category are in FY2002k, an
  escalation of 3 per year is shown separately.
  Total costs are 22.8M

19
UF Equipment Costs

• Detailed Information on Tier-1 Facility Costing
• See Document in Your Handouts!
• All numbers in FY2002k

20
Installed Capacity Tier-1 Facility
5 DC RD Tier-1 System
20 DC Prototype Tier-1 System
Fully Functional Facilities At 40 Capacity
for Physics
21
UF Manpower Needs

• Detailed Information see UF WBS document in your
  Handouts
• These numbers come from the resource-loaded WBS

22
UF Manpower Profile

23
Total Project Costs

• In AYM

24
Budget and DOE/NSF Guidance Profile

25
Budget and DOE/NSF Proposal Profile

26
DOE and NSF Profiles

27
Total Integrated Project Costs

28
Cost Variance vs Guidance

29
Variance vs DOE Guidance, NSF Proposal

30
Funding Status FY2001

• Received 1500k from DOE in April
• 500k labeled as Equipment funds
• Tier-1 upgrade, see V.ODells talk
• 500k Loan from Construction Project
• Charge PPD budget code by reporting CAS
  engineering efforts
• Received additional 285k from DOE in August
• Allocated to Tier-1 equipment procurements
• Received funding from NSF for 3rd engineer in
  October 2001

31
Total FTE Spent on UF in FY2001

• Total FTE funded in FY2001 was 4.05!
• see Quarterly Reports

32
Total FTE on CAS in FY2001

• Total FTE funded was 7.75
• see Quarterly Reports

33
Allocation of DOE Funds in FY2002

• Total budget from the DOE in FY2002 2.5M
• partly out of a line item, partly out of PKs
  funds, and the rest out of OFallons reserve
• Continue the (DOE) CAS effort at the same level
  as this year.
• keep the staff of 6 FTE
• 2 FTE at Caltech, 1 FTE at UCDavis, 1 FTE at
  Princeton, 2 FTE at Fermilab
• In addition we are increasing the NSF funded CAS
  effort
• 2 FTE at Northeastern 1 more to be hired at NEU
  (job advertised!)
• New proposal has one additional FTE in 2002
• Ramp up the User Facility FTEs by 1 FTE
• One more hire will bring us to 5 CPs at Fermilab.
  There will be continued additional effort of
  about 2 FTE from guest scientists and engineers,
  bringing the total effort to about 7 FTE (5th CP
  Bill Tanenbaum started last month).
• This leaves us short of 4.5 FTE from the total
  11.5 FTE needed according to Nov WBS. This is
  accommodated by stretching the necessary RD.

34
Allocation of DOE Funds in FY2002 contd

• Ramp up the User Facility Equipment at Fermilab
  with a FY2002 allocation of 445k.
• The project has spent the FY2001 allocation of
  410k, after that in FY2002 we are investing
  in total 630 at the Tier-1
• There is a quite detailed hardware plan, which is
  documented in the WBS, see V.ODells talk in the
  UF breakout session
• Staff project office with 1 FTE
• Bakul Banerjee started last week as our Project
  Engineer
• Leave a management reserve of nominal 10.
• Short fallings are already identified,also
  reserve for Construction Project request for
  pay-back,in case DOE fails to identify the funds
  to pay back last years shortcomings

35
Allocation of DOE Funds

• All DOE funds are to be allocated to
  FermilabFTEs at Caltech, UC Davis, Princeton
  will get sub-contracted from Fermilab.
• In the process of writing the SOW to support that
  (Princeton U. first)
• The "baseline" costs for a DOE CAS CP this year
  is set to 150k
• we have input from the institutes what is
  actually needed
• small differences between the institutes
• fine adjustments acc. to the actual needs will be
  done
• Procedures as described in the PMP
• The subcontract needs to be supported by an MOU
  and a yearly SOW
• The CAS L2 will start negotiating those
• DOE and Project Oversight is completely on board
  with this

36
FY2002 Allocation Request to DOE

• FY2002 DOE Funding Allocation Request Break Out
• UF Labor costs 795k
• CAS labor costs 870k
• PO labor costs 150k
• Management Reserve 240k
• UF Equipment 445k
• Total 2500k to be allocated to Fermilab
• The first DOE FinPlan will have half of this sum,
  i.e. 1.25M.
• this should be used as operations money, mostly
  to take care of labor costs,
• the equipment part of 445 should be part of the
  next batch.

37
NSF Funds in FY2002

• No clear time table for NSF proposal exist yet
• Our expectation are
• 3 FTE for CAS at NEU
• the iVDGL funding to run Tier-2 prototypes
• The 500k to pay for the Tier-2 prototype
  equipment
• the rest as requested in the proposal
• One more CAS FTE
• Some funds for pT2 support personnel (300k)
• In total 2M from the NSF
• We also NEED to cover 1st pT2 center at
  Caltech/UCSD

38
Request for NSF Funds for pTier2 2000/2001

• Equipment Costs for the prototype Tier-2
  installations at Caltech and UCSD unfortunately
  still for the funding from the NSF
• Following guidance from NSF, CMS requested a
  total of 500K for the Californian prototype
  Tier2 facility in FY2001
• 300k for initial equipment
• 50k for maintenance and upgrades in 2001
• 150k for personnel
• Actually committed/spent are
• 189k for equipment at Caltech
• 175k for equipment at UCSD
• 36 for effort at Caltech
• 18k for effort at UCSD this totals to 418,
  and more funds for equipment and personnel needed
• BTW currently unbalance in NSF funding for ATLAS
  vs CMS

39
FY2003 Allocation

40
Budget Issue in FY2003

• Shortfall of 1M in FY2003
• Not fully covered by the reserve, as we have to
  pay back 500k loan
• Request to get this from DOE to help with the RD
  phase!
• Or advance from NSF (original guidance was
  2M!)
• Otherwise need to delay some of the hiring for UF
  some more

41
Conclusions

• Self-Consistent baseline Project Plan
• Scope CAS UF, Tier-1 Tier-2
• CAS provides 25 of software engineering effort
  towards CCS (total 13 FTE)
• Functional Prototypes ? Fully Functional Software
• Checkpoints CCS TDR 2003 and Physics TDR in 2004
• UF 40 capacity computing facilities for physics
  in 2006, 100 in 2007
• ready for physics in low-lumi run for 2006
• Get there by a series of milestones
• Prototypes, Data Challenges and Technical Design
  Reports
• RD Systems for Tier-1 and Tier-2 ? 5 Data
  Challenge ? CCS TDR
• Prototype Tier-1 and pilot Tier-2 ? 20 Data
  Challenge ? UF TDR
• Budget is very tight, contingency is in the Scope
  ? Build to Cost
• Need some flexibility early 1M in FY2003
• additional funding for 20 data challenge in
  FY2004 extremely valuable
• This plan should be the baseline to a successful
  LHC Research Phase