Computer Space Task Force Report

1
Computer Space Task Force Report

• IT-SPARCC Meeting
• January 24, 2007

2
Outline

• Motivation for this work
• Executive Summary
• Computer Space Task Force
• Bruns-Pak Analysis
• Data center activity at peer institutions
• Economic analysis
• Summary
• Appendices

3
Motivation For This Work
4
Why Consider Data Center Options For MIT?

• Number and size of HPC clusters growing
• Rapidly rising space change costs to house each
  of these clusters in separate non-scalable campus
  locations
• Institute does not have suitable unused available
  or additional space to locate more clusters on
  campus
• Appropriate computational research facilities are
  becoming a growing factor and may impact
• Faculty recruitment
• Faculty retention
• Graduate student recruitment
• Research volume

5
IT-SPARCC

• In June 2005 IT-SPARCC voted unanimously to
  endorse four recommendations to the Provost and
  EVP (Appendix)
• http//web.mit.edu/it-sparcc/recommendations_06_0
  5.html
• Council forms Computer Space Task Force (CSTF) to
  examine HPC, academic, and administrative data
  center options, key business issues, peer
  institutions (December 2006)

6
Computer Space Task Force Members

• Patrick Dreher (Chair CSTF)
• Proj Dir. IT Strategic Planning Initiatives
• Deputy Chair IT-SPARCC
• Gerbrand Ceder
• Professor, Dept of Materials Science and Eng
• John Donahue
• Office of Sponsored Programs
• John P Dunbar.
• Asst to the Provost for Space Planning
• Alan Edelman
• Prof, Dept of Mathematics
• Christopher Hill
• PRS, EAPS
• Jeremy Kepner
• Lincoln Lab Sr.Technical Staff

• Angie Milonas
• Sr. Manager Finance, IST
• Jim MorganInstitute Controller
• Theresa Regan
• Director, Ops Infrastruct Srvcs, IST
• Mark RiedeselAssoc Dir, Ops Infrastruct Srvcs,
  Sloan
• Gregory Rutledge
• Prof., Dept of Chemical Engineering
• MacKenzie Smith
• Associate Director for Technology, Library
• Boleslaw Wyslouch
• Prof., Department of Physics

7
Executive Summary CSTF Recommendations

• Multiple data center options
• Seedling and prototype clusters near faculty
  research groups (on-campus DLC buildings)
• On-campus central data center (W91 currently
  serving this role)
• Potential off-site placement for large research
  production clusters, servers, disks, and other
  hardware

8
Executive Summary CSTF Recommendations

• Off-site data center should be added as a
  placeholder within the list of projects in the
  MIT capital plan
• The CSTF should continue with
• Work with faculty and researchers to further
  understand HPC infrastructure needs and
  requirements
• Detailed site review and cost analysis for
  various potential off-site locations for a future
  data center
• Detailed requirements review and cost analysis
  for the long-term strategic options for on-campus
  data center (i.e. W91 or successor)

9
Computer Space Task Force (CSTF)

• Charge to CSTF (appendix)
• Focus on two main types of computing and review
  for current status and strategic planning
• HPC
• Academic and administrative
• Activity to date mainly focused on HPC

10
Current StatusHPC co-lo in W91

• W91's co-location facility can accommodate up to
  200kW of electrical power devoted to HPCs.
• LNS request is for 16-racks through FY 2008
  consuming approximately 120kW. W91 can support
  an additional 8-to-10 racks with the remaining
  non-UPS power dedicated to W91-130 depending upon
  the kW per rack.
• As "Bates" comes on-line, the LNS space within
  W91 becomes available again.

11
Computer Space Task Force (CSTF)

• Proposed HPC initial specs and capacity plan
  (appendix)
• Retained Bruns-Pak to
• review specs
• develop statement of data center requirements
• Overview estimate of costs and schedule
• Task Force works with Bruns-Pak to develop Data
  Center Statement of Requirements

12
Bruns-Pak Analysis
13
Bruns-Pak Analysis

• Defining the Modular Planning Increment
• Based on (1) 1100-ton Chiller Module
• Total cooling demand 880-tons
• Total UPS grade computer equipment power demand
  540kW
• Total non-UPS grade computer equipment power
  demand 1620kW
• (1) 750kVA/675kW UPS plus (4) equivalent power
  trains w/o UPS
• (1) 1750kW standby generator and (2) 3750kVA
  utility services
• Provides for
• 25 Low Density HPC clusters _at_ 5 racks/cluster and
  15kW/rack SAN and network racks 158 total
  racks 10,000 sq.ft. Data Center
• or
• 25 High Density HPC clusters _at_ 3 racks/cluster
  and 25kW/rack SAN and network racks 108 total
  racks 8,000 sq.ft. Data Center

14
Bruns-Pak Preliminary Cost and Schedule Estimate

• 8,000 to 10,000 sq.ft. Data Center
• 20,500,000 to 23,900,000
• Schedule from design through commissioning is 18
  months
• Estimate based on open-shop labor and 2007/2008
  dollars
• Estimate is not site specific and will need to be
  reevaluated after selection of a site. Estimate
  is based on a renovation of an existing facility,
  NOT new ground-up construction
• Estimate does not include costs of computing
  hardware, software or connectivity including
  external and internal networking

15
Data Center Activity At Peer Institutions
16
Data Center Activity at Peer Institutions

• Are other research universities acting on the
  issue of data centers?
• Common Solutions Group data center survey
• Current size of data center
• Age of current facilities
• Size of the planned expansion
• Timeline for implementing that expansion
• Recent data center expansions (appendix)
• Senior IT CIO/staff from 30 top research
  universities held workshop in Madison WI in
  Spring 06 focused on data centers with follow-up
  meeting at USC in January 07

17
Who Participated

• 100 Participation by CSG Member Schools

• Stanford University
• University of California, Berkeley
• University of Chicago
• University of Colorado
• University of Delaware
• University of Michigan
• University of Minnesota
• University of Southern California
• University of Texas, Austin
• University of Virginia
• University of Washington
• UW-Madison
• Virginia Tech
• Yale University

• Brown University
• Carnegie Mellon University
• Columbia University
• Cornell University
• Dartmouth College
• Duke University
• Georgetown University
• Harvard University
• Harvard, FAS
• Indiana University
• MIT
• New York University
• Penn State University
• Princeton

18
(No Transcript)
19
(No Transcript)
20
(No Transcript)
21
(No Transcript)
22
Economic Analysis
23
Identify Key Points

• How can MIT develop a program with win-win
  components and incentives for both the faculty
  and MIT administration?
• No magic formula to solve this situation
• CSG survey shows there are different key factors
  for each institution

24
Identify Key Points

• Factors impacting MIT
• Faculty and Institute incentives
• Opportunity cost of space
• Cost of electric power
• Energy initiative to reduce campus carbon
  footprint
• Number of potential production clusters and other
  HPC hardware that may be able to utilize an
  remote-site data center (appendix)

25
Faculty Incentives

• Computational research is now internationally
  recognized as an accepted methodology on par with
  theoretical and experimental research
• Active computational research programs in all MIT
  schools
• Cutting-edge computational research requires
  research infrastructure support (components of
  cyberinfrastructure) either from within facultys
  lab or from Institute
• What incentives will attract faculty to trust
  more economic and scalable Institute options?

26
Institute Incentives

• Funding agencies disallow line items for HPC
  service center charges on grants (overhead)
• MIT has larger more scalable economic resources
• The mechanism for access to Institute-wide
  research computing infrastructure support is
  through a service center model that most faculty
  cannot or will not support
• What might incentivize Institute to provide funds
  for HPC support to faculty and also assure
  recovery of these Institute costs?

27
Issue of Service Centers

• MIT service centers constructed for situations
  where individaual depts/faculty utilize services
  whose costs must be recovered from those specific
  depts/faculty without taxing the broad spectrum
  of depts/faculty at the Institute
• Because computational research is ubiquitous
  across the Institute service center may be an not
  optimal mechanism for charging costs where there
  is Institute-wide practice and usage

28
Economic AnalysisOpportunity Cost of Space

• Without an Institute Data Center any space for
  locating computer clusters at MIT must be
  identified through a combination of
• Utilizing existing space and/or
• Displacing current campus groups to rental space
• Current Cambridge rental space cost is 50/sf
• Based on demand for computer space the
  corresponding rental space needs may
• Range anywhere from 0 sq to 15,600 sf over a 1 to
  10 yr period
• Cost anywhere from 0 to 7.5M based on todays
  rates

29
Economic AnalysisPower Cost Comparison

• Bruns-Pak estimates that 158 racks can be located
  in a 10,000 sq ft data center with operational
  life of 10 yrs
• Current electric rates (2007)
• Cambridge location .136/kw-hr (tied to price
  of natural gas)
• Remote site in MA .04/kw-hr (published rate)
• Assuming no increases in energy costs for natural
  gas over the next 10 year period the total data
  center power cost comparison between Cambridge
  and remote location is
• Cambridge 51M
• Remote site 15M
• Difference 36M
• Any natural gas price increases will increase
  this amount

30
Economic AnalysisWalk the Talk

• On average (worst case scenario) coal fired power
  plants put out 7500 tons of CO2/yr for each
  megawatt of power generated
• Bruns-Pak estimates that 10k / 20k sf data center
  consumes 4.32 / 8.65 MW power which translates
  into 32,500 / 65,000 tons of CO2/yr emitted
  into the atmosphere (all coal)
• MIT Energy Research Council report studied
  projected campus CO2 emissions/year thru 2020
• (http//web.mit.edu/erc/docs/erc-report-060502.pd
  f )

31
(No Transcript)
32
Reduction level in MIT CO2 emissions (white bar)
for a 10k sf HPC remote data center
33
Reduction level in MIT CO2 emissions (white bar)
for a 20k sf HPC remote data center
34
Green Energy

• Potential faculty energy research initiatives
• Architecture green machine rooms
  environmentally manageable buildings
• Integrate geothermal energy into data center
  design
• Better environmental stewardship and options for
  the dams fish ladders
• Flexible air management systems (winter/summer,
  solar)
• Additional research projects
• Favorable initial response from the State of MA

35
Potential Plan

• Potential alternative is build incentive
  mechanisms
• Institute invests funds with mechanisms to
  provide proper type and level of remote support
  for HPC clusters
• No charge to faculty to utilize center
• Institute recovers costs through savings in
  electric power, space change, and rental costs,
  and reduced campus carbon footprint
• Reduced Institute support for HPC computers that
  are candidates for remote site locations but
  choose more expensive options/locations

36
Next Steps for HPC

• Confirm sufficient campus and Lincoln Lab
  production type clusters to utilize data center
• Identify and satisfy faculty needs and
  requirements for locating HPC clusters at remote
  site location
• Demonstrate that faculty needs are being met
  transparently in a production environment

37
Next Steps for HPC (contd)

• Examine specific rental and off-campus sites
  because building, operations, network, and rental
  costs vary at each site
• Properly choose off-campus HPC location that will
  provide incentives to both faculty and the
  Institute
• In case of hydroelectric power determine whether
  there is capacity to displace carbon based
  emissions

38
A Compelling Case for Consideringan Institute
HPC Data Center Option

• Major savings in electric power costs for MIT
• Savings in opportunity costs of space and
  redundant space changes for local computer rooms
• Major opportunity for MIT to walk-the-talk
• reduce carbon footprint campus CO2 emissions
• Helps meet green energy goals
• Initial indications of MA state support
  possible state program to upgrade remote power
  facility gt increased efficiency which further
  decrease carbon impact
• Potential for faculty research green energy
• Potential mechanism to reach agreement for
  Institute investment in data center with
  mechanism to cost recover the funds w/o
  unworkable service centers

39
Discussion
40
Appendices
41
Charge to Computer Space Task Force

• Gather relevant MIT administrative and technical
  information for current data center - co-location
  usage
• Identify, analyze major data center cost factors
• Summarize the key findings for the current
  campus, local commute, and remote data center
  options
• Survey current practices, operations at peer
  institutions.
• Suggest potential business models, options for
  procuring and operating such a facility
• Present the findings and solicit feedback from
  the MIT community
• Present a prioritized set of recommendations to
  the Council for final consideration

42
Four IT-SPARCC Recommendations

• That all Institute building projects (new and
  renovation) require planning and review for
  current and future computing needs for expected
  tenants as part of the Institutes capital
  approval process. Funding considerations should
  be part of this process.
• That the Institute plan for and provide economic
  and scalable computing, communications, and
  "co-location" spaces to meet the current and
  future requirements for computing and
  communications hardware and related equipment.
• That the Provost and EVP use Institute funds for
  co-location where possible and agreeable to all
  parties. Such funding should apply to requests
  for computer infrastructure space, power, cooling
  and network connectivity in co-location spaces
  that have been specifically outfitted for these
  needs. Institute funds should be considered for
  individual DLC sites if the DLC and IST jointly
  present to the Provost and EVP that the Institute
  co-location cannot meet the needed computer
  hardware infrastructure requirements.
• That all DLCs identify computing needs and
  requirements in conjunction with research
  proposals and annual plans, taking a similar
  approach to identification of other capital
  needs. IT-SPARCC will review this information in
  aggregate to ensure that IT considerations feed
  directly into the decisions for Institute capital
  allocation, annual budgets, and rate structures.

43
CSTF Estimate of Generic MIT Data Center
Specifications Supplied to Bruns-Pak

• 10,000 s.f. data center now (6500-7500 s.f.
  useable machine room floor space)
• Potential expansion to 20,000 s.f. data center
• 18 raised floor
• 8 rack height capability with 3 additional
  clearance height to the ceiling
• In the case of catastrophic power loss
• For mission critical administrative computing UPS
  capability for 24 hours with necessary generator
  capacity
• HPC only needs sufficient time for a graceful
  shutdown of hardware)
• 1 Gbit network bandwidth today with 10 Gbit
  capability for HPC within 2 years
• Administrative server racks
• 4-5 kw power requirements today
• estimated power demand of 4 -8 kw by 2010
• High performance computing (HPC) racks
• 10 kw power requirements today
• estimated power demand of 15 -25 kw by 2010
• Power for data center cooling requires an
  additional 30 - 40 over and above that needed to
  power the racks
• Data center floor loading of 4000 lb per HPC rack
  
• Data center floor loading of 3500 lb per disk
  storage rack

44
CSTF Generic MIT Data Center Capacity Planning
Estimate Supplied to Bruns-Pak

• Lincoln Laboratory has estimated that they could
  utilize 50 of a 10,000 sf datacenters useable
  floor space
• It is not possible to predict levels of
  individual faculty/PI campus research computing
  hardware over more than a 1-2 year period.
  However, there are a set of general statements
  that can be used for estimates and capacity
  planning
• Assume MIT has 1000 faculty and that somewhere
  between 5 - 10 utilize high performance
  computing as an integral component in their
  research
• By 2010 this group will on average have 128 node
  HPC cluster for each researcher
• This translates into a range of between 6,400 -
  12,800 nodes of computing hardware
• If there are 25 nodes/rack (low density) there
  will be anywhere between 256 - 512 racks of HPC
  computer hardware utilized by faculty
• If there are 40 nodes/rack (high density) there
  will be anywhere between 160 - 320 racks of HPC
  computer hardware utilized by faculty

45
Examples of New Data Centers
46
UM/MITC Data Center
47
Shared Data Center

• MITC building built and owned by MAVDevelopment,
  opened in January 2005
• 10,000 SF data center shell on lower level
• Capable of supporting a Tier III installation
• U-M 85
• MITC (Internet2/Merit) 15
• Main Electrical Room
• MDF network ISP meet-me space
• Prep and storage space near loading dock

48
(No Transcript)
49
(No Transcript)
50
University of Southern California

• Expansion from the USC 10,000 sq ft facility
• New 30,000 sq ft facility is a combined HPC,
  academic and administrative data center (Jan
  2007)
• 8 MW power available
• Multiple types of areas for different power
  density clusters
• Off campus facility

51
Large HPC Clusters
52
Estimated Size of HPC Clusters for Possible
Location to Institute Data Center

• Large HPC clusters that may have some potential
  option for locating some/all of the clusters off
  campus
• Tier 2 40 racks within 3 yrs
• Chem Eng up to 40 racks within 5 yrs
• LIGO 10-20 racks ??
• EAPS on site today
• DMSE 300 nodes ?? on site today
• Others ?? on site today
• Future funded
• HPC equipment

53
LLGrid Near Term Roadmap

• Growing 3X every year for 4 years.
• Current capability is
• LLGrid 1500 CPUs 750 nodes 40 racks
• Satellites 18 x 2 racks 40 racks
• Total 80 racks

Future growth halted until new data center is
built
54
LLGrid Strategic Roadmap
With Campus FY2008/2009/2010
FY2007
FY2006
DoD HPCMO Investment?
Full Grid 5000 CPUs 400 users 48 satellites
Hardware Capability (CCT)
DoD HPCMO Investment
Optimize Prepare for Next Jump
Full Grid 1500 CPUs 200 users 18 satellites
Full Grid 1500 CPUs 200 users 24 satellites

• Projected near term Lincoln requirements
• LLGrid 5000 CPUs 2500 nodes 120 racks
• Satellites 48 x 2 racks 100 racks
• Total 220 racks
• Long term would expect data center could
  eventually grow to 2x to 3x this

55
Potential HPC Demand Lincoln Lab Experience

• If there is a data center at MIT then faculty and
  LL can compete for HPC clusters as a national
  center
• Federal agencies award large hardware clusters to
  sites with working existing data centers
• If MIT would be awarded HPC equipment under such
  a classification, some of the operating costs
  will likely be covered by the federal sponsor
• Without such a facility MIT and LL cannot compete
  for these opportunities.
• Lincoln Lab has competed for such awards