Commodity Data Center Design

1
Commodity Data Center Design

• James Hamilton
• 2007-10-08
• JamesRH_at_microsoft.com
• http//research.microsoft.com/jamesrh

2
Containerized Products
Nortel Steel Enclosure Containerized telecom
equipment
Caterpillar Portable Power
Rackable Systems Concentro 1,152 Systems in 40
(9,600 cores/3.5 PB)
Datatainer ZoneBox
Rackable Systems Container Cooling Model
Google WillPower Will Whitted
Petabox Internet Archive Brewster Kahle
Sun Project Black Box 242 systems in 20
3
Cooling, Feedback, Air Handling Gains
Verari
Intel

• Tighter control of air-flow increased delta-T and
  overall system efficiency
• Expect increased use of special enclosures,
  variable speed fans, and warm machine rooms
• CRACs closer to servers for tighter temp control
  feedback loop
• Container takes one step further with very little
  air in motion, variable speed fans, tight
  feedback between CRAC and load

Intel
4
Shipping Container as Data Center Module

• Data Center Module
• Contains network gear, compute, storage,
  cooling
• Just plug in power, network, chilled water
• Increased cooling efficiency
• Variable water air flow
• Better air flow management (higher delta-T)
• 80 air handling power reductions (Rackable
  Systems)
• Bring your own data center shell
• Just central networking, power, cooling, security
  admin center
• Can be stacked 3 to 5 high
• Less regulatory issues (e.g. no building permit)
• Avoids (for now) building floor space taxes
• Political/Social issues
• USA PATRIOT act concerns regional restrictions
• Move resources closer to customer (CDN
  mini-centers)
• Single customs clearance on import
• Single FCC compliance certification
• Distributed, incremental fast built mini-centers

5
Manufacturing H/W Admin. Savings

• Factory racking, stacking packing much more
  efficient
• Robotics and/or inexpensive labor
• Avoid layers of packaging
• Systems-gtpacking box-gtpallet-gtcontainer
• Materials cost and wastage and labor at customer
  site
• Data Center power cooling expensive consulting
  contracts
• Data centers are still custom crafted rather than
  prefab units
• Move skill set to module manufacturer who designs
  power cooling once
• Installation design to meet module power,
  network, cooling specs
• More space efficient
• Power densities in excess of 1250 W/sq ft
• Rooftop or parking lot installation acceptable
  (with security)
• Stack 3 to 5 high
• Service-Free
• H/W admin contracts can exceed 25 of systems
  cost
• Sufficient redundancy that it just degrades over
  time
• At end of service, return for remanufacture
  recycling
• 20 to 50 of systems outages caused by Admin
  error (A. Brown D. Patterson)

6
Systems Power Density

• Estimating datacenter power density difficult
  (15 year horizon)
• Power is 40 of DC costs
• Power Mechanical 55 of cost
• Shell is roughly 15 of DC cost
• Cheaper to waste floor than power
• Typically 100 to 200 W/sq ft
• Rarely as high as 350 to 600 W/sq ft
• Modular DC eliminates impossible shell to power
  trade-off
• Add modules until power is absorbed
• 480VAC to container
• High efficiency DC distribution within
• High voltage to rack can save gt5 over 208VAC
• Over 20 of entire DC costs is in power
  redundancy
• Batteries able to supply up to 12 min at some
  facilities
• N2 generation at over 2M each
• Instead, use more smaller, cheaper data centers
• Eliminate redundant power bulk of shell costs
• Resource equalization

7
Where do you Want to Compute Today?
Slides posted soon to http//research.microsoft.c
om/JamesRH