Holistic, Energy Efficient Design Cardiff

1
Holistic, Energy Efficient Design_at_ Cardiff
Going Green Can Save Money

• Dr Hugh Beedie
• CTO
• ARCCA INSRV

2
Introduction

• The Context
• Drivers to Go Green
• Where does all the power go ?
• Before the equipment
• In the equipment
• What should we do about it ?
• What Cardiff University is doing about it

3
The Context (1)

• Cardiff University receives 3M grant to purchase
  a new supercomputer
• A new room is required to house it, with
  appropriate power, cooling, etc
• 2 tenders
• Data Centre construction
• Supercomputer

4
The Context (2)

• INSRV Sustainability Mission To minimise CUs IT
  environmental impact and to be a leader in
  delivering sustainable information services.
• Some current recent initiatives
• University INSRV Windows XP image default
  settings
• Condor saving energy, etc compared to a
  dedicated supercomputer
• ARCCA INSRV new Data Centre
• PC Power saving project standby 15 mins after
  logout (being implemented this session)

5
Drivers Why do Green IT?

• Increasing demand for CPU Storage
• Lack of Space
• Lack of Power
• Increasing energy bills (oil prices doubled)
• Enhancing the Reputation of Cardiff University
  attracting better students
• Sustainable IT
• Because we should (for the Planet )

6
Congress Report Aug 2007

• US Data Centre electricity demand doubled
  2000-2006
• Trends toward 20kW per rack
• Large scope for efficiency improvement
• Obvious more efficiency at each stage
• Holistic approach necessary facility and
  component improvements
• Less obvious virtualisation (up to 5X)

7
Where does all the power go? (1)

• Up to 50 is used before getting to the Server
• Report to US Congress Aug 2007
• Loss 50,000 p.a. for every 100kW
• supplied to the room

8
Where does all the power go? (2)
9
Where does all the power go? (3)

• How ?
• Power Conversion - before it gets to your room,
  you lose in the HV-gt LV transformer
  Efficiency98 not 95
• Return On Investment (ROI)?
• New installation, ROI 1 month
• Replacement, ROI 1 year
• Lifetime of investment 20 yrs !!!!!

10
Where does all the power go? (4)

• How?
• Cooling infrastructure
• Typical markup 75
• Lowest markup 25-30 ?
• Est ROI 2-3 years (lifetime 8 years)

11
Where does all the power go? (4)

• How?
• Backup power (UPS) ( load vs efficiency)
• Efficiency 80-95
• Est. ROI for new installation - lt1year
• Replacement not so good, UPS life 3-5 yrs only ?

12
Where does it go? Bull View
13
Where does it go? Intel View
14
Where does it go? APC View
15
Server Power Consumption
16
Options for Cardiff (1)

• Carry on as before
• Dual core HPC solutions
• Wait for quad core
• Saves on Flops/watt
• Saves on infrastructure (fewer network ports)
• Saves on management (fewer nodes)
• Saves on space
• Saves on power

17
Options for Cardiff (2)

• High density solution
• Needs specialist cooling over 8kW per rack
• Carry on as before (6-8kW per rack)
• Probable higher TCO
• Low density solution (typically)
• BT free air cooling
• Allow wider operating temp range warranty
  issues ?
• Not applicable here (no space)

18
What did Cardiff do? (1)

• Ran 2 projects
• HPC equipment
• Environment
• TCO as key evaluation criterion
• Plus need to measure and report on usage
• Problems
• Finger pointing (strong project mgt)
• Scheduling (keep everyone in the loop)

19
Timetable
20
What did Cardiff do? (2)

• Bought Bull R422 servers for HPC
• 80W Quad core Harpertown
• 2 dual socket, quad core servers in 1U common
  PSU
• Larger fans (not on CPU)
• Other project in same room
• IBM Bladecentres
• Some pizza boxes

21
Back Up Power
APC 160kW Symmetra UPS

• Full and half load efficiency 92
• Scaleable Modular could grow as we grew
• Strong environment management options
• Integrated with cooling units
• SNMP
• Bypass capability
• Bought 2 (not fully populated)
• 1 for compute nodes
• 1 for management and another project

Enhanced existing standby generator
22
Cooling Inside the Room
APC Netshelter Airflow
APC Inline RC units Provides residual cooling
to the room Resilient against loss of an RC
unit Cools hot air without mixing with cold air
Front
Servers
InRow CoolingUnit
InRow Cooling Unit
23
Cooling Outside the Room
3 Airedale 120kW Chillers Ultima Compact Free
Cool 120kW Quiet model Variable speed fans N1
arrangement
24
Free-cooling vs Mechanical cooling
System operated on mechanical cooling only(38
of Year)
System operated on 100 Free-Cooling (12 of
Year)
System operated on partial Free-Cooling (50 of
Year)
Cooling Load
-7oC
3.5oC
12.5oC
Mechanical (compressor) cooling
Ambient Temp.
Free-cooling
25
Cost Savings Summary

• Low loss transformer
• 10k p.a.
• UPS
• 20k p.a.
• Cooling
• 50k p.a. estimated
• Servers
• 80W part - 20k p.a.
• Quad core same power but twice the grunt

26
Lessons learned from SRIF3 HPC Procurement -
Summary

• Strong project management essential
• IT and Estates liaison essential but difficult
• Good Supplier relationship essential
• Major savings possible
• Infrastructure (power cooling)
• Servers (density and efficiency)