ESS Rapid Response Meeting

1
ESS Rapid Response Meeting

• Horst D. Simon
• NERSC/LBNL
• May 15, 2002

2
Earth Simulator

• 40 Tflop/s system in Japan
• completion April 2002
• driven by climate and earthquake simulation
  requirements
• built by NEC
• 640 CMOS vector nodes

3
Earth Simulator Building
4
Earth Simulator
5
ESS configuration of a general purpose
supercomputer

• Processor Nodes (PN) Total number of processor
  nodes is 640. Each processor node consists of
  eight vector processors of 8 GFLOPS and 16GB
  shared memories. Therefore, total numbers of
  processors is 5,120 and total peak performance
  and main memory of the system are 40 TFLOPS and
  10 TB, respectively. Two nodes are installed
  into one cabinet, which size is 40x56x80. 16
  nodes are in a cluster. Power consumption per
  cabinet is approximately 20 KVA.
• 2) Interconnection Network (IN) Each node is
  coupled together with more than 83,000 copper
  cables via single-stage crossbar switches of
  16GB/s x2 (Load Store). The total length of the
  cables is approximately 1,800 miles.
• 3) Hard Disk. Raid disks are used for the system.
  The capacities are 450 TB for the systems
  operations and 250 TB for users.
• 4) Mass Storage system 12 Automatic Cartridge
  Systems (STK PowderHorn9310) total storage
  capacity is approximately 1.6 PB.

6
ESS complete system installed 4/1/2002
7
ESS highlights four failure modes of US
investments in supercomputing

• Lack of investment in supercomputing capability
  computing for basic science
• Lack of interest/incentives for US vendors to
  invest in the supercomputing market
• Lack of investment in basic research in
  supercomputing architectures and technologies in
  universities
• Lack of coordination among agencies (and within
  Office of Science)
• We are not in crisis mode because of ESS and
  Japan, but because of seven years of neglect of
  strategy and investments in supercomputing.

8
ESS- applications results

• Test run on global climate model reported
  sustained performance of 14.5 TFLOPS on 320
  nodes. The model was an atmospheric global
  climate model (T1279L96) developed originally by
  CCSR/NEIS and tuned by ESS.
• The next best climate result reported in the
  US is about 375 Gflop/s on NERSC-3 a factor of
  40 less (and this is using only half of the ESS
  configuration)

• MOM3 result (code from GFDL/Princeton). The
  horizontal resolution is 0.1 degrees and the
  number of vertical layers is 52. It took 275
  seconds for a week simulation by means of 175
  nodes.
• A real full scale application result!

9
Kflops per Inhabitant
10
Vendor Issues

• Only two US vendors left with innovative
  architecture ideas there were at least ten in
  1992
• Slow rate of innovation in interconnects
• IBM has only three generations in a decade
• reliance on technology from small companies
  (Quadrics, Myrinet)
• lost T3E investment
• Only US investment in vendors since 1995 is ASCI
  Path Forward
• at best tens of M versus hundreds of M for ESS
• impact of DARPA HPCS not clear yet

11
Basic Research Issues/Observations

• Only a handful of supercomputing relevant
  computer architecture projects at US
  universities versus of the order of 50 in 1992
• Lack of interest in supporting supercomputing
  relevant basic research
• parallel language and tools research has been
  almost abandoned
• focus on grid middleware and tools
• WIMPS2002 Petaflops 1997
• no significant progress in five years

12
Lack of Coordination and Long Term Planning

• DOE Office of Science has no overall computing
  roadmap
• supercomputing investments are made locally
  (optimized by institution or program)
• DOE NNSA (ASCI) investments are focused on narrow
  mission
• relevance to computation for basic science is
  not the primary objective

13
Required Steps

• Investment in capability supercomputing to close
  the gap
• a) Develop productive ways to nurture
  relationships with vendors
• b) Provide incentives for vendors to develop new
  technology
• 3. Initiate new programs in university EE and CS
  departments to stimulate SC architecture and
  technology research provide coupling to basic
  science applications
• 4. Start develop long term roadmaps for SC
  technology (similar to SIA roadmaps)