HPSS National Center for Computational Sciences

1
HPSS - National Center for Computational Sciences

• Mitchell Griffith
• June 7 - 9, 2005
• griffithmr1_at_ornl.gov
• The submitted manuscript has been authored by a
  contractor of the U.S. Government under Contract
  No. DE-AC05-00OR22725. Accordingly, the U.S.
  Government retains a non-exclusive, royalty-free
  license to publish or reproduce the published
  form of this contribution, or allow others to do
  so, for U.S. Government purposes.

2
HPSS at NCCS

• NCCS/HPSS Mission
• Provide Stable, Ample,
• and Nimble storage to
• users and systems
• Currently using HPSS 4.5, upgrade to
• 5.1.1 (3Q2005)
• HSI main transfer utility, ftp (limited support)

3
NCCS HPSS Storage Environment
STK 9310 (4) 9840 (8) 9940B
IBM p630 arm18 1 TB FAStT 700 HPSS Core
Services DCE cds/secd master rep.
STK 9310 (6) 9840 (2) 9940A
IBM SSA 1 TB
IBM H70 arm17 .5 TB SSA 6 - 9840
IBM p660 Fozzie 278 GB SSA 4 - 9840
Brocade Switches 1 - SilkWorm 2800 2 - SilkWorm
3800 1 - SilkWorm 3900
IBM p660 Mooch Disk 1 TB DataDirect
STK 9310 (4) 9840-SCSI (4) 9840
IBM p630 jupiter 1 TB DataDirect 8 - 9940 DCE
cds/secd rep.
IBM 44P beagle 4 - SCSI 9840
IBM p630 saturn HPSS ndapi STK ACSLS 1 TB
DataDirect 4 - 9840 8 - 9940
STK 9310 (8) 9940B
DataDirect S2A8000 5 TB
IBM FAStT700 1 TB
4
NCCS Networking

• Internal networks are GigE, being upgraded to
  10GigE this year.
• External connections to
• ESnet OC192
• Internet2 OC192
• National Lambda Rail 2 x OC192
• UltraScienceNet Testbed up to 16 x OC192

5
NCCS Development Activities

• NCCS is responsible for the software to manage
  HPSS the Storage System Management (SSM)
  components.
• Several elements
• System Manager (SM)
• Data Server (DS) through 5.1
• GUI and command-line interfaces (for operators
  and admins)
• Developers Deryl Steinert, Vicky White and Tom
  Barron with Kathleen Tinch, IBM and Debbie
  Morford, LLNL
• Huge effort
• 310 Java source files 265,000 source lines
• 170 C source files 230,000 source lines
• 272 auto-generated files 173,000 generated lines
• 121,000 executable statements
• 100 screens, 1000 variables

6
NCCS HPSS Statistics

• Production System
• Over 6.8 Million files
• 576 TB stored
• Growing 7TB/week
• Moving 20 TB/day

7
NCCS HPSS Statistics
8
Testing Shared File Systems
Likely to begin testing Lustre as a
shared global File System 4Q - 2005
Large Shared Global Filesystem
Where we are today
HPSS
Where we need to go
For more information on Lustre Filesystem see
http//www.clusterfs.com
9
NDAPI Statistics

• Total Logins -----gt 30
• Total Puts -------gt 606
• Total Put Errors -gt 1
• Total Gets -------gt 128
• Total Get Errors -gt 3

Host(s) Storing Files ------------------ arm18.ccs
.NCCS.gov cheetah48.ccs.NCCS.gov taurus.ccs.NCCS.g
ov
Host(s) Retrieving Files ------------------ cheeta
h48.ccs.NCCS.gov ram1.ccs.NCCS.gov
10
FTP Statistics
Host(s) Storing Files ----------------------- chee
tah48.ccs.NCCS.gov taurus.ccs.NCCS.gov
Host(s) Retrieving Files ------------------------
cheetah48.ccs.NCCS.gov ram1.ccs.NCCS.gov

• Total Puts -------gt 606
• Total Gets -------gt 128

11
NCCSs Science Support

• NCCS supports research in a number of scientific
  disciplines
• with the HPCC and Storage resources located in
  the National
• Center for Computational Sciences (NCCS).
• Astrophysics -- http//www.ccs.NCCS.gov/astro/
• Atmospheric Radiation Measurement --
  http//www.arm.gov/docs/
• Climate and Carbon research -- http//www.ccs.NCCS
  .gov/CCR/
• Computational Biology -- http//www.ccs.NCCS.gov/c
  bi/
• Computational Materials -- http//www.ccs.NCCS.gov
  /mri/
• Computational Sciences -- http//www.csm.NCCS.gov/
  
• Fusion Simulation -- http//www.ccs.NCCS.gov/fsi/
• Nanoscience -- http//www.cnms.NCCS.gov/
• Neutron Sciences -- http//www.sns.gov/
• Transportation -- http//www.NCCS.gov/info/NCCSrev
  iew/v33_3_00/features.htm

12
Atmospheric Radiation Measurement (ARM) Program

• The ARM Program is the largest global change
  research program supported by the U.S. Department
  of Energy (DOE).
• The ARM Programs primary objective is to develop
  and test parameterizations of clouds and their
  effect on the radiative energy balance, with the
  ultimate goal of improving general circulation
  models used for climate research and prediction.
• To meet this goal, data is gathered at a number
  of sites
  around the globe to measure key aspects of the
  radiation field under
  vastly different climate conditions.
• ARM data stored in HPSS at NCCS accounts
• for 25 of the data stored, and half of the
• total number of files.

The ARM Program's award-winning Millimeter Wave
Cloud Radar (MMCR) is an active sensor that can
measure cloud boundaries and microphysical
properties with high space and time resolution.
This data sample shows a thunderstorm passing
above the Central Facility at ARM's Southern
Great Plains site

• For more information see
• http//www.arm.gov/docs/
• http//www.arm.gov/docs/data

13
Computational Platforms _at_ NCCS

• Extremely low latency, high bandwidth,
  interconnect
• Efficient scalar processors, balanced
  interconnect
• Shares interconnect technology with Cray X2
• Front end-hosted environment - system calls and
  I/O

• Proven architecture for performance and
  reliability
• Most-powerful processors and interconnect
• Scalable, globally addressable memory and
  bandwidth
• Leverages commodity where possible
• Offers capability computing for key applications

optimized for Applications
14
Cray X1 - Phoenix

• Largest X1 in the world today at 512 MSPs (6.4
  TF)
• 4 GB of memory per MSP (2 TB total)
• 32 TB of directly attached disks

• System will be upgraded to X1E over the summer
• 1,024 MSPs
• 18.5 TF

15
Cray XT3 - Jaguar

• Cray XT3 (based on Red Storm _at_ Sandia)
• 5000 processors
• (Opteron)
• 10 TB memory
• 120 TB disk
• 25 TeraFlops

16
Wish List

• Real Relational Database (HPSS 5.1)
• Real-time job monitor (Mikes Monitor)
• Better small file support
• BBCP-like HSI
• HPSS/Lustre integration
• Tuning Document (best practice)

17
Questions?THE END
18
Potential Breakthroughs
19
The NCCS Facility

• The 170,000 square foot Eugene P. Wigner Center
  for Computational Sciences Building is home to
  the National Center for Computational Sciences
  (NCCS) 40,000 sq. ft. computing facility for
  unclassified scientific computing.
• A unique feature of the building is that it was
  built entirely using private money and is owned
  by a private developer who leases it to
  UT-Battelle Development Corp. who then subleases
  it to the DOE. Construction was completed in 15
  months from breaking ground to move-in.

20
Neutron Science The TeraGrid

• NCCS intends to bring neutron science to the
  TeraGrid through the Southeastern TeraGrid
  Extension for Neutron Science (SETENS).
• Provide access to active experimental tools
• High Flux Isotope Reactor (HFIR)
• Spallation Neutron Source (SNS)
• Access to live and archived neutron scattering
  data
• Real-time access to neutron science data
• Archived data stores through HPSS

• For more information see
• http//www.csm.NCCS.gov/
• http//www.sns.gov/
• http//www.teragrid.org