SimMillennium: Computer Systems, Computational Science and Engineering in the Large

1
SimMillenniumComputer Systems, Computational
Science and Engineering in the Large

• Jim Demmel, David Culler
• E. Brewer, J. Canny,
• A. Joseph, J. Landay, S. McCanne
• A. Neureuther, C. Papadimitrou, C. Sequin, K.
  Yelick
• EECS, UC Berkeley
• www.millennium.berkeley.edu
• NSF CISE EIA RI and MII PIs Workshop
• Aug 7-9 1999

2
Project Goals

• Enable major advances in Computational Science
  and Engineering
• Simulation, Modeling, and Information Processing
  becoming ubiquitous
• Many participants outside CS
• Explore novel design techniques for large,
  complex systems
• Fundamental Computer Science problems ahead are
  problems of scale
• Use Capitalism, not Socialism (i.e. not Computer
  Center)
• Develop fundamentally better ways of
  assimilating and interacting with large volumes
  of information
• and with each other
• Explore emerging technologies
• networking, OS, devices

3
Outline

• Background on UC Berkeley
• Millennium infrastructure description
• Other infrastructure contributions
• Systems research
• Networking
• Computational Economy
• Applications
• List of all participants
• A few highlights
• Conclusions

4
Background at UC Berkeley

• Mammoth NSF RI (1988-1993)
• CM-5
• Titan NSF RI (1994-1999)
• Culler, spoke yesterday
• NOW Network of Workstations
• Curriculum
• CS 267 - Applications of Parallel Computing
• MS in Comp Sci Eng Curriculum
• Proposed Comp Eng Sci undergrad program
• NERSC at LBNL
• National Energy Research Scientific Computing
  Center
• Supercomputer center next to campus
• SimMillennium (1998-2003)

5
Planned Millennium Infrastructure
6
The Community
Business
School of Info. Mgmt and Sys.
BMRC
Chemistry
Computer Science
Electrical Eng.
Biology
Astro
Mechanical Eng.
Physics
Nuclear Eng.
Math
IEOR
Inst. Of Transport
Economy
Civil Eng.
Geo
7
NT Workstations for Sci. Eng.
Business
SIMS
BMRC
Chemistry
C.S.
E.E.
Biology
Astro
M.E.
Physics
N.E.
Math
IEOR
Transport
Economy
Civil Eng
Geo
8
SMP storage, small-scale parallelism
Business
SIMS
BMRC
Chemistry
C.S.
E.E.
Biology
Astro
M.E.
Physics
N.E.
Math
IEOR
Transport
Economy
Civil Eng
Geo
9
Group Cluster of SMPs Parallelism
Business
SIMS
BMRC
Chemistry
C.S.
E.E.
Biology
Astro
NERSC
M.E.
Physics
N.E.
Math
IEOR
Transport
Economy
Civil Eng
Geo
10
Campus Cluster large-scale Parallelism
Business
SIMS
BMRC
Chemistry
C.S.
E.E.
Biology
Astro
NERSC
M.E.
Physics
N.E.
Math
IEOR
Transport
Economy
Civil Eng
Geo
11
Gigabit Ethernet Connectivity
Business
SIMS
BMRC
Chemistry
C.S.
E.E.
Biology
Gigabit Ethernet
Astro
NERSC
M.E.
Physics
N.E.
Math
IEOR
Transport
Economy
Civil Eng
Geo
12
Physical Connectivity
13
Visualization and Novel User Interfaces
14
Current Infrastructure (by end of 8/99)

• All 195 desktops and 20 SMPs delivered
• All 18 16-(or smaller)-processor clusters (8 dual
  SMPs)
• Mostly running Myrinet interconnects, some 100Mb
  Ethernet
• NOW functioning as large central cluster
• Cluster build service
• Millennium wide services
• .5 Tbyte file server
• distributed system services and software for UNIX
  and NT
• Optical fiber for Gigabit in place
• 10 Gigabit switches purchased
• 2 Vision Maker Digital Desks purchased

15
A Millennium Cluster

• 16x2 Processor
• 400 MHz Pentium II
• 100 MHz Memory Bus
• 33 MHz 32-Bit PCI
• 100BaseTX Ethernet
• Myrinet M2F
• Windows NT 4.0 or LINUX
• Terminal Server Edition

16
Industrial / Academic Collaboration

• Computers via Intel Technology 2000 grant
• 200 NT desktops
• 16 department 4-way SMPs
• 10 8x2 Group Clusters,
• 1 200x2 Campus Cluster
• PPro Pentium II Pentium III
• Additional storage via IBM SUR grant
• 0.5 TB this year 4 TB
• NT tools via Microsoft grant
• Solaris x86 tools via SMCC grant
• Nortel discounts the gigabit Ethernet 70
• Campus provides 3 technical staff, fiber
• Research provides the prog. and system support

200 Gflop/s 150 GB memory 8 TB disk
17
What NSF is paying for

• Fast internal networks for clusters
• Gigabit ethernet switches
• Interesting I/O devices
• Large displays
• 3D glasses
• Haptic mice
• Position sensors
• One staff person

18
Primary Faculty Participants - 1

• CS
• D. Culler, J. Demmel, E. Brewer, J. Canny, A.
  Joseph, R. Katz, J. Landay, S. McCanne, C.
  Papadimitriou, C. Sequin, R. Wilensky, K. Yelick
• Systems, Numerical Methods, Services, HCI,
  Networking, Computational Economics, Digital
  libraries, Parallel languages
• EE
• A. Neureuther
• Technology CAD for EBEAM Lithography
• Civil Engineering
• S. Govindjee, G. Fenves
• Earthquake Engineering, Finite Element Modeling
• Physics
• B. Price, J. Wurtele, D. Lowder
• Processing neutrinos and muons at South Pole
• SIMS
• H. Varian, R. Larson, M. Hearst
• Computational Economics, User Interfaces

19
Primary Faculty Participants - 2

• Astronomy
• J. Arons, C. McKee. P. Marcus
• Star Formation, Geophysical Turbulence
• Transportation Studies
• S. Sastry, A. Kanafani
• Redesign of Nations Air Traffic Control System

20
Secondary Faculty Participants - 1

• Geology/Geophysics
• M. Richards, D. Dreger
• Mantle modeling
• Math
• D. Eisenbud, B. Poonen, A. Grunbaum, T. Slaman,
  B. Sturmfels, P. Vojta
• Crystal growth modeling, tomography, symbolic
  computing
• Berkeley Multimedia Research Center
• L. Rowe
• Video effects processing
• Mechanical Engineering
• V. Carey, M. Frenklach, A. Packard, P.
  Papadopoulos, P. Marcus
• Modeling Automated Highways, Material Processing
• Biology
• D. Lindberg, S. Brenner
• Reconstruct Phylogenetic Tree of Life, Genome
  studies
• Nuclear Engineering
• J. Vujic
• Planning radiotherapy for Brain Tumors

21
Secondary Faculty Participants - 2

• NERSC
• W. Saphir
• Business
• N. Hakansson
• Computational Finance Laboratory
• Chemistry
• K. Durkin, D. Chandler, D. Harris, W. Lester, W.
  Miller, R. Stevens, B. Whaley
• Computational Chemistry, Molecular Dynamics
  Visualization
• Economics
• A. Nevo
• Market Modeling

22
The CS Research Agenda

• High Performance Cluster Computing Environment
• Fast communication on Clusters of SMPs
• Compiler Techniques for Performance and Ease of
  use
• Numerical Techniques and Solvers
• Particles, FFT, AMR, Multigrid, Sparse and Dense
  Lin. Alg.
• Novel System Design Techniques
• Clusters of clusters
• Computational Economy
• Open infrastructure services
• Novel modes of interacting with large amounts of
  data
• User interfaces, Digital Libraries

23
Communication Interface Revolution

• Low Overhead Communication Happens
• Academic Research put it on the map
• Active Messages (AM), FM, PM, Unet
• Memory Messaging (Get/Put, Reflective, VMMC, Mem.
  Chan.)
• Intel / Microsoft / Compaq recognized it
• Virtual Interface Architecture 1.0 released
  12/16/97
• Berkeley VIA over Myrinet released on NT and
  Linux

VIA
24
World-Record Datamation Sort
Old Record (NOW)
25
Computational Economy Approach

• System has a supply of various resources
• Demand on resources revealed in price
• distinct from the cost of acquiring the resources
• User has unique assessment of value
• Client agent negotiates for system resources on
  users behalf
• submits requests, receives bids or participates
  in auctions
• selects resources of highest value at least cost

26
Advantages of the Approach

• Decentralized load balancing
• according to users perception of what is
  important, not systems own metric
• adapts to system and workload changes
• Creates incentive to adopt efficient modes of use
• exploit under-utilized resources
• maximize flexibility (e.g., migratable,
  restartable applications)
• Establishes user-to-user feedback on resource
  usage
• basis for exchange rate across resources
• Powerful framework for system design
• Natural for client to be watchful, proactive, and
  wary
• Generalizes from resources to services
• Rich body of theory ready for application

27
Current Prototype

• Specify procs p and value on job
• rexec -n 16 -value 20 fft.mpi
• Market-based Proportional Sharing
• Bidder i gets fraction bi / Sk bk of resource
• If one bidder, no cost to use resource
• Resource may be CPU, Memory, Network, I/O
• Existing OS mechanisms/policies insufficient
• New proportional CPU schedule for LINUX
• New page replacement policy
• Game theoretic analysis
• Preliminary experience in CS267
• Students trusted system to allocate fairly, so
  they did not try to flood system with jobs
• Future work
• other mechanisms and analysis (Vikrey auction,
  batch vs interactive)
• package up and market services (make, popular
  simulators, DB search)

28
Application Highlights

• PEER - NSF Earthquake Engineering Center
• FE modeling of Bay Area during Big One
• Need better parallel sparse linear system solvers
• EBEAM - Electron Beam Lithography
• Simulate next generation chip manufacturing
• Need better parallel N-body force calculation
• AMANDA
• Antarctic Muon and Neutrino Detector Array
• Need to process many detector tracks for events
• Web Page Design
• Better human interfaces using novel devices
• Digital Library
• Support access to large active document
  collection

29
Earthquake Modeling

• PEER Pacific Earthquake Engineering Research
  Center
• UC, Caltech, Stanford, USC, U Washington
• Model behavior of civil infrastructure in Big One
• Improve earthquake resistant designs
• Requires large scale FE models
• Buildings, roads, bridges, etc. coupled to ground
• Simulate effects of earthquakes
• Requires solution of very large sparse linear
  systems
• Collaboration on software
• G. Fenves - OO Finite Element Modeling System
• J. Demmel - direct and iterative parallel linear
  equation solvers
• Prometheus - a multigrid solver for FE problems
  (M. Adams)
• SuperLU - sparse Gaussian elimination (X. Li)
• Port from Cray T3E to Millennium

30
SuperLU Scales well on Millennium
31
Millennium sometimes beats a Cray T3E
32
EBEAM - Lithography Simulation

• A. Neureuther and J. Demmel
• Simulate future chip manufacturing devices which
  will use electron beams instead of light
• Computational Bottleneck computing electrostatic
  forces on electrons
• Pbody (D. Blackston)
• Parallel O(N) or O(N log N) N-Body code
• Incorporates Barnes-Hut / Fast Multipole Method /
  Anderson in unified framework
• Portable across many platforms
• Easy to tune for accuracy and performance
• Now used in production runs
• Will be packaged as Web service on Millennium,
  with other TCAD tools

33
Performance of Pbody

• Over 90 efficient on 4 Millennium procs
• 61 secs for 200K electrons on 1 proc
• (vs 46 secs for Cray T3E)
• 500x faster than direct O(N2) method

34
AMANDA

• Antarctic Muon And Neutrino Detector Array
• International project to detect particles in 1
  km3 of ice at South Pole
• 98 scientists, 15 universities, 4 countries
• Millennium uses
• Machine at South Pole for data collection,
  webcast, teleconference
• Used in PBS broadcast Passport to Knowledge
  Live from the Pole in 1998 linking
  schoolchildren in Mississippi to South Pole crew
• Simulation of AMANDA events and calculating
  optical properties of the ice

35
(No Transcript)
36
Better Interfaces for Web Site Design

• J. Landay, Raecine Sapien (SUPERB student)
• Most web designers do not like to program, edit
• Provide a more natural user interface that
  matches their style
• Exploit large displays, position trackers, vision
  and gesture recognition to make design easier and
  faster
• Prototype (minus vision) built over summer,
  evaluated on a group of designers

37
Web Site Design Issues
Taken from Contextual Design Beyer Holtzblatt
38
System Components

• Physical components
• Vision system
• CrossPad
• Digital Desk
• Command Area

39
Second User Tests
40
Digital Libraries Rethinking Scholarly
Information Dissemination and Use

• Robert Wilensky
• Principal Investigator
• David Forsyth
• Co-principal Investigator
• The UC Berkeley Digital Library Team

41
Goal Complete Rethinking of How we Use
Information

• Must support
• entire information cycle creation,
  dissemination and collaboration
• in addition to organization, access, presentation
  and preservation
• non-textual material (photos, video, maps)
• in addition to text-based content
• primary data sources, informal publication
• as well as traditional archival product
• radically new modes of use
• Scholarly information use is an especially
  attractive place to start.

42
GIS Viewer Streetfinder example
43
GIS Viewer Example
44
ConclusionsWhat is Millennium About?

• An experiment in large-scale system design
• Advance the state of computational science and
  engineering
• Exploring novel design techniques
• Exploring important new technologies
• NSF support essential