The OptIPuter

1
The OptIPuter From SuperComputers to
SuperNetworks

• Vanguard NextGens Conference
• Coronado, CA
• November 19, 2002

Dr. Larry Smarr Director, California Institute
for Telecommunications and Information
Technologies Professor, Dept. of Computer Science
and Engineering Jacobs School of Engineering, UCSD
2
California Has Initiated Four New Institutes for
Science and Innovation
California Institute for Bioengineering,
Biotechnology, and Quantitative Biomedical
Research
Center for Information Technology Research in
the Interest of Society
UCD
UCM
UCB
UCSF
California NanoSystems Institute
UCSC
California Institute for Telecommunications and
Information Technology
www.ucop.edu/california-institutes
3
Cal-(IT)2An Integrated Approach to the Future
Internet
220 UC San Diego UC Irvine Faculty Working in
Multidisciplinary Teams With Students, Industry,
and the Community
The States 100 M Creates Unique Buildings,
Equipment, and Laboratories
www.calit2.net
4
Over Fifty Industrial Sponsors From a Broad
Range of Industries
Computers Communications Software Sensors Biomedic
al Startups Venture Capital
Newport Corporation Oracle Orincon
Industries Panoram Technologies Printronix QUALCOM
M Quantum The R.W. Johnson
Pharmaceutical Research Institute SAIC Samueli,
Henry (Broadcom) SciFrame, Inc. Seagate Storage
Products SGI Silicon Wave Sony STMicroelectronics,
Inc. Sun Microsystems TeraBurst Networks Texas
Instruments Time Domain UCSD Healthcare WebEx

• Akamai Technologies Inc.
• AMCC
• Ampersand Ventures
• Arch Ventures
• Avalon Ventures
• The Boeing Company
• Broadcom Corporation
• CAIMIS, Inc.
• Chiaro Networks
• Conexant Systems, Inc.
• Connexion by Boeing
• Cox Communications
• Diamondhead Ventures
• Dupont
• Emulex Corporation Network Systems
• Enosys Markets
• Enterprise Partners
• Entropia, Inc.
• Ericsson

IdeaEdge Ventures The Irvine Company Intersil
Corporation Irvine Sensors Corporation JMI,
Inc. Leap Wireless International Link, William J.
(Versant Ventures) Litton Industries,
Inc. MedExpert International Merck Microsoft
Corporation Mission Ventures NCR
140 Million in Industrial Matching
5
Closing in on the Dream
Boston
What we really have to do is eliminate distance
between individuals who want to interact with
other people and with other computers. ? Larry
Smarr, Director National Center for
Supercomputing Applications, UIUC
Illinois
Using satellite technologyto demo what It might
be like to have high-speed fiber-optic links
between advanced computers in two different
geographic locations. ? Al Gore, Senator Chair,
US Senate Subcommittee on Science, Technology and
Space
SIGGRAPH 89 Science by Satellite
Source Maxine Brown, EVL, UIC
6
The Move to Data-Intensive Science
Engineering-e-Science Community Resources
LHC
ATLAS
7
Four Large Hadron Collider Experiments The
Petabyte to Exabyte Challenge

• ATLAS, CMS, ALICE, LHCBHiggs New particles
  Quark-Gluon Plasma CP Violation

Data stored 40,000 Terabytes/Year and UP
Source Harvey Newman, Caltech
8
NIH is Funding a Brain Imaging Federated
Repository
Biomedical Informatics Research Network (BIRN)
NIH Plans to Expand to Other Organs and Many
Laboratories
Part of the UCSD CRBS Center for Research on
Biological Structure
National Partnership for Advanced Computational
Infrastructure
9
NSFs EarthScope Rollout Over 14 Years Starting
With Existing Broadband Stations
10
The Grid is the EmergingDistributed
Cyberinfrastructure Middleware
Applications and Capabilities
Open Grid Services Architecture (OGSA) Integrates
Globus with Web Services
11
Grid Computing is Becoming Mainstream
12
Why Optical Networks Are Emerging as the 21st
Century Driver for the Grid
Scientific American, January 2001
13
The Crossing Exponentials Requires a Rethinking
of Grid Architecture
A global economy designed to waste transistors,
power, and silicon area -and conserve bandwidth
above all- is breaking apart and reorganizing
itself to waste bandwidth and conserve power,
silicon area, and transistors." George Gilder
Telecosm (2000)
14
The Rapid Increase in Bandwidth is Driven by
Parallel Lambdas On Single Optical Fibers
(WDM)
Parallel Lambdas Will Drive This Decade The Way
Parallel Processors Drove the 1990s
15
A LambdaGrid Will Be the Backbone for an
e-Science Network
Apps Middleware
Clusters
C O N T R O L P L A N E
Dynamically Allocated Lightpaths
Switch Fabrics
Physical Monitoring
Source Joe Mambretti, NU
16
The Next S-Curves of NetworkingExponential
Technology Growth

Lambda Grids
Experimental Networks
Production/ Mass Market
DWDM
100
Technology Penetration
Internet2 Abilene
Experimental/ Early Adopters
Connections Program
0
Research
Gigabit Testbeds
Time Technology S-Curve
1990s 2000
2010 Networking Technology S-Curves
17
Data Intensive Scientific Applications Require
Experimental Optical Networks

• Large Data Challenges in Neuro and Earth Sciences
• Each Data Object is 3D and Gigabytes
• Data are Generated and Stored in Distributed
  Archives
• Research is Carried Out on Federated Repository
• Requirements
• Computing Requirements ? PC Clusters
• Communications ? Dedicated Lambdas Over Fiber
• Data ? Large Peer-to-Peer Lambda Attached Storage
  
• Visualization ? Collaborative Volume Algorithms
• Response
• OptIPuter Research Project

18
Illinois I-WIREThe First State Dark Fiber
Experimental Network
Source Charlie Catlett 12/2001
19
iGrid 2002 September 24-26, 2002, Amsterdam, The
Netherlands

• Fifteen Countries/Locations Proposing 28
  Demonstrations Canada, CERN, France, Germany,
  Greece, Italy, Japan, The Netherlands, Singapore,
  Spain, Sweden, Taiwan, United Kingdom, United
  States
• Applications Demonstrated Art, Bioinformatics,
  Chemistry, Cosmology, Cultural Heritage,
  Education, High-Definition Media Streaming,
  Manufacturing, Medicine, Neuroscience, Physics,
  Tele-science
• Grid Technologies Grid Middleware, Data
  Management/ Replication Grids, Visualization
  Grids, Computational Grids, Access Grids, Grid
  Portal

Sponsors HP, IBM, Cisco, Philips, Level (3),
Glimmerglass, etc.
UIC
www.startap.net/igrid2002
20
iGrid 2002 Was Sustaining 1-3 Gigabits/s
Total Available Bandwidth Between Chicago and
Amsterdam Was 30 Gigabit/s
21
The NSF TeraGridA LambdaGrid of Linux
SuperClusters
This will Become the National Backbone to Support
Multiple Large Scale Science and Engineering
Projects
Applications
Visualization
Intel, IBM, Qwest Myricom, Sun, Oracle
Compute
Data
53Million from NSF
22
From SuperComputers to SuperNetworks--Changing
the Grid Design Point

• The TeraGrid is Optimized for Computing
• 1024 IA-64 Nodes Linux Cluster
• Assume 1 GigE per Node 1 Terabit/s I/O
• Grid Optical Connection 4x10Gig Lambdas 40
  Gigabit/s
• Optical Connections are Only 4 Bisection
  Bandwidth
• The OptIPuter is Optimized for Bandwidth
• 32 IA-64 Node Linux Cluster
• Assume 1 GigE per Processor 32 gigabit/s I/O
• Grid Optical Connection 4x10GigE 40 Gigabit/s
• Optical Connections are Over 100 Bisection
  Bandwidth

23
The OptIPuter is an Experimental Network
Research Project

• Driven by Large Neuroscience and Earth Science
  Data
• Multiple Lambdas Linking Clusters and Storage
• LambdaGrid Software Stack
• Integration with PC Clusters
• Interactive Collaborative Volume Visualization
• Lambda Peer to Peer Storage With Optimized
  Storewidth
• Enhance Security Mechanisms
• Rethink TCP/IP Protocols
• NSF Large Information Technology Research
  Proposal
• UCSD and UIC Lead CampusesLarry Smarr PI
• USC, UCI, SDSU, NW Partnering Campuses
• Industrial Partners IBM, Telcordia/SAIC, Chiaro
  Networks
• 13.5 Million Over Five Years

24
The OptIPuter is an Experimental Network
Research Project

• Driven by Large Neuroscience and Earth Science
  Data
• Multiple Lambdas Linking Clusters and Storage
• LambdaGrid Software Stack
• Integration with PC Clusters
• Interactive Collaborative Volume Visualization
• Lambda Peer to Peer Storage With Optimized
  Storewidth
• Enhance Security Mechanisms
• Rethink TCP/IP Protocols
• NSF Large Information Technology Research
  Proposal
• UCSD and UIC Lead CampusesLarry Smarr PI
• USC, UCI, SDSU, NW Partnering Campuses
• Industrial Partners IBM, Telcordia/SAIC, Chiaro
  Networks
• 13.5 Million Over Five Years

25
Metro Optically Linked Visualization Wallswith
Industrial Partners Set Stage for Federal Grant

• Driven by SensorNets Data
• Real Time Seismic
• Environmental Monitoring
• Distributed Collaboration
• Emergency Response
• Linked UCSD and SDSU
• Dedication March 4, 2002

Linking Control Rooms
UCSD
SDSU
Cox, Panoram, SAIC, SGI, IBM, TeraBurst
Networks SD Telecom Council
44 Miles of Cox Fiber
26
OptIPuter NSF Proposal Partnered with National
Experts and Infrastructure
SURFnet CERN
Asia Pacific
CAnet4
Vancouver
Seattle
CAnet4
Pacific Light Rail
Portland
Chicago
NYC
UIC NU
PSC
San Francisco
TeraGrid DTFnet
Asia Pacific
NCSA
CENIC
USC
UCI
Los Angeles
UCSD, SDSU
Atlanta
San Diego (SDSC)
AMPATH
Source Tom DeFanti and Maxine Brown, UIC
27
OptIPuter LambdaGridEnabled by Chiaro Networking
Router
Medical Imaging and Microscopy
Chemistry, Engineering, Arts

• Cluster Disk
• Disk Disk
• Viz Disk
• DB Cluster
• Cluster Cluster

Chiaro Enstara
San Diego Supercomputer Center
Scripps Institution of Oceanography
www.calit2.net/news/2002/11-18-chiaro.html
28
The UCSD OptIPuter Deployment
The UCSD OptIPuter LambdaGrid Testbed
To Other OptIPuter Sites
To Other OptIPuter Sites
Phase I, Fall 02
Phase I, Fall 02
Phase II, 2003
Phase II, 2003
Collocation point
Collocation point
SDSC
SDSC
SDSCAnnex
SDSCAnnex
Preuss
High School
JSOE
Engineering
CRCA
Arts
SOM
Medicine
6th College
UndergradCollege
Chemistry
Phys. Sci -Keck
Node M
Collocation
SIO
Earth Sciences
29
OptIPuter Transforms Individual Laboratory
Visualization, Computation, Analysis Facilities
Fast polygon and volume rendering with
stereographics


GeoWall

3D APPLICATIONS
Underground Earth Science
Earth Science
GeoFusion GeoMatrix Toolkit
Rob Mellors and Eric Frost, SDSUSDSC Volume
Explorer
The Preuss School UCSD OptIPuter Facility