CS780-3 Lecture Notes

1

• CS780-3 Lecture Notes
• In Courtesy of Mengjun Xie

2
About

• NEESgrid will link earthquake researchers across
  the U.S. with leading-edge computing resources
  and research equipment, allowing collaborative
  teams (including remote participants) to plan,
  perform, and publish their experiments.
• NEESgrid, the systems integration component of
  the NEES project, uses the newest and fastest
  communications technologies to tie the NEES
  network together.

3
NEES Resources
Remote Users
Instrumented Structures and Sites
(Faculty, Students, Practitioners)
Simulation Tools Repository
Laboratory Equipment
Field Equipment
Curated Data Repository
Leading Edge Computation
Global Connections (FY 2005 FY 2014)
Remote Users (K-12 Faculty and Students)
Laboratory Equipment
4
NEES

• Network for Earthquake Engineering Simulation
• NEES is a distributed array of experimental
  sites, grid-based data repositories, tool
  archives, and computational resources, all
  seamlessly linked (hopefully!)
• NEES has four components
• The consortium, which will run NEES after 2004
• The consortium development (CD) builds the
  consortium
• The experimental sites, which provide data and
  content
• The systems integration (SI) effort, termed
  NEESgrid
• Network drivers include telepresence, curated
  repositories, scalable HPC, experimental-numerical
  coupling, short- and long-term QoS issues.

5
NEES Network Stakeholders

• Experimental Facilities
• Shake tables, centrifuges, wave tanks, field
  sites
• Resource providers
• Computers, software, storage, networks
• End users
• Researchers, practicing engineers, students,
• Operational facilities
• NCSA/NEESgrid ? NEES Consortium in 2004

6
NEESgrid?

• A coordinated and secure architecture/environment
  
• A modular and extensible environment with a
  customizable user interface
• Provides common tools that allow leveraging
  resources and experiences
• Goal the Cyber Infrastructure that will
  facilitate this next generation of
  experimentation/simulation in earthquake
  engineering

A Distributed Virtual Laboratory for Advanced
Earthquake Experimentation and Simulation.
-- B. F. Spencer, Jr.
(PI Project Director)
7
NEESgrid?

• Through the NEESgrid, researchers can
• perform tele-observation and tele-operation of
  experiments
• publish to and make use of a curated data
  repository using standardized markup
• access computational resources and open-source
  analytical tools
• access collaborative tools for experiment
  planning, execution, analysis, and publication.

8
Telepresence

• Telepresence means the capability to participate
  remotely in experimental trials. There are two
  main categories
• Tele-observation the ability to observe the
  experiment and capture trial data from a remote
  site
• Tele-operation the ability to interact with the
  experiment equipment from a remote site.

9
Collaboration and Visualization

• The tasks of Collaboration and Visualization
  project include
• prototype a Grid-based collaborative environment
  
• integrate support for visualization tools into
  the collaborative environment
• adapt the Comprehensive Collaborative Framework
  (CHEF) for collaborative visualization
  applications and services and
• produce appropriate final documentation of the
  collaboration and visualization components of
  NEESgrid.
• CHEF is a flexible web-based environment for
  remote collaboration.

10
Main Components

• Tele-Control Services and APIs
• Tele-Observation and Data Visualization
• E-Notebook
• Streaming data services
• DAQ and related services
• Data and Metadata Services
• Remote Collaboration and Visualization tools and
  services
• Core Grid Services, deployment efforts, packaging
  
• Simulation Component

11
Progress

• The task of designing and creating this
  infrastructure has been awarded to the NCSA at
  UIUC.
• The components of the NEESgrid system will be
  completed by September, 2004, when management and
  operation of the NEES system will be turned over
  to a consortium of earthquake engineer
  researchers and practitioners.

12
Partner
Randy Butler Deployment, Operations,
and SupportMark Marikos
ManagementJoe Futrelle Data and Metadata
National Center for Supercomputing Applications
University of Illinois at Urbana-Champaign
Bill Spencer ManagementDan Abrams Community
Building
Argonne National Laboratory
Nestor J. Zaluzec TelepresenceIan Foster
System Configuration
Information Sciences Institute
Carl Kesselman System Configuration
University of Southern California
Jean-Pierre Bardet Integrated
Demonstrations
School of Information, University of Michigan
Joseph Hardin CollaborationTom Finholt User
Requirements
13
System Architecture
14
System Architecture
15
System Architecture
16
System Architecture
17
System Architecture
18
System Architecture
19
System Architecture
20
Pre-Experiment Collaboration

• The remote user (RU) logs into the equipment
  site's NEESpop. After authentication, RU can
  access all granted NEESgrid resources through
  SSN.
• RU can, via CHEF, read and participate in
  discussions related to the equipment site, see a
  calendar, share documents, make announcements,
  and correspond with other researchers to plan an
  experiment.
• Through NEESpop, RU can also access and edit
  electronic lab notebooks hosted on the TPM.

21
Monitoring Network Status

• RU contacts CHEF and requests a summary of the
  system's health and status.
• The CHEF server contacts a central NCSA server
  that monitors NEESgrid.
• Standard grid services (GIIS at NCSA, GRIS for
  other services) are continually collecting
  information on system connectivity and the
  availability of grid services such as GridFTP.
• The grid information is summarized and sent to
  RU's browser.

22
Setting Up the Experiment

• Using a CHEF teamlet, the equipment specialist at
  the equipment site enters the metadata describing
  the configuration of the experimental setup.
• The metadata repository on the NEESpop holds this
  configuration information, making it available
  for browsing by RUs.
• The NEESpop metadata repository also sends
  configuration information to the DAQ system(s).

23
Subscribing to Trial Data Streams

• RU contacts CHEF to browse available NEESgrid
  resources and discover what data will be
  available for streaming.
• A CHEF teamlet uses GIIS and GRIS to query and
  report features and availability of NEESgrid
  resources, as well as details of NEES equipment
  and available data streams.
• RU contacts the NEESpop and requests a
  subscription to several data streams.
• The NEESpop negotiates with the NSDS to broadcast
  the selected data streams.

24
Streaming Near-Real-Time Data

• Instrument information is sent in real time from
  the DAQ system through a NSDS daemon to the NSDS.
• Data is streamed by NSDS to the RU. An applet in
  the RU's browser displays the results.
• Video services are managed through the TPM server
  and streamed to the RU via a separate video
  streaming server.

25
Ending the Trial

• The DAQ equipment moves the trial data to a data
  repository hosted on the NEESpop.
• The data is associated with appropriate metadata
  information, such as experiment ID, channel ID,
  and sensor type.

26
Analyzing the Trial Data

• The trial data can now be browsed via CHEF.
• The trial data and metadata can be downloaded to
  the RU via GridFTP service.
• The RU can then use either standard software,
  such as MatLAB, or their own customized software
  to analyze the trial data.

27
Login Authentication
28
Chat
CHEF Environment
29
E-Notebook
CHEF Environment
30
Data Viewer
CHEF Environment
31
NTCP Data
CHEF Environment
32
NEESgrid Central Repository
CHEF Environment
33
Summary

• Characterizations of NEES project
• A very specific application
• A wide range of resource sharing
• Raw data, experimental results, equipment,
  computation/software tools
• Centralized controls, monitoring, and management.
  
• Does this follow the three principles of Grid?
• Not really it is subject to central control, not
  open source software based.
• But it is a successful model of grid
  applications.
• A Global and general grid is an ideal model, and
  may become realistic after many successful NEES
  projects.