AppLeS, NWS and the IPG

1
AppLeS, NWS and the IPG

• This presentation will probably involve audience
  discussion, which will create action items. Use
  PowerPoint to keep track of these action items
  during your presentation
• In Slide Show, click on the right mouse button
• Select Meeting Minder
• Select the Action Items tab
• Type in action items as they come up
• Click OK to dismiss this box
• This will automatically create an Action Item
  slide at the end of your presentation with your
  points entered.

• Fran Berman
• UCSD and NPACI
• Rich Wolski
• UCSD, U. Tenn. and NPACI

2
AppLeS and the IPG
Usability, Integration development of basic IPG
infrastructure Development of persistent IPG
testbed
Experience with Pilot IPG Development of
prototype performance-oriented
applications Development of necessary
research
Performance IPG - aware programming

Short-term
Medium-term
Long-term
Integration of schedulers and other tools,
performance interfaces
Application scheduling Resource
scheduling Throughput scheduling
Multi-scheduling Resource economy
3
A Model for the Future

• Adaptation is key to the ultimate IPG program
  development and execution environment.
• Exchange of performance information fundamental
  to the success of IPG applications

Grid Application Development System (GrADS)
4
Why Application Schedulers?

• Application performance can conflict with
  performance goals of other system components
• Goal of application scheduler is to prioritize
  performance of the application over other system
  components

5
Agent-based Application Scheduling
NWS (Wolski)
AppLeS (Berman and Wolski)
6
Performance Prediction

• Given monitored bandwidth data, what will happen
  next?

7
NWS Predictions

• Monitored data provides a snapshot of what has
  happened.
• What we really want to know is What will happen?

8
Monitoring vs. Prediction

• Last value not always the best predictor
• Hard to develop accurate forecasting models --
  why not use all feasible models?

Monitored data

9
Do AppLeS and NWS Improve Application Performance?

• Good results with many applications including
• SARA AppLeS
• CompLib AppLeS
• Jacobi2D AppLeS
• AppLeS/NWS applications demonstrate that
• prediction is possible in high-variance
  environments
• adaptivity can improve performance

10
SARA AppLeS

• SARA Synthetic Apperture Radar Atlas
• application developed at JPL and SDSC
• Goal Process radar images from distributed
  database for users desired image
• AppLeS focuses on resource selection problem

11
SARA Experiments
Data Servers
Compute Servers
Client
. . .
12
CompLib AppLeS

• Problem Find the best matches between two gene
  sequence libraries
• Apply FASTA algorithm to all sequence pairs to
  determine similarity
• Developed for DOCT testbed

13
CompLib Experiments
14
Jacobi2D AppLeS

• Important component of many scientific
  applications
• Time-balancing used to achieve minimal execution
  time
• Scheduler solves time-balancing equations for
  Area

15
Jacobi2D Experiments

• Comparison of AppLeS with and without NWS info,
  and load-balancing

16
Applying AppLeS/NWS Methodology to the IPG

• AppLeS/NWS methodology can be used to develop
  performance-efficient IPG applications
• IPG FY99 projects leverage FY98 project and
  previous AppLeS/NWS development and research

17
IPG FY99 Project A Parameter Sweep Template

• INS2D representative of larger class of critical
  NASA applications
• AppLeS parameter sweep template will build on
  INS2D model and experiments to target larger
  class of applications and platforms
• Template will serve as a prototype IPG PSE
  workbench tool

18
AppLeS Project Plan FY99 (Berman,UCSD)

• Expand INS2D AppLeS
• to NASA IPG testbed
• to include batch systems
• to target Globus
• Development of Parameter Sweep AppLeS template
• Goal To provide framework for improving
  turnaround time of parameter study component of
  complex AES applications
• AppLeS scheduling agents prototype autonomous
  agent technology for IPG
• Requires development of strategy for scheduling
  in mixed batch and interactive environments
• Project Personnel Berman, Casanova (UCSD)
• Collaborators Wolski (U. Tenn.), Kesselman
  (ISI/USC)

19
NWS Project Plan FY99 (Wolski, U. Tenn.)

• Enhance the NWS to support AppLeS parameter sweep
  template in NASA Globus environment
• NWS API for parameter sweep template
• integration with Globus
• Integrate NWS with IPG and Globus application
  performance monitoring tools
• use NWS performance techniques to predict
  application performance dynamically
• Investigate strategies for monitoring and
  forecasting batch system performance
• queue wait times in the presence of user
  priorities, etc.
• Project Personnel Wolski (U. Tenn)
• Collaborators Berman (UCSD), Moore (SDSC),
  Kesselman (ISI/USC)

20
Possible Additional IPG Projects

• AppLeS/NWS-enhanced Storage Resource Broker
• Project Enhance SRB performance through
  agent-based scheduling
• Project Personnel Berman, Wolski
• Collaborator Moore
• AppLeS/NWS-enhanced NetSolve over Globus
• Project Improve scheduling component of NetSolve
  using AppLeS/NWS techniques, deploy on Globus IPG
  platform
• Project Personnel Berman, Wolski, Casanova,
  Dongarra
• Collaborator Kesselman

21
Possible Additional IPG Projects

• AppLeS/NWS Applications on Condor
• Project Develop AppLeS application which can
  achieve performance in the Condor environment
  integrate Condor and NWS information leverage
  Condor/Globus integration
• Project Personnel Berman, Wolski
• Collaborator Livny, Kesselman

22
Project Information

• AppLeS Home Page http//www-cse.ucsd.edu/groups/
  hpcl/apples.html
• NWS Home Page
• http//nws.npaci.edu

23
Project Information

• NWS Home Page
• http//nws.npaci.edu
• AppLeS NWS Project Personnel
• Francine Berman
• Rich Wolski
• Walfredo Cirne
• Marcio Faerman
• Jaime Frey
• Jim Hayes
• Graziano Obertelli

• AppLeS Home Page http//www-cse.ucsd.edu/groups/
  hpcl/apples.html
• Jenny Schopf
• Gary Shao
• Neil Spring
• Shava Smallen
• Alan Su
• Dmitrii Zagorodnov