OSG Integration

1
Welcome to CW 2008!!!
2
The Condor Project (Established 85)

• Distributed Computing research performed by a
  team of 35 faculty, full time staff and students
  who
• face software/middleware engineering challenges
  in a UNIX/Linux/Windows/OS X environment,
• involved in national and international
  collaborations,
• interact with users in academia and industry,
• maintain and support a distributed production
  environment (more than 4000 CPUs at UW),
• and educate and train students.

3

• Since the early days of mankind the primary
  motivation for the establishment of communities
  has been the idea that by being part of an
  organized group the capabilities of an individual
  are improved. The great progress in the area of
  inter-computer communication led to the
  development of means by which stand-alone
  processing sub-systems can be integrated into
  multi-computer communities.

Miron Livny, Study of Load Balancing Algorithms
for Decentralized Distributed Processing
Systems., Ph.D thesis, July 1983.
4
Main Threads of Activities

• Distributed Computing Research develop and
  evaluate new concepts, frameworks and
  technologies
• Keep Condor flight worthy and support our users
  
• The Open Science Grid (OSG) build and operate a
  national High Throughput Computing infrastructure
• The Grid Laboratory Of Wisconsin (GLOW) build,
  maintain and operate a distributed computing and
  storage infrastructure on the UW campus.
• The NSF Middleware Initiative - Develop, build
  and operate a national Build and Test facility
  powered by Metronome

5
Future ofGrid Computing
CHEP 07
6

• The Tulmod says in the name of Rabbi Yochanan,
• Since the destruction of the Temple, prophecy
  has been taken from prophets and given to fools
  and children.  
• (Baba Batra 12b) 

7
The Grid Computing Movement

• I believe that as a movement grid computing ran
  its course.
• No more an easy source of funding
• No more an easy way to get the troops mobilized
• No more an easy sell of software tools
• No more an easy way to get your papers published
  or your press releases posted

8
Introduction The term the Grid was coined in
the mid 1990s to denote a proposed distributed
computing infrastructure for advanced science and
engineering 27. Considerable progress has
since been made on the construction of such an
infrastructure (e.g., 10, 14, 36, 47) but the
term Grid has also been conflated, at least in
popular perception, to embrace everything from
advanced networking to artificial intelligence.
One might wonder if the term has any real
substance and meaning. Is there really a
distinct Grid problem and hence a need for new
Grid technologies? If so, what is the nature
of these technologies and what is their domain of
applicability? While numerous groups have
interest in Grid concepts and share, to a
significant extent, a common vision of Grid
architecture, we do not see consensus on the
answers to these questions. The Anatomy of the
Grid - Enabling Scalable Virtual Organizations
Ian Foster, Carl Kesselman and Steven Tuecke
2001.
9
Distributed Computing

• Distributed computing is here to stay and to
  continue to evolve as processing, storage and
  communication resources get more powerful and
  cheaper
• Big science is inherently distributed
• Most scientific disciplines (and many commercial
  sectors) depend on High Throughput Computing
  (HTC) capabilities

10

• Keynote 3 When All Computing Becomes Grid
  Computing
• Speaker Prof. Daniel A. Reed
• Chancellors Eminent Professor
• Director, Renaissance Computing Institute
• University of North Carolina at Chapel Hill
• Abstract
• Scientific computing is moving rapidly from a
  world of reliable, secure parallel systems to a
  world of distributed software, virtual
  organizations and high-performance, though
  unreliable parallel and distributed systems with
  few guarantees of availability and quality of
  service. In addition, a tsunami of new
  experimental and computational data poses equally
  vexing problems in analysis, transport,
  visualization and collaboration. This
  transformation poses daunting scaling and
  reliability challenges and necessitates new
  approaches to collaboration, software
  development, performance measurement, system
  reliability and coordination. This talk describes
  Renaissance approaches to solving some of todays
  most challenging scientific and societal problems
  using Grids and parallel systems, supported by
  rich tools for performance analysis, reliability
  assessment and workflow management.

11
As we return to the fundamentals and stay away
from hype and the technologies of the moment, we
will advance the state of the art in distributed
computing
12
Our HTCCommunity is Strongerthan Ever
13
Downloads per month
14
Fractions per month
15
Language Weaver Executive Summary

• Incorporated in 2002
• USC/ISI startup that commercializes
  statistical-based machine translation software
• Continuously improved language pair offering in
  terms of language pairs coverage and translation
  quality
• More than 50 language pairs
• Center of excellence in Statistical Machine
  Translation and Natural Language Processing

16
IT Needs

• The Language Weaver Machine Translation systems
  are trained automatically on large amounts of
  parallel data.
• Training/learning processes implement workflows
  with hundreds of steps, which use thousands of
  CPU hours and which generate hundreds of
  gigabytes of data
• Robust/fast workflows are essential for rapid
  experimentation cycles

17
Solution Condor

• Condor-specific workflows adequately manage
  thousands of atomic computational steps/day.
• Advantages
• Robustness good recovery from failures
• Well-balanced utilization of existing IT
  infrastructure

18
The Road Ahead

• Green Computing
• Computing in the Clouds
• Launch and Leave Computing
• Turn-on of the LHC
• Broader and larger community of contributors
• More and bigger campus grids
• Fetching work from other sources
• Multi-Core nodes
• Low latency and short jobs
• Staging data through Storage Elements

19
Thank you for building such
a wonderful community