DIANE

1
DIANE Distributed Analysis Environment for
semi-interactive simulation and analysis in
Physics Jakub T. Moscicki, CERN/IT
2
The need for distribution

• do the analysis/simulation job in parallel tasks
• to speed up the work
• by using powerful, worldwide distributed
  computentional resources,
• acessing the data in mass storage systems
  otherwise too big to fit on your laptop.

3
Practical Example

• example simulation with analysis
• each task produces a file with histograms
• job result sum of histograms produced by tasks
• master-worker model
• client starts a job
• workers perform tasks and produce histograms
• master integrates the results

4
(No Transcript)
5
(No Transcript)
6
Comments

• using middleware directly requires a lot of
  manual work
• integration of task results
• keeping track of failed task and resubmiting
  workers
• not easy to monitor the job progress and cancel
  jobs
• only one task per worker
• very inefficient if worker initialization time is
  long

7
User Wishlist

• automatic integration of task results
• monitoring of job progress and individual tasks
• automatic error-recovery policies
• granularity of the size of the task may change
  independently of the number of workers -- natural
  load-balancing and optimization of performance
• performance fine tuning workers may be mapped
  to threads, processed or machines depending on
  the context
• uniform, transparent and easy user interface and
  API hiding complexity of underlying middleware
  mechanisms
• the same API and UI is used when running local
  jobs and GRID jobs
• batch, interactive and semi-interactive operation
  mode

8
Wishlist (cntd)

• a lightweight add-on framework which drives the
  execution of parallel jobs in master worker model
  over any specific middleware implementation
• application oriented target common HEP use cases
• independent from any particular analysis tool
• with layered and modular architecture which is
  easy to adapt to new environment important for
  middleware transition
• integrated in modern scripting environment e.g.
  python
• using standards e.g. exploit AIDA for analysis
  making it easy to plug your favourite analysis
  tool
• To address these issues DIANE Project was set up
  in CERN/IT

9
DIANE Overview

• DIANE RD Project
• started in 2001 in CERN/IT with very limited
  resources (1FTE)
• collaboration with Geant 4 groups at CERN, INFN,
  ESA
• succesful prototypes running on LSF and EDG

10
(No Transcript)
11
DIANE for HEP workgroup clusters

• features
• many users, many jobs
• diverse applications
• ntuple analysis, simulation, ...
• interactive ... semi-interactive ... batch
• 100s of machines
• dynamic environment
• users may submit their analysis code
• mixed CPU and I/O intensive
• some applications may be preconfigured
• general analysis e.g. ntuple projections or
  experiment specific apps
• load balancing important

12
DIANE for Simulation in Medical Apps

• example brachytherapy
• optimization of the treatment planning by MC
  simulation
• features
• CPU intensive
• few users, few jobs
• one preconfigured application
• interactive seconds .. minutes
• 10s of machines
• ongoing joint collaboration with G4and hospital
  units in Torino, Italy

13
(No Transcript)
14
DIANE Prototype and Testing

• scalability tests
• 70 worker nodes
• 140 milion Geant 4 events

15
(No Transcript)
16
DIANE Web Interface
17
References

• more informarion
• cern.ch/diane
• www.ge.infn.it/geant4/techtransf
• aida.freehep.org

18
The end