2001 Summer Student Lectures Computing at CERN Lecture 2

1
2001 Summer Student LecturesComputing at
CERNLecture 2 Looking at DataTony Cass
Tony.Cass_at_cern.ch
2
Data and Computation for Physics Analysis
event filter (selection reconstruction)
detector
processed data
event summary data
raw data
batch physics analysis
event reconstruction
analysis objects (extracted by physics topic)
event simulation
interactive physics analysis
3
Central Data Recording

• CDR marks the boundary between the experiment and
  the central computing facilities.
• It is a loose boundary which depends on an
  experiments approach to data collection and
  analysis.
• CDR developments are also affected by
• network developments, and
• event complexity.

4
Monte Carlo Simulation

• From a physics standpoint, simulation is needed
  to study
• detector response
• signal vs. background
• sensitivity to physics parameter variations.
• From a computing standpoint, simulation
• is CPU intensive, but
• has low I/O requirements.
• Simulation farms are therefore good testbedsfor
  new technology
• CSF for Unix and now PCSF for PCs and Windows/NT.

5
Data Reconstruction

• The event reconstruction stage turns detector
  information into physics information about
  events. This involves
• complex processing
• i.e. lots of CPU capacity
• reading all raw data
• i.e lots of input, possibly readfrom tape
• writing processed events
• i.e. lots of output whichmust be written
  topermanent storage.

6
Batch Physics Analysis

• Physics analysis teams scan over all events to
  find those that are interesting to them.
• Potentially enormous input
• at least data from current year.
• CPU requirements are high.
• Output is small
• O(102)MB
• but there are many different teams andthe output
  must be stored for future studies
• large disk pools needed.

7
Symmetric MultiProcessor Model
Experiment
Tape Storage
TeraBytes of disks
8
Scalable modelSP2/CS2
Experiment
Tape Storage
TeraBytes of disks
9
Distributed Computing Model
CPU Server
Disk Server
Switch
10
Todays CORE Computing Systems
11
Todays CORE Computing Systems
SHIFTData intensive services
Simulation Facility
CORE Physics Services
CERN
CSF - RISC servers
PCSF - PCs NT
Central Data Services
Shared Tape Servers
200 computers, 550 processors (DEC, H-P, IBM,
SGI, SUN, PC) 25 TeraBytes embedded disk
25 H-P PA-RISC
10 PentiumPro 25 Pentium II
4 tape robots 90 tape drives Redwood, 9840
DLT, IBM 3590, 3490, 3480 EXABYTE, DAT, Sony D1
32 IBM, DEC, SUN servers
PC Farms
Data Recording, Event Filter and CPU
Farms for NA45, NA48, COMPASS
Shared Disk Servers
2 TeraByte disk 10 SGI, DEC, IBM servers
60 dual processor PCs
DXPLUS, HPPLUS,RSPLUS,LXPLUS, WGSInteractiveSer
vices
Home directories registry
PaRC EngineeringCluster
70 systems (HP, SUN, IBM, DEC, Linux)
13 DEC workstations 3 IBM workstations
CERN Network
12
Todays CORE Computing Systems
SHIFTData intensive services
Simulation Facility
CORE Physics Services
CERN
CSF - RISC servers
PCSF - PCs NT
Central Data Services
Shared Tape Servers
350 computers, 850 processors (DEC, H-P, IBM,
SGI, SUN, PC) 40 TeraBytes embedded disk
25 H-P PA-RISC
6 tape robots 90 tape drives Redwood, 9840
DLT, IBM 3590, 3490, 3480 EXABYTE, DAT, Sony D1
10 PentiumPro 25 Pentium II
25 PC servers4 others for HPSS
RSBATCH Public BatchService
PC Farms
Data Recording, Event Filter and CPU
Farms for NA45, NA48, COMPASS
Shared Disk Servers
1 TeraByte disk 3 Sun servers
32 PowerPC 604
250 dual processor PCs
DXPLUS, HPPLUS,RSPLUS,LXPLUS, WGSInteractiveSer
vices
Home directories registry
PaRC EngineeringCluster
80 systems (HP, SUN, IBM, DEC, Linux)
13 DEC workstations 5 dual processor PCs
CERN Network
13
Todays CORE Computing Systems
Queue shared Linux Batch Service
Timeshared Linuxcluster
Central Data Services
Shared Tape Servers
350 dual processor PCs
10 tape robots 100 tape drives 9940, Redwood,
9840, DLT, IBM 3590E, 3490, 3480 EXABYTE, DAT,
Sony D1
200 dual processor PCs
25 PC servers
PC EIDE baseddisk Servers
Shared Disk Servers
5 TeraByte disk 3 Sun servers 6 PC based servers
40TB mirrored disk (80TB raw capacity)
NAP - accelerator simulation
service
10-CPU DEC 8400 12 DEC workstations 20 dual
processor PCs
Dedicated RISCclusters
CERN Network
300 computers, 750 processors (DEC, HP, SGI, SUN)
14
Hardware Evolution at CERN, 1989-2001
15
Interactive Physics Analysis

• Interactive systems are needed to enable
  physicists to develop and test programs before
  running lengthy batch jobs.
• Physicists also
• visualise event data and histograms
• prepare papers, and
• send Email
• Most physicists use workstationseither private
  systems or central systems accessed via an
  Xterminal or PC.
• We need an environment that provides access to
  specialist physics facilities as well as to
  general interactive services.

16
Unix based Interactive Architecture
17
PC based Interactive Architecture
18
Event Displays
Standard X-Y view
Clever processing of events can also highlight
certain featuressuch as in the V-plot views of
ALEPH TPC data.
V-plot view
Event displays, such as this ALEPH display help
physicists to understand what is happening in a
detector. A Web based event display, WIRED, was
developed for DELPHI and is now used elsewhere.
19
Data Analysis Work
Most of the time, though, physicists will study
event distributions rather than individual events.
By selecting a dE/dx vs. p region on this scatter
plot, a physicist can choose tracks created by a
particular type of particle.
RICH detectors provide better particle
identification, however. This plot shows that the
LHCb RICH detectors can distinguish pions from
kaons efficiently over a wide momentum
range. Using RICH information greatly improves
the signal/noise ratio in invariant mass plots.
20
CERNs Network Connections
SWITCH
National Research Networks
RENATER
1Gb/s
Mission Oriented Link
2Mb/s
IN2P3
155Mb/s
45Mb/s
TEN-155 Trans-European Network at 155Mb/s
WHO
39/155 Mb/s
TEN-155
CERN
Public
155Mb/s
KPNQwest (US)
1Gb/s
Commercial
C-IXP
21
CERNs Network Traffic5th March 1998 - 5th April
1998
Incoming data rate
2.5Mb/s
1.7Mb/s
1.1Mb/s
7Mb/s
0.4Mb/s
4Mb/s
2Mb/s
0.7Mb/s
0.7Mb/s
Outgoing data rate
Link Bandwidth
0.1Mb/s
0.1Mb/s
20Mb/s
2Mb/s
1 TB/month in each direction 1TB/month
3.86Mb/s 1Mb/s 10GB/day
22
CERNs Network TrafficMay - June 1999
Incoming data rate
2.5Mb/s
1.7Mb/s
0.6Mb/s
40Mb/s
1Mb/s
6Mb/s
20Mb/s
1.9Mb/s
1.8Mb/s
Outgoing data rate
Link Bandwidth
0.1Mb/s
0.1Mb/s
100Mb/s
2Mb/s
1 TB/month in each direction 1TB/month
3.86Mb/s 1Mb/s 10GB/day
23
CERNs Network TrafficJune/July 2000
Incoming data rate
6.2Mb/s
5.2Mb/s
3.6Mb/s
40Mb/s
9Mb/s
KPNQwest(US)
34Mb/s
45Mb/s
6.3Mb/s
9Mb/s
Outgoing data rate
Link Bandwidth
0.1Mb/s
0.1Mb/s
100Mb/s
2Mb/s
4.6-5.6 TB/month in/out 1TB/month
3.86Mb/s 1Mb/s 10GB/day
24
CERNs Network TrafficJune/July 2001
Incoming data rate
4.7Mb/s
5.5Mb/s
5.2Mb/s
40Mb/s
14Mb/s
KPNQwest(US)
45Mb/s
155Mb/s
20Mb/s
25Mb/s
Outgoing data rate
Link Bandwidth
0.1Mb/s
0.1Mb/s
100Mb/s
2Mb/s
10 TB/month in/out 1TB/month 3.86Mb/s 1Mb/s
10GB/day
25
Outgoing Traffic by ProtocolMay 31st-June 6th
1999
350
Note1999 data
300
250
200
Elsewhere
USA
GigaBytes Transferred
Europe
150
100
50
0
ftp
www
X
afs
int
rfio
mail
news
other
Total
Protocol
26
Incoming Traffic by Protocol May 31st-June 6th
1999
350
Note1999 data
300
250
200
Elsewhere
USA
GigaBytes Transferred
Europe
150
100
50
0
ftp
www
X
afs
int
rfio
mail
news
other
Total
Protocol
27
European US Traffic GrowthFeb 97-Jun 98
USA
Start of TEN-34 connection
EU
28
European US Traffic GrowthFeb 98-Jun 99
USA
EU
29
European US Traffic GrowthMar 99-Jul 00
USA
EU
30
European US Traffic GrowthMar 99-Jul 00
2000 Data. Not worth updating for 2001. Networks
are no longer a bottleneck.
USA
EU
31
Traffic GrowthJun 98 - May/Jun 99
Note1999 data
Total
EU
Other
US
32
Traffic Comparison 1998?1999? 2000

• Two significant changes this year
• BaBar data flow from SLAC to IN2P3 via CERN, and
• increased use of ssh
• prevent a quantitative comparison of 2000
  traffic patterns with those of previous years.
• Qualitatively, however, physics data transfer is
  taking an increasing fraction of the overall
  traffic.
• BaBar traffic is one good example.
• CMS have also relied on the network to transfer
  Monte Carlo data during their tests of their High
  Level Trigger algorithm.

33
Round Trip times and Packet Loss rates
Round trip times for packets to SLAC
1998 Figures.
This is measured with ping A packet must arrive
and be echoed back if it is lost, it does not
give a Round Trip Time value.
5 seconds!
Packet Loss rates to/from the US on the CERN link
But traffic to, e.g., SLAC passes over other
links in the US and these may also lose packets.
34
Looking at DataSummary

• Physics experiments generate data!
• and physcists need to simulate real data to model
  physics processes and to understand their
  detectors.
• Physics data must be processed, stored and
  manipulated.
• Central computing facilities for physicists
  must be designed to take into account the needs
  of the data processing stages
• from generation through reconstruction to
  analysis
• Physicists also need to
• communicate with outside laboratories and
  institutes, and to
• have access to general interactive services.