Oxford University Particle Physics Unix Overview

1
Oxford University Particle Physics Unix Overview

• Pete Gronbech
• Senior Systems Manager and
• SouthGrid Technical Co-ordinator

2
Particle Physics Strategy The Server / Desktop
Divide
Servers
General Purpose Unix Server
Linux Worker nodes
Group DAQ Systems
Linux FileServers
Web Server
Win XP PC
Win XP PC
Win XP PC
Linux System
Win XP PC
Desktops
Approx 200 Windows XP Desktop PCs with Exceed,
xming or ssh used to access central Linux systems
3
Particle Physics Linux

• Aim to provide general purpose Linux based system
  for code development and testing and other Linux
  based applications.
• Unix Admin (Ewan MacMahon).
• Interactive logins servers and batch queues are
  provided
• Systems run Scientific Linux which is a free Red
  Hat Enterprise based distribution
• Currently in a major reorganisation as we migrate
  from SL3 to SL4
• All new students should only use SL4
• Worker nodes form a PBS cluster accessed via
  batch queues. These are being migrated from SL3
  to SL4

4
Current Clusters

• Particle Physics Local Batch cluster
• CDF (Matrix) cluster
• Oxfords Tier 2 Grid cluster

5
Scientific Linux 3 - 40 slots
PP Linux Batch Farm (Oct 06)
2 2.8GHz P4
pplxwn17
Migration to Scientific Linux 3 completed Sept
2005 Migration to SL 4 planned for Winter 2006
2 2.0GHz P4
pplxwn16
2 2.0GHz P4
pplxwn15
2 2.0GHz P4
pplxwn14
2 2.0GHz P4
pplxwn13
2 2.0GHz P4
pplxwn12
2 2.8GHz P4
pplxwn11
2 2.8GHz P4
pplxwn10
6TB SATA RAID Array
2 2.4GHz P4
pplxwn08
6TB
2 2.4GHz P4
pplxwn07
2 2.4GHz P4
pplxwn06
4TB SATA RAID Array
pplxwn04
2 2.4GHz P4
4TB
2 2.4GHz P4
pplxwn03
pplxwn02
2 2.4GHz P4
1.1TB
1TB Internal SCSI disks
ppslgen
4 AMD 285 2.6GHz
2 2.2GHz P4
pplxgen
pplxfs1
pplxwn05
8 700MHz P3
pplx2
2 1GHz P3
2 1GHz P3 File Server
2 800MHz P3
pplx3
6
Scientific Linux 4 12 slots
PP Linux Batch Farm (Oct 07)
Scientific Linux 3 10 slots
2 2.8GHz P4
pplxwn17
2 2.0GHz P4
pplxwn16
2 2.0GHz P4
Migration to SL 4 planned for Winter 2006,
delayed as CERN Clusters and grid software not
yet ported. Currently middle of the migration.
pplxwn15
2 2.0GHz P4
pplxwn14
2 2.0GHz P4
pplxwn13
2 2.0GHz P4
pplxwn12
2 2.8GHz P4
pplxwn11
2 2.8GHz P4
pplxwn10
2 2.4GHz P4
6TB SATA RAID Array
pplxwn08
6TB
2 2.4GHz P4
pplxwn07
4TB SATA RAID Array
2 2.4GHz P4
pplxwn06
4TB
2 2.4GHz P4
pplxwn04
2 2.4GHz P4
pplxgen
10TB
10TB SATA RAID Array
Intel 5130
pplxconfig
ppslgen
AMD 285 2.6GHz
pplxfs2
SL4
2 2.2GHz P4
pplxusers
Dual core 2GHz Intel Woodcresst 5130 File Server
pplxgenng
AMD 285 2.6GHz
7
PP Linux Batch Farm (Nov 07)
Scientific Linux 4 46 slots
pplxwn17
PP Local Batch farm upgrade
pplxwn16
6TB SATA RAID Array
pplxwn21
6TB
pplxwn20
pplxwn11
pplxwn19
4TB SATA RAID Array
pplxwn10
4TB
pplxwn18
pplxwn08
4 new servers each with dual Quad Core Intel
5345 2.33GHz, with 2GB RAM /core, gives 32 cpu
cores,each 2.5 faster than the old2.8 GHz cpus
10TB
10TB SATA RAID Array
pplxwn06
pplxwn04
pplxfs2
SL4
pplxconfig
9TB
9TB SATA RAID Array
pplxusers
pplxfs3
SL4
pplxgenng
8
Particle Physics General Purpose Batch Farm
PP batch farm currently in room 650, will be
migrated to the new computer room on Level 1.
Where the new systems will be installed.
Eonstor SATA RAID arrays of 16 250GB disks gives
approx 4TB , and another in Sept 05 with 16400GB
disks about 6TB. The third array gives 10TB
usable space.
9
CDF Linux Systems
Morpheus is an IBM x370 8 way SMP 700MHz
Xeon with 8GB RAM and 1TB Fibre Channel
disks Installed August 2001 Purchased as part of
a JIF grant for the CDF group Runs Scientific
Linux 3.0.5 Uses CDF software developed
at Fermilab and Oxford to process data from the
CDF experiment.
CDF kit is now 6 years old, and reaching end of
life.
10
Second round of CDF JIF tender Dell Cluster -
MATRIX
11 Dual 2.4GHz P4 Xeon servers running Scientific
Linux 3.0.5 Installed December 2002
Approx 9 TB for SCSI RAID 5 disks are attached to
the master node. Each shelf holds 14 146GB
disks.These are shared via NFS with the worker
nodes. OpenPBS batch queuing software is
used. This cluster will be merged into the PP
local batch cluster.
11
Southgrid Member Institutions

• Oxford
• RAL PPD
• Cambridge
• Birmingham
• Bristol
• Warwick
• JET at Culham

12
Oxford Tier 2 centre for LHC
Two racks each containing 20 Dell dual 2.8GHz
Xeons with SCSI system disks. Systems are loaded
with LCG software known as glite version
3.0.2. 74 WN cpuss 1.6 TB Storage (DPM)
second 1.6TB is a DPM pool The systems have been
heavily used by the LHCb, Atlas and Biomed Data
Challenges.
13
Oxford Tier 2 GridPP Cluster Summer 2005
LHCb
ATLAS
Biomed Data Challenge, Supported a non LHC EGEE
VO For about 4 weeks.
Start of August 2005
14
Oxford Tier 2 Grid Upgrade 2007

• 11 systems, 22 servers, 44 cpus, 176 cores. Intel
  5345 clovertown cpus provide 350KSI2K
• 11 servers each providing 9TB usable storage
  after RAID 6, total 99TB
• Two racks, 4 Redundant Management Nodes, 4 PDUs,
  4 UPSs

15
Two New Computer Rooms will provide excellent
infrastructure for the future
The New Computer room being built at Begbroke
Science Park jointly for the Oxford Super
Computer and the Physics department, will provide
space for 55 (11KW) computer racks. 22 of which
will be for Physics. Up to a third of these can
be used for the Tier 2 centre. This 1.5M project
is funded by SRIF and a contribution of 200K
from Oxford Physics. All new Physics HPC clusters
including the Grid will be housed here when it is
ready in October / November 2007.
16
Local Oxford DWB Physics Infrastructure Computer
Room
Completely separate from the Begroke Science park
a computer room with 100KW cooling and gt200KW
power is being built. 150K Oxford Physics
Money. Local Physics department Infrastructure
computer room. Completed last month (Sept 2007).
This will relieve local computer rooms and
house T2 equipment until the Begbroke room is
ready. Racks that are currently in unsuitable
locations can be re housed.
17
http//www-pnp.physics.ox.ac.uk/ganglia
18
The end for now

• Ewan will give more details of use of the
  clusters next week
• Questions.

19
Network

• Gigabit connection to campus operational since
  July 2005.
• Second gigabit connection installed Sept 2007.
• Gigabit firewall installed. Purchased commercial
  unit to minimise manpower required for
  development and maintenance. Juniper ISG 1000
  running netscreen.
• Firewall also supports NAT and VPN services which
  is allowing us to consolidate and simplify the
  network services.
• Moving to the firewall NAT has solved a number of
  problems we were having previously, including
  unreliability of videoconferencing connections.
• Physics-wide wireless network. Installed in DWB
  public rooms, Martin Wood,AOPP and Theory. New
  firewall provides routing and security for this
  network.

20
Network Access
Super Janet 4
2 1Gb/s with Super Janet 5
Physics Firewall
Physics Backbone Router
1Gb/s
OUCS Firewall
1Gb/s
10Gb/s
Backbone Edge Router
10Gb/s
100Mb/s
Campus Backbone Router
1Gb/s
10Gb/s
depts
Backbone Edge Router
depts
100Mb/s
depts
100Mb/s
depts
21
Physics Backbone
Linux Server
1Gb/s
Physics Firewall
Server switch
1Gb/s
Win 2k Server
1Gb/s
1Gb/s
Particle Physics
1Gb/s
100Mb/s
Physics Backbone Router
100Mb/s
1Gb/s
desktop
Clarendon Lab
100Mb/s
1Gb/s
desktop
1Gb/s
1Gb/s
100Mb/s
Astro
Atmos
Theory