MAP Project

1
MAP Project

• T. Bowcock, A. Kinvig, I. Last
• M. McCubbin, A. Moreton
• C. Parkes, G. Patel
• University of Liverpool

2
Introduction

• Monte Carlo Array Processor
• justification
• Status
• Hardware
• Software
• COMPASS
• Summary

3
Monte Carlo

• At LHCb about 1 interaction /25ns !
• 41014/year
• if you want to do physics you need to know the
  backgrounds
• generating just the signals doesnt work
• need to generate large MC samples
• O(107) to O(108) events.
• LHCb needs to do this now!

4
Philosophy

• Cheapest possible that works
• No Gbit ethernet until price falls
• Dont buy top of range processors
• No SMP boards
• No tapes
• obsolete?
• Develop architecture with future in mind

5
MAP hardware

• 300 processors
• 400MHz PII
• 128 Mbytes memory
• 3 Gbytes disk
• D-Link 100BaseT ethernet hubs
• commercial units BUT
• custom boxes for packing and cooling

6
MAP
7
MAP contd
8
MAP contd
9
MAP Architecture
MAP Slaves
Master
External Ethernet
Hub
100BaseT
Hub
10
MAP software

• Overview
• Linux
• based on RedHat 5.2
• stripped down version
• Batch System
• Network
• Control At the UDP level
• Robust Packet Handling

Overloading of master ethernet interfaces (300 at
once) implied need for total control of data flow
Broadcast of control required phased reply
11
MAP user

• Prepare a job
• Submit to Batch Queue
• Histograms/Ntuples transmitted back at end of
  job/DSTs
• Random Numbers handled automatically

12
MAP Status

• In production for about 6 weeks
• 300 Processors
• produced about 240,000 LHCb events 24/hrs
• 5 million events produced to date
• Also produced DELPHI DSTs (500,000 24Hrs)
• All Processors tested
• Further Air-Conditioning installed
• fully commissioned 22/11/99

13
MAP Issues

• Packet Loss
• At UDP (or frame level) have to handle with code.
  Now not a probem(!)
• Higher performance with shielded cables?
• no
• Power
• Infrastructure for cooling
• Power up/down

14
Emergency Power Down

• Unplanned power interruption
• Exploding substation!
• About 4 of PCs need manual intervention

15
MAP capabilites

• Can be used in throwaway mode
• Also write events as genenerated
• MAP possesses 1Tbyte internal storage
• 3 Gbytes/machine
• events stored locally (1million events)
• repeatedly analyse QUICKLY
• MAP can handle interprocess communication

16
MAP
17
COMPASS

• Computerized Analysis and Storage Server

18
COMPASS

• Purpose
• Will show this in place and working with MAP
• Model for LHC analysis
• store events on disks (cheap!)
• move JOB to the DATA
• NO HSM

19
Outline

• Hardware
• Linux Device Drivers
• Linux Installation and Limits
• Benchmarking Tests
• Results
• Future

20
Trial Hardware

• Dell PowerEdge Server, 450 Mhz Pentium III, 256
  Mb RAM with 4 internal SCSI disks.
• 4 PowerVault 1200 Disk Servers each with 8 Ultra
  Wide SCSI LVD disks.(spindle 7200 rpm)Total gt
  1Tb disk space
• Adaptec Ultra Wide SCSI cards.

21
ITS

• Purchased Rack mounted
• 1TByte based on 50GByte 7200 rpm disks
• Redundant Power Supplies
• 15KGBP/Tbyte including 2 500MHz PIII
• More storage underway

22
Linux Devices Drivers

• Linux Device Drivers
• Devices accessed through special files in /dev
  directory specifying block or character device
  and major / minor number pairs.
• Major number refers to a device driver e.g. 8
  is a SCSI disk (see /usr/src/linux/include/linux/m
  ajor.h)
• For disks, minor number refers to disk /
  partition on disk e.g. /dev/sda major8
  minor0 first SCSI disk found on system
  /dev/sda1 major8 minor1 first partition
  /dev/sda15 major8 minor15 last partition
  on first disk /dev/sdb major8
  minor16 second SCSI disk found on system
• minor numbers are 8-bit i.e. only have values in
  range 0-255? only 16 disks per disk major number.

23
Linux Installation Limits

• RedHat Linux 5.2 Kernel 2.0.x
• Used at Liverpool and CERN problem only one
  SCSI major number is defined maximum of 16 SCSI
  disks allowed.
• Kernel hacking necessary to register new SCSI
  major number with system.
• RedHat Linux 6.0 Kernel 2.2.x
• Defines 8 SCSI major numbers 8, 65-71- max. 128
  SCSI disks.
• Have to create some special files in /dev by hand
  relatively trivial with mknod
• Physical limit of only 4 PCI slots for SCSI cards
  on motherboard

24
Benchmarking Tests

• Use CERN sequential IO tests for read / write /
  calibration.
• Block sizes from 1024 Bytes to 0.5MBytes
• Calculates average write rate over previous 10
  writes
• Read ...
• Calibration Comment out write statement and run
  write tests again.
• Modified version of above calculates averages
  over the whole file.

25
Results

• All disks accessible
• Performance uniform
• writing about 20MBytess
• reading at 50 MBytes (or better)
• large block-sizes faster

26
Future

• Can we find funding for large(r) scale prototype?
• Applications outside of Physics
• Interdisciplinary funding

27
Summary

• MAP yields high performance at low cost
• Storage can be cheap
• RD to Enhance performance
• Production for LHCb vertex detector