LUSI Controls and Data Systems W'B'S' 1'6

1
LUSIControls and Data SystemsW.B.S. 1.6

• Gunther Haller
• Project Manager
• July 15, 2009
• Breakout Presentation

2
Content

• Overview
• Controls System
• Data System
• CXI and XPP Detector Control/Data Chain
• Common Diagnostics

3
Controls W.B.S. Scope
Near Experimental Hall
X-ray Transport
Far Experimental Hall
1
3
2
5
4
6
XCSMono
XPP
AMO
SXR
XCS
CXI
H6
Part of LCLS
Installation Part of LCLS
MEC ARRA Funds

• Separate WBS 1.6 to combine all LUSI control
  data needs due to commonality in requirements,
  design, implementation, installation and
  integration
• XPP, CXI, XCS, Diagnostics Common Optics
• Common control and data systems design for LUSI
  and rest of photon beam-line instruments (AMOS,
  SXR, FES)

4
Scope WBS 1.6 Control Data Systems

• Included in W.B.S. 1.6
• All controls DAQ, labor and MS, for XPP, CXI,
  XCS instrument components and diagnostics/common
  optics included in baseline
• Includes controllers, racks, cables, switches,
  installation
• Data-storage and processing for FEH instruments
• Initial offline (more effort will be on operating
  budget)
• Input-signals to LCLS machine protection system
  link-node modules
• Provided by LCLS X-Ray End Station controls (CAM
  is G. Haller)
• Personnel protection system
• Machine protection system (LCLS modules, fibers)
• Laser safety system
• Accelerator timing
• Femto-second laser timing
• Network architecture security
• Data-storage and processing for NEH
• User safeguards
• Laser controls
• CXI 2D detector controls

5
ESDs and ICDs

• Two types of documents required for each
  instrument
• Engineering Specification Documents (ESDs)
• Interface Control Documents (ICDs)
• XPP
• SP-391-001-21 XPP Controls ESD
• SP-391-001-22 XPP Controls DAQ ICD
• SP-391-001-23 XPP DAQ ESD
• CXI
• SP-391-001-13 CXI Controls ESD
• SP-391-001-14 CXI Controls DAQ ICD
• SP-391-001-18 CXI DAQ ESD
• XCS
• SP-391-001-24 XCS Controls ESD
• SP-391-001-25 XCS Controls DAQ ICD
• SP-391-001-26 XCS DAQ ESD
• Diagnostics
• SP-391-001-19 LUSI Common Diagnostics Optics
  ESD
• All documents released at

6
Reviews

• Process Flow for Reviews
• Instrument Preliminary Design Review
• Controls Preliminary Design Review
• Instrument Final Design Review
• Controls Final Design Review
• Preliminary Design Reviews (PDR) for Instrument
  Controls and Data Systems held after instrument
  PDR and before the Instrument FIDR
• XPP Controls and Common Diagnostics PDRs
• Held February 7, 2009
• http//confluence.slac.stanford.edu/display/PCDS/X
  PPPreliminaryDesignReview
• CXI and XCS PDRs
• Held May 11, 2009
• http//confluence.slac.stanford.edu/display/PCDS/C
  XI_XCS-PDR
• Instrument FIDRs have recently been completed
• Final Design Reviews (FDR) for Instrument
  Controls and Data Systems planned for early
  October 09
• XRay End-Station (XES) Reviews under LCLS XES,
  not part of LUSI
• Common services e.g. Networking, DAQ, PPS, LSS,
  MPS

7
Controls

• EPICS
• In use at BaBar, APS, ALS
• It is the LCLS control system
• Basic EPICS Control and Monitoring
• Vacuum Instruments, connecting pipes
• Valve control
• Timing/triggering (timing strobe from EVR)
• Motion control (stages)
• Camera control
• Bias voltage supplies
• 120-Hz (slow) Analog-Digital Converters
• Digital IO bits/states
• Temperatures
• Hardware
• As much as feasible chosen from LCLS repertoire
• Added new controllers based on instrument
  requirements

8
Common Controls Hardware

• Examples
• Racks
• VME Crates
• Motorola CPUs
• Timing EVR PMC cards
• Cameralink PMC cards
• VME ISEG HV supplies
• Analog-digital converter modules
• Solenoid controllers
• PLCs
• Network switches
• Terminal servers (Ethernet-to-Serial Port)

9
EPICS/Python/Qt

• EPICS (Experimental Physics and Industrial
  Control System)
• Control software for RT systems
• Monitor (pull scheme)
• Alarm
• Archive
• Widely used at SLAC and other labs
• More http//www.aps.anl.gov/epics/
• Python/Qt is a user interface between the EPICS
  drivers and records and the user
• System is used for XTOD and AMO, provided as part
  of the XES Photon Controls Infrastructure

10
Example of Python/Qt User Interface
11
Example Vacuum

• All gauge controllers are MKS 937A
• Interface
• Terminal server DIGI TS16 MEI
• Automation Direct PLC
• All ion pump controllers are Gama Vacuum DIGITEL
  MPC dual
• All valves are controlled by PLC relay module
• The out/not-out state of all valves go into the
  MPS system to prevent damage if a valve closes
  unexpectedly.

12
Example Motion

• Control System provides support for all motions
• Motors
• IMS MDrive Plus2 integrated controller and motor
• IMS MForce Plus2 controller for control of in
  vacuum and other specialized motors
• Newport motor controllers
• Others as required
• Pneumatic motion
• Solenoid Driver chassis, SLAC 385-001
• Articulated Detector Holder (robot arm)
• Controls group to work with outside integrator to
  interface to EPICS control system

13
High-Level Applications

• To allow commissioners and users and of each
  experiment to
• Use a common interface to both the DAQ system and
  EPICS
• Speed up the development cycle by using a high
  level programming language, but still be able to
  easily build critical sections in C/C
• Easily develop new applications
• Provide a GUI integrated with the programming
  language
• Re-use code developed by other LUSI experiments
• Python as high level scripting language
• Easy to learn, fast dev cycle, extensible,
  open-source, powerful, relatively fast
• QT as graphical user interface
• Framework and support for scientists provided by
  PCDS

14
Controls Status

• ESD and ICDs released for all instruments
• Hardware order lists for LUSI XPP, CXI, XCS are
  available
• XPP items being ordered

15
Overall Status

• Control and data systems hardware and software
  components to be provided are agreed on and
  documented
• XPP controls data systems items are being
  ordered
• Following services required by XPP are already in
  place in hutch 3 or soon will be in place, months
  before required by XPP
• Hutch Protection System
• Laser Safety System
• User Safeguards
• Machine Protection System
• Network
• Timing (accelerator as well as femto-second
  laser)
• Racks including AC connections and cooling
• Data processing and storage
• Offsite data access and transport
• Racks for XPP on order, long-haul cable
  installation contracts in progress
• Software in progress

16
Data Sub-System

• Data Systems
• Challenge is to perform data-correction and image
  processing while keeping up with continuous
  incoming data-streams
• LUSI benefits that SLAC Particle Physics and
  Astro-Physics group is involved which has
  substantial experience acquiring, processing, and
  archiving large data volumes at high rates
• Use common dataflow/processing/storage offline
  interface DAQ for instrument components in the
  real-time detector data chain (BNL Cornell 2-D
  detectors, future SXR detector, waveform sampler,
  etc)
• Minimizes development, production, commissioning,
  and maintenance effort

17
Data System Architecture
To SCCS Offline

• Level 0 Control
• Run configuration control
• Run telemetry monitoring

• Level 2 Processing
• Pattern recognition, sort, classify, alignment,
  reconstruction

• Level 3 Online Archiving
• NEH/FEH local data-cache
• Local cache can buffer up to 4-days worth of data
• Offline will transport data to tape staging area
  in SCCS Computer Center

• Level 1 Acquisition
• Image acquisition, calibration
• Event-building with beam-line data
• Correction using calibration constants
• Data reduction (vetoing, compression)

18
LUSI Data Acquisition

• Cornell and Brookhaven 2-D pixel detectors are
  configured read out using the SLAC ATCA
  Reconfigurable Cluster Element modules
• Details in following slides
• XPP XAMP Detector with custom ASIC
• CXI Detector with custom ASIC

19
XAMP 2D-Detector Control and DAQ Chain
Beamline Instrument Detectors
Fiber
ATCA crate with SLAC DAQ boards, e.g. the SLAC
Reconfigurable Cluster Element Module
SLAC FPGA front-end board
Brookhaven XPP/XCS 2D detector-ASIC

• XAMP (XPP) LUSI instrument custom integrated
  circuits from Brookhaven are already connected at
  SLAC to SLAC LCLS high-performance DAQ system
• XPP BNL XAMP Detector 1,024 x 1,024 array
• Uses 16 each 64-channel FexAmps BNL custom ASICs
• Instantaneous readout 4 ch x 20 MHz x 16bit 20
  Gbit/sec into FPGA
• Output FPGA 250 Mbytes/s at 120 Hz
  (1024x1024x2x120)
• In addition BNL has ATCA crate with SLAC modules
  to develop software and test with detector

• ATCA
• Advanced Telecommunication Computing Architecture
  
• Based on backplane serial communication fabric,
  10-G E
• 2 SLAC custom boards (also used in other SLAC
  experiments)
• 8 x 2.5 Gbit/sec links to detector modules
• Dataflow and processing
• Managed 24-port 10-G Ethernet switching
• Essentially 480 Gbit/sec switch capacity
• Naturally scalable

20
Example XPP Online Processing

• Electronics gain correction (in RCE)
• Response of amplifying electronics is mapped
  during calibration
• Science data images are corrected for channel
  gain non-uniformity non-linearity.
• Dark image correction (in RCE)
• Dark images accumulated between x-ray pulses
• Averaged dark image subtracted from each science
  data image
• Flat field correction (in RCE)
• Each science data image is corrected for
  non-uniform pixel response
• Event filtering (in RCE or later)
• Events are associated with beam line data (BLD)
  via timestamp and vetoed based upon BLD values.
  Veto action is recorded.
• Images may be sparsified by predefined regions of
  interest.
• Event binning (processing stage)
• Images (and normalization) belonging to the same
  bin (dt, Eg, ..) are summed together

21
Example XPP Monitoring

• A copy of the data is distributed (multicast) to
  monitoring nodes on the DAQ subnet.
• The monitoring nodes will provide displays for
  experimenters viewing
• Corrected XAMPS images at 5 Hz
• Histories of veto rates, beam intensity, other
  BLD values.
• Reduced analysis of sampled binned data (versus
  scan parameter)
• Implemented with Qt (C/Python open source GUI)

22
Example XPP XAMPS Data Rates
4 x 2.5Gb PGP
10 GbE
L1 RCE
L2 Processing
L3 Cache
XAMPS
10 GbE
240 MB/s (480 MB/s)
lt 200 MB/s
n x (200 MB 20 GB)
Binned data archived at end of run (mins hrs)
Expect 6 60 GB / day
23
CXI 2D-Detector Mechanical/Electrical Vacuum
Assembly
SLAC PPA Engineering

• Positioning plate
• Supports quadrant raft
• Mounts to drive system

Cam follower mounted to torque ring
Hole size remotely adjustable Via PCDS Controls
One Quadrant Raft Removed
Pixel Detectors
Cut-outs in base plate for cold straps and cables
Cold strap
24
Quadrant Board and Electrical Interfaces

• Quadrant raft provides structural support and
  stability for the double-detector packages
• Feet mount on quadrant raft through holes in the
  quadrant boards
• This is also the thermal path
• Quadrant boards provide grounding, power, and
  signal interface to the PAD detector package
• 1 flex cable per detector
• 1 FPGA on each quadrant board

Cold strap
Quadrant raft
Mounting feet
Double-detector package (4 per quadrant)
Detector
Quadrant board 2
Quadrant board 1
25
ASIC Board

• Rigid-flex ASIC board (SLAC design)
• ASIC Bump-bonded to detector
• ASIC/detector package bonded to carrier board

26
CXI 2D-Detector Control and DAQ Chain
Vacuum
Ground-isolation
Fiber
Cornell detector/ASIC with SLAC quadrant board
Carrier Board
ATCA crate with SLAC DAQ Boards

• Each Cornell detector has 36,000 pixels
• Controlled and read out using Cornell custom ASIC
• 36,000 front-end amplifier circuits and
  analog-to-digital converters
• Initially 16 x 32,000-pixel devices, then up to
  64 x 32,000-pixel devices
• 4.6 Gbit/sec average with gt 10 Gbit/sec peak

27
DAQ Status

• Re-used significant fraction of Babar DAQ
  software
• Implemented zero-copy transmission/reception of
  network data (hard in Linux)
• Running full DAQ Chain (EVG/EVR/L0/L1/L2/L3)
  Configuring/Reading out e.g. Acqiris/Opal1000
  with zero-copy of objects in memory (better
  performance)
• Generating official data files. Iterating over
  them.
• XPP and CXI detector/ASIC connected to LCLS
  system and functional

28
Common Diagnostics Readout

• E.g. intensity, profile monitor, intensity
  position monitors
• E.g. Canberra PIPS or IRD SXUV large area diodes
  (single or quad)
• Amplifier/shaper/ADC for control/calibration/reado
  ut

FEL

• Four-diode design

• On-board calibration circuits not shown

• Board designed, fabricated, loaded, is in test

29
Interface to LCLS

• Interface to LCLS/X-Ray End-Station
  Infrastructure
• Machine timing ( 20 psec jitter)
• Laser timing (lt 100 fsec jitter)
• 120 Hz beam data
• Machine protection system
• Hutch protection system
• Laser safety system
• Networking
• EPICS server

30
120-Hz Data Feedback Loop

• Low latency 120 Hz beam-line data communication
• Use existing second Ethernet port on IOCs
• No custom hardware or additional hardware
  required
• UDP multi-cast
• Raw Ethernet packages

RF Phase Cavity
Accelerator
Experiment
IOC
IOC
IOC
120-Hz network
Timing

• Realtime per-pulse information can be used for
  e.g.
• Vetoing of image samples (using accelerator data)
• Adjustment of accelerator or photon beamline
  components based on instrument/diagnostics
  results
• Compensation of drifts, etc
• Transport of electro-optics timing result to
  hutch experiments

31
Organization

• 1.6. CAM G. Haller
• Deputy (P. Anthony)
• Online (A. Perazzo)
• Controls (B. Hill)
• DAQ (C. OGrady)
• Infrastructure (R. Rodriguez)
• Offline Computing (I. Gaponenko)
• Technical leaders are also responsible for AMO,
  SXR, and XES-provided photon area
  controls/DAQ/infrastructure needed by LUSI
• Provides low risk having interface issues,
  provides high efficiency
• Ensures common solutions
• No issue with man-power, plus instruments are
  time-phased.
• Scientist
• XPP (D. Fritz)
• CXI (S. Boutet)
• XCS (A. Robert)
• Diagnostics/Common Optics (Y. Feng)
• Detectors (N. Van Bakel)

32
(No Transcript)
33
(No Transcript)
34
Some Milestones

• XPP
• Controls PDR (done) Feb 2009
• Controls FDR Oct 2009
• Start installation of controls Jan 2010
• Controls ready to use Jun 2010
• CXI
• Controls PDR May 2009
• Controls FDR Oct 2009
• Start installation of controls Apr 2010
• Controls ready to use Oct 2010
• XCS
• Controls PDR May 2009
• Controls FDR Jan 2010
• Start installation of controls Oct 2011
• Controls ready to use Apr 2011

35
Summary

• Interface and Requirements documents released
• Clear what needs to be done. No issues, design
  meets requirements
• Design complete
• Most items are already used (hardware and
  software) in XTOD and AMO, ahead of XPP (and CXI,
  XCS)
• XPP, CXI, XCS Preliminary Design Review completed
• Most items similar to XTOD and AMO which both
  already had Final Design Reviews for Controls and
  Data Systems (XTOD is being installed, AMO will
  follow in August 09)
• Technical and cost/schedule risks are low
• Already know what is being used and quantity of
  items
• Already ordering XPP items
• Configuration and data acquisition for 2D
  detectors using SLAC ATCA system well advanced
• Data processing for XPP defined and is in
  progress
• Team