Fermilab Control System ACNET

1
Fermilab Control System(ACNET)

• J. Patrick
• February 17, 2005

2
Fermilab Control System

• Overview
• Applications
• Central Services
• Front-ends/Instrumentation
• Timing
• Security and Remote Access
• Summary

3
Fermilab Control System

• aka ACNET
• Unified control system for the entire complex
• 400 MeV Linac
• 8 GeV Booster Synchrotron
• 120 GeV Main Injector Synchrotron
• 1 TeV Tevatron Synchrotron
• antiproton source target/debuncher/accumulator
• antiproton Recycler storage ring
• fixed target lines
• Simultaneous operation of
• Tevatron proton-antiproton collider (storage
  ring)
• Antiproton production and storage (0.5 Hz)
• 120 GeV fixed target to Meson lab (0.1 Hz)
• 120 GeV fixed target to NUMI/MINOS (0.5 Hz)
• 8 GeV fixed target to MiniBoone (5 Hz)

4
Fermilab Accelerator Complex
5
Control System Overview
Console Applications
Java Applications
Web Applications
Database
Application
CentralServices
Open Access Clients
Servlets
Consolidators
Central
ethernet
IRM Front-Ends
MOOC Front-Ends
Labview Front-Ends
Front-End
field bus VME, SLD, Arcnet, ethernet,
Field Hardware
CAMAC, VME, PMC, IP, Multibus, CIA, GPIB,
6
History

• Originally developed for Tevatron in early 80s
• Substantial evolution over the years
• CAMAC field hardware -gt very broad diversity
• PDP-11/Mac-16 front-ends-gt x86 Multibus-gt VME
  based processors Labview
• DEC PCL network-gt IEEE 802.5 token ring-gt
  ethernet
• VAX database -gt Sybase
• VAX central services -gt Java
• PDP-11 applications -gt VAXstations-gt Java
  application infrastructure-gt Port of VAX code to
  Linux in progress

7
System Scale

• 200,000 devices/350,000 properties
• 475 front-ends
• 90 Sun Netras 12 Linux PCs that run Java based
  data loggers, servlets, open access clients
• VAXes to run VAX central services
• 70 VAXes that run high console level applications
• 70 Linux PCs to to parallel VAXes until VAX
  retirement.
• X-displayed out to desktop PCs

8
Device Model

• lt 8 character devices names
• e.g. TBEAM
• Informal conventions provide some rationality but
  no enforcement
• More verbose hierarchy layer created, but not
  utilized/maintained
• Each device has one or more properties
• Reading, setting, analog digital alarms,
  digital status control
• Device descriptions stored in Sybase database
• Scaling transformations defined in device
  database
• Add new devices with DABBEL command language on
  VAX
• Templates available for various device types
• View/modify device definitions with GUI
• Monotype data, arrays are transparently supported
• Structured data is often done, though support is
  not transparent
• Application (library) layer must properly
  interpret the data from the front-end and perform
  required data translations

9
Device Database GUI
10
Communication

• Applications/Central Services/Front-ends
  communicate via ACNET protocol
• Custom protocol now exclusively based on UDP
• Extensive network/switches etc
• Asynchronous request/reply request multiple
  reply
• Timeout/retry mitigates UDP unreliability.
• RETDAT/SETDAT protocol for readings/settings
• Can have multiple devices/packet
• Fast Time Plot (FTP) returns blocks of readings
  taken at up to 1440 Hz every 0.5 seconds
• Snapshot protocol returns blocks of 2048 points
  taken at an arbitrarily high rate

11
Communication

• Application communication with front-ends passes
  through central layer
• Datapool Manager on VMS via shared memory
• Data Acquisition/Server Engine (DAE) in Java (via
  RMI)
• Consolidation of requests to avoid overloading
  front-ends
• Centralizes security, monitoring, error logging
• Issues include
• Limited packet size and internal device address
  space.
• Timestamps not part of protocol packet, applied
  on data receipt
• GPS sync clock event sent to all front-ends, can
  be included in structured devices
• New get/set32 protocol addresses the above, only
  on centrals now
• Lack of transparent support for structured data.
• Bridging to network devices that dont directly
  support ACNET is via front-ends or Open Access
  Clients

12
Database

• Central Sybase database
• Three parts
• Device and node database
• Application database
• Used by applications as they wish
• Save/Restore/Shot Data database
• MySQL used for data loggers, and for some other
  specialized purposes. Not for general application
  use

13
Application Environment

• Two major high level application environments
• Console environment
• VAX based
• C language ( legacy FORTRAN)
• Port to Linux in progress
• Expect some operational utilization in the fall
• Java environment
• Platform independent
• Windows/Sun/Linux/Mac all supported and utilized
• Full access to entire control system
• Both environments have a standard application
  framework that provides a common look and feel
• All applications launched from Index Pages
• Currently separate pages for Console, Java
  environments
• Also some browser applications of varying styles

14
Index Page (Console)
15
Application Index (Java)
Web Browser
Standalone Application
16
Application Overview

• Utility applications
• Parameter Page
• Fast Time Plot
• Data Logger viewer
• Database Viewer
• Synoptic displays
• Diagnostics
• Save/Restore
• Alarms
• Machine Specific Applications
• 600 Primary 125 Secondary applications in
  console environment
• gt100 applications in Java environment
• Mostly utility, shot data analysis

17
Parameter Page (Console)
18
Parameter Page (Java)
19
Fast Time Plot (Java)
20
Synoptic Display (Console)

• Graphical BuilderLex Draw

21
Synoptic Builder (Java)
22
Synoptic Display (Web)
23
Sequencer (Console)

• Automatically Sequence Collider Shots, other
  activities
• Relies on ACL scripting language

24
Finite State Machine (Java)

• Graphical Builder
• Primary current usage is Pelletron control
• Keeps track of HV sparks and beam related trips

25
Sequenced Data Acquisition

• Automated acquisition of data for collider shots
• Configure readings for various stages of collider
  shot
• Snapshots/FTP as well as single scalar readings
• SDAEdit tool makes it easy to add new things
• Data acquired automatically and stored in Sybase
  database
• Summary html reports generated when store goes in
• Step efficiencies, error reports, luminosity
• Supertable captures 200 most popular items
• Html, Excel, Java Analysis Studio formats
  automatically generated
• Suite of tools to view detailed data
• Data can be directly read by Excel, Java Analysis
  Studio, potentially other external tools
• Simple Java API for user written code

26
Alarms Display
27
Scripting

• Accelerator Control Language (ACL)
• Console environment
• Access to much of the functionality in the
  control system
• VMS DCL style syntax
• Extensively used in Sequencer
• Jython
• Java API in principle accessible to Jython
• Some but limited testing/use to date

28
External Tools

• Excel
• Excel based parameter page
• Easy cut/paste of Java tables into Excel
• Excel data can be imported into a variety of
  tools
• MathCAD, OptimA used locally
• MATLAB
• MATLAB supposedly has a standard Java interface,
  in theory it should be straightforward to
  integrate with the control system
• However we havent tried it
• An attempt to do this with Mathematica gt 1 year
  ago was not successful, it does not use a
  standard Java distribution
• Java Analysis Studio
• Currently data logger and shot data plug-ins

29
Excel Parameter Page
30
Java Analysis Studio
31
Save/Restore

• Devices may be assigned to save/restore lists
• Console applications to save/restore to/from
  database on user request
• Scheduled saves of the entire control system 4
  times/day

32
Electronic Log

• Based on Oak Ridge product
• Web browser client
• Perl/cgi back-end
• Shift log machine / subsystem logs
• Used by A0
• API to add entries programmatically, including
  images
• Both Console and Java environments
• Java application framework includes standard menu
  item to make posts.

33
Central Layer

• Open Access Clients (virtual front-ends)
• Java Servlets
• Bridge requests from outside firewall
• Request Consolidation
• Alarms server
• Front-end download
• Other things

34
Open Access Clients

• Processes with no user interface, always running
• Obey same communication protocol as front-ends
• Compare to EPICS Soft IOC
• Several classes
• Utility Data loggers, scheduled data
  acquisition, virtual devices
• Calculations Both database driven and custom
• Process control Autotune of fixed target lines
• Bridges to ethernet connected scopes,
  instrumentation, control systems
• Easy access to devices on multiple front-ends
• Friendlier programming environment than VxWorks
  front-ends
• Framework transparently handles ACNET
  communication
• Access to database, other high level operating
  system features
• Do not provide hard real-time response
• Clock events via ethernet multicast
• gt 100 Almost all run in Java framework VAX
  framework deprecated

35
Data Loggers

• 70 instances running in parallel on central
  nodes
• 80 GB compressed storage per node in MySQL
  database
• Circular buffer wraparound time depends on
  device count/rates
• Data at maximum 1 Hz rate extracted to Backup
  logger every 24 hours
• Backup logger is available online gt data
  accessible forever
• Generally allocated to various departments to
  configure as they wish
• No central management of archiving lists, so
  inevitable duplication
• Tasks follow the Open Access Client architecture
• Simple GUI to specify logging lists
• Single readings or snapshots on clock events or
  at fixed rates to 15 Hz
• Programmatic logging interface available for
  custom loggers
• Variety of retrieval tools available
• Standard console applications, Console and Java
  environments
• Web applications
• Java Analysis Studio
• programmatic APIs

36
Front-Ends

• Interface field hardware to the high level
  control system
• Several architectures
• MOOC Minimally Object-Oriented Communication
• Very wide diversity of functionality
• Processors all VME (VXI) based
• Very wide diversity of field hardware supported
• Links to many other form factors such as CAMAC,
  etc.
• MVME 162 (68k) and 2400 (Power PC)
• VxWorks Operating System
• 275 in the system

37
Front-Ends - IRM

• IRM Internet Rack Monitor
• MVME 162 PSOS operating system
• Chassis includes 64 channel x1 KHz 16 bit
  continuous digitizer
• 8 x 10-20 MHz option available
• 8 bytes external digital I/O
• 8 x 12 bit DAC outputs
• Expandable via IndustryPak I/O
• 125 in the system
• Eight used at A0 photo-injectorVariety of
  functions
• A few at TTF for modulators

38
Front-Ends

• Linac front-ends
• VME based Power PC processors
• Non-MOOC infrastructure tied to 15 Hz Linac
  cycle rate
• 50 in the system
• HRM Hotlink Rack Monitor
• Evolution of IRM architecture
• I/O chassis separate from processor chassis
• 64 x 10 KHz ADC up to 2 per chassis
• 10-20 MHz snapshot board
• Timestamps included with ADC data
• 8 byte digital I/O per chassis
• Multiple chassis per PPC processor
• VxWorks (not MOOC)
• Initial build of 15 units in progress

39
Labview

• Front-Ends
• General control/acquistion from high level
  controlsystem
• Interact via ACNETprotocol written inLabview VI
  language
• No FTP/Snap
• Non-local consoleaccess viaPCAnywhere,Timbuktu
  etc.
• 25 in system
• Instrumentation
• Flying Wires
• Sample Bunch Display
• Both PCs Macs

40
Field Hardware

• Wide variety of hardware supported
• CAMAC, VME, VXI, Multibus, CIA,
• Too extensive to describe here
• Support for wide variety of commercial
  instrumentation
• Scopes, spectrum analyzers, signal generators,
  DVMs,
• Ethernet-gtGPIB direct ethernet
• Through front-ends through OACs direct in some
  cases
• Mostly in console environment, ports to Java as
  needed
• Siemens-Moore APACS PLC supervisor
• Used for cryogenic control
• Interface through Open Access Client using
  library from company
• Other PLCs also supported through front-ends

41
Bridge to External Systems

• Accelerator NETwork control system (AccelNET) is
  a software package designed for control of
  electrostatic particle accelerator systems.
• Runs on PC hardware under the Linux operating
  system
• Modular Linux programs that communicate to each
  other through a central database server using
  TCP/IP.
• CAMAC- GROUP3 I/O
• ACNET??AccelNET Bridge

42
Timing and Control Networks

• TCLK (Clock Events)
• 10 MHz Manchester encoded signal with 8 2
  bits/event
• 256 possible events defined for the complex
• Decoding hardware/signal generators for variety
  of buses
• Standalone version of this system used at A0
  photo-injector
• MDAT (Machine Data)
• 10 Mbit/sec serial link 8 bit type 16 bits
  data per frame
• Originally developed to rapidly distribute ramp
  currents around the main ring and Tevatron usage
  has broadened
• Beam Sync revolution markers derived from RF
• States
• State devices defined
• Any node may issue transition request, gets
  multicast everywhere
• Can be distributed via MDAT
• Beam permit/abort
• CAMAC interfaces.

43
Security

• Control system network inside a firewall
• Restricted access from/to outside the accelerator
  division network
• Privilege classes assigned based on account,
  service, (node)
• Control from where applications may be run
• Control which programs/users/nodes may do
  settings
• Emphasis on accountability as well as security
• Setting log
• Application log

44
Remote Access

• Console environment via remote X display
• Administrative controls on usage, particularly
  settings etc.
• Java applications can be launched from anywhere
  from a browser via Web Start
• Access to inner parts of control system only via
  VPN
• Applications to be run remotely w/o VPN must be
  designed for that
• Applications that work via servlets
• Secure Controls Framework Subset of data
  access API implemented in this way, better
  performance than servlets/xml
• Access from outside the standard application
  environments
• xml-rpc is preferred method
• Readings of any device setting of virtual
  devices
• No setting of hardware associated devices
• Used by all current experiments

45
ACNET Summary

• Should meet requirements for SMTF
• Has been/is being continually modernized
• Extensive experience at FNAL.
• Less time to get a system running
• A broad range of people could contribute
  quickly
• Limited experience outside of Fermilab
• Remotely accessible but full system not portable
• Documentation