NLC Archiving Requirements Preliminary

1
NLC Archiving Requirements(Preliminary)
2
Why does the NLC need a new Archiver?

• Since the NLC will be such a complex machine,
  there needs to be greater flexibility in the data
  archived than is currently available in order to
  troubleshoot problems
• The archived data for some devices need to be on
  the scale of milliseconds for a short period of
  time (seconds) while data for other devices may
  be on the scale of minutes over a longer period
  of time (10s of minutes to hours)
• Data viewing from many different control system
  applications should have a consistent interface
  such as archived data, correlation plots data,
  buffered data acquisitions, configuration files,
  etc.

3
Why does the NLC need a new archiver (cont)?

• Collaborators will be internationally based so
  access to data must be accessible from around the
  world
• The amount of data is huge (not including
  fast-timed based archives)
• 600,000 pvs
• x 80 bytes/pv
• x 20 sets of data/hour
• ---------------------------------
• 1.0 Gbyte/hour
• x 24 hours
• ---------------------------------
• 24 Gbytes/day

4
Requester and Archiver

• Requester
• Interface between user and archiver
• Supports time-based entries, monitor-based
  entries, triggered entries, time-based
  monitor-based entries and fast-timed entries
  (ring buffers)
• Validates requests
• Bookkeeper of archived channels/rate/requesters
• Limits requests when necessary
• Generates diagnostic reports

• Archiver
• Retrieves pv data at specified rates or monitor
  intervals and stores the data in files quickly
  and efficiently
• Limited ring buffer capability
• Ability to create other data files for
  configuration data, correlated data, etc.
• Must be able to handle up to 600,000 pvs
• Much of the data will come in bursts instead of
  regular intervals

5
Data Converter and Data Retriever

• Data Converter
• Runs periodically to convert data from fast file
  save format to fast file access format
• Not part of the archiver
• Ages and compresses data when necessary
• Calculates statistics of compressed data
• Keeps master index of data locations

• Data Retriever
• Accepts requests to retrieve data
• Validates requests
• Utilizes master index to locate data
• Can process data in fast file save format and
  also fast file access format
• Can retrieve data from archiver, correlation
  plots, buffered data acquisitions and
  configuration files
• Sends data back to requester

6
Browser

• User interface for data retrieval
• Web based component
• Capability to display graphs and histograms
• Capability to perform simple calculations (mean,
  std dev, etc.) and simple fits
• Ability to zoom, pan and lasso graphics
• Ability to export data to various output formats
  (MATLAB, spreadsheets, etc)
• Ability to display time-based data, correlated
  data, correlated history data, configuration
  data, etc

7
Requester/Archiver Questions

• Requester Questions
• Can anyone make archiver requests? Should there
  be special privileges for different types of
  requests (based on rate? Subsystem?)
• Do people based internationally need this
  capability? Should it be web-based?
• Archiver Questions?
• How closely syncd in time do the IOCs need to
  be?
• What are the requirements for fast-time based
  entries (ring buffers)? How many data points
  should be available? What is the maximum
  frequency which needs to be sampled (based on
  hardware requirements)?

8
Data Converter/Data Retriever Questions

• Data Converter Questions
• How is data aged? Should statistics be kept on
  all aged data? If so, what type? (May depend on
  hardware requirements).
• Do users have input as to how much data is kept?
• How long should data be kept available online?
• How often should data be converted? Once daily?
  More often?
• Performance requirements?
• Data Retriever Questions
• What are the performance requirements? How much
  data, how fast so that it is acceptable to users?

9
Browser Questions

• Does the browser need security imposed or can
  anyone request data?
• What are the display performance requirements?
  How many pvs? How many graphs? Timing?
• How sophisticated should the calculation
  capability be? Can the browser do simple
  calculations and simple fits and rely on
  exporting data for sophisticated calculations and
  fits?
• Or, do we want the browser to encompass
  everything the user wants to do with the data?
  Should MATLAB (or some other calculation engine)
  and some spreadsheet program be integrated into
  the browser?

10
Technical Challenges

• Optimizing algorithms which stores data and
  accesses data fast and efficiently
• Finding storage mediums to handle the volume of
  data which will be archived and that is capable
  of meeting performance requirements
• Creating a robust system with built-in
  redundancies which can handle failovers
  seamlessly
• Designing a system which meets requirements now
  yet is flexible since the requirements will
  inevitably change over time