Best Ever Archive Utility, Yet

1
Best Ever Archive Utility, Yet

• Creating a BEAST was easy.
• BEAUtY seems harder.
• kasemirk_at_ornl.gov
• April 2009

2
Previous Setup
IOC
ASCII Config.
Channel Access
Archive Engine
config.xml
Binary Data Files
Data Server
XML-RPC

• History
• 2000Started byBob Dalesio
• 2003Index Tools,Data Server
• 2007CSS Client

CSS-based OPI
3
Details

• Older Java archive viewer also available.
• Archive Engines running on sampling computer,
  data server on serving machine.
• Scripts to restart engines, copy data, update
  indices.
• Headaches with data maintenance
• Index time grows with data
• Stuck when index files reach 2GB
• SNS Users faced with 80 sub-archives
• No clue what needs fixing after network/power
  problems
• No idea who contributes how many samples
• No way to remove selected channels or time ranges
• Improving on this means implementing an RDB

4
New Setup
IOC
Channel Access
ArchiveEngine
Samples
Config.
config.xml
RDB(Oracle/MySQL)
EngineConfig -Import
Other tools for config samples
CSS-based OPI
5
CSS Data Browser Handles Both

• New URL
• Just oneRDBsub-archive
• Old and newdata can becombined inone plot

6
Configure Sample Engines

• Configuration is in RDB
• Directly use SQL
• EngineConfigImport for legacy config files
• View/Edit via web
• Hierarchical (as before)
• Sampling engine (name, where to run, )
• Groups
• Channels
• No more duplicate channels!

7
Web Interface to Engine Config in RDB

• Tomcat/JSP/Servlets to view and edit

8
Web Config View Channel Stats
9
Web Config View Sample Stats
??
OK
10
Stats

• 34 sample engines, 79400 channels
• Host that runs sampling engine
• CPU load 45, zero disk I/O wait, very responsive
• Oracle performance
• 8000 samples/sec peak in write tests
• Operationally maybe ¼ of that
• Better configurationwould likely havefewer
  samples/sec

11
Report Example, SNS Beam Instr. Group

• May be SNS-specific, but shows usefulness of data
  in RDB

12
Summary

• Still testing, many Oracle setup issues
• Ran since Oct 08, then..
• Oracle cluster setup since March
• Updated partitioning in April
• Plan for about a year now
• Make operational in maybe 2 months
• Thanks go to
• Oracle Jeff Patton
• JSP Katia Danilova
• Beam Instr. Reports Mariano Padilla, Jeff,
  Douglas Edwardson
• IT Greg Lawson

13
Stuff
14
Hurdles

• Months Get new Oracle server configured
• Interface cards for storage array, fiber switches
• Firewall holes for office access, backup, admin
• Changes in 10g
• SELECT MIN(stamp), MAX(stamp) -gt NULL, NULL
• No range partitioning on Index-Organized-Tables
• Configuration issues
• ORA-01654 unable to extend
• ORA-00257 archiver error
• What used to be impossible is now easy, but
  still expensive
• DELETE FROM SAMPLES WHERE

15
Other Ideas

• Jlabs MyA
• Operational, but
• Wrapper code around MySQL to create cluster
• Handles less meta info (units, limits, )
• Viewer is one-of C/TCL/Tk
• Gabriele Carcassi mentioned RDDTool
• Toolset for logging data with data-aging
• Command-line RPN tools, web viewer
• May be faster than general-purpose RDB
• but lacks advantage of gpp RDB

16
SNS Setup for EngineConfigImport Tool

• Imports existing XML config files into RDB
• On srv02 in /imported_config
• .xml files copied from archive1, includessome
  editing to remove duplicate channels
• Review .log files
• 'chmk/chmkhebt/PPS_HEBTPLC_BChmk100AlarmContactA
  already in 'pps/pps_hebt
• Maybe edit your .xml file, refer to import.sh
  to re-import

17
Getting Sources, Building binaries

• Get sources from DESY CVS repository
• Maybe binaries real soon from ChannelArchiver
  web page
• Build binaries
• Find somebody who understands Eclipse RCP, how
  to use
• org.csstudio.archive.rdb/EngineConfigImport.produc
  t
• org.csstudio.archive.engine/ArchiveEngine.product
• org.csstudio.sns.product/SNS_CSS.product (or DESY
  CSS.product)
• Settings
• Compile-time defaults usually in
  preferences.ini and plugin_customization.ini
  files.
• Either change, or prepare file for
  -pluginCustomization command-line option

18
RDB Setup

• Currently supported MySQL or Oracle
• Main table difference Oracle TIMESTAMP has
  nanosecs, MySQL has separate nanosecs col.
• All tools auto-configure based on JDBC URL
• Table Schema
• ERD, MySQL DBD in org.csstudio.archive.rdb/dbd
• Setup for Oracle (table types, partitioning) not
  final

19
Run Archive Engine

• Command-line tool similar to previous engine
• More settings (RDB, Logging, JCA/CAJ, )
• Compiled-in defaults from preferences.ini and
  plugin_customization.ini
• Can specify overrides via
• -pluginCustomization my_plugin_customization.ini
• Mandatory command-line options for instance
• engine_name to locate config in RDB
• port for consistency, must match settings in RDB
• Built-in web server for status, restart, stop
• http//hostport/ Empty page (used to be main)
• ./main Main page
• ./reset Reset statistics
• ./restart Stop, re-read configuration, start
  again
• ./stop Stop