JAS3 A General Purpose Data Analysis Tool.

1
JAS3 A General Purpose Data Analysis Tool.
by Victor Serbo, SLAC
2
Contents

• Introduction to JAS3 (Java Analysis Studio)
• Basic Opening Files, Plots, Printing, etc.
• Web Browser, Editor, Java Compiler, Scripting
• Miscellaneous Spreadsheet, Configuration
  Save/Restore, Preferences, etc.
• Analysis
• AIDA Abstract Interfaces for Data Analysis
• JAIDA Java Implementation of AIDA
• AIDA JNI using JAIDA from C
• Analysis with JAIDA and JAS3
• JAS3 Plugins
• What is Plugin, examples
• Demo AmbientPlugin and Geant4 Plugin
• Demo JAIDA Web Interface

3
JAS3 Overview

• A completely new version of JAS
• Design based on Application Shell, into which
  many (optional) modules can be plugged
• Highly customizable for different application
  domains
• HEP/Astrophysics/Other
• Data analysis/Online Monitoring/GRID analysis
• Experiment/User specific modules
• Modules can be updated independently of shell
• Possible to release bug fixes fast
• Includes support for programming in many
  languages
• Scripting Pnuts, Python, .
• Command prompt
• Java (compiled)
• Analysis (histograms, tuples, fitting) based on
  AIDA standard
• Not technically backwards compatible with JAS2
• But migration is straightforward.

4
JAS3 Lightning Tour

• Tour designed to give you an overview of the
  capabilities of JAS3, you can try them out for
  yourself
• http//jas.freehep.org/jas3

Welcome Page, gives initial info and links to
example scripts and programs
Memory monitor
5
Opening Files
Use file menu
Drag from explorer
6
Plot Manipulation and Printing
Or copy/paste into Word, PowerPoint etc.
Can send individual plots or full page direct to
printer
Or save as PS, EPS, PDF, SWF, SVG, PNG, GIF
7
Java Editor, Compiler and Loader
Tree shows loaded programs
Built-in editor for writing analysis code
JAS3 allows any Java program to be loaded. This
example main routine is taken directly from the
AIDA manual
Built-in Java compiler
8
JAS3 Scripting

• JAS3 has multi-language OO scripting support
• Command line, Console, Editor
• Major components (e.g. AIDA) have scripting
  interfaces
• Currently have plugins to support
• Pnuts syntax almost identical to Java, fast,
  well documented and feature complete
• Python
• Use Jython (the Java implementation of Python)
• Limitation can not call Python libraries written
  in C
• More scripting languages can be added
• not restricted to Java implementations (e.g.
  could use C-Python, JPE)

9
Pnuts Language

• Currently support Pnuts scripting language
• Complete and well documented
• http//javacenter.sun.co.jp/pnuts/doc/guide.html
• Fast (although not as fast as compiled Java)
• Syntax very similar to Java
• Can easily call compiled Java classes from
  scripts best of both worlds

10
JAS3 Scripting II
Can also write and run scripts
Console allows direct interaction with scripting
language
11
JAS3 Spreadsheet

• Simple spreadsheet plugin
• for
• Displaying results
• Calculations
• Simple Plots
• Supports reading/writing
• .csv files
• Excel files
• Cut/Paste with Excel etc
• Coming Soon
• Scripting interface
• GUI for building plots
• User defined functions
• Java, scripting

12
Miscellaneous Features
Save/Restore configuration
User Preferences
Plugin Manager
13
AIDA - Abstract Interfaces for Data Analysis

• AIDA is a standard set of interfaces for creating
  and manipulating histograms, n-tuples and related
  data analysis objects, like Plotter, Fitter, etc.
• AIDA defines storage XML format.
• It has been created cooperatively by a group of
  developers working on high-energy physics data
  analysis tools. The goal of the AIDA project is
  to provide the user with a powerful set of
  interfaces which can be used regardless of which
  analysis tool they are using.
• Only define protocol for analysis objects, no
  internal details
• AIDA defines behavior, analysis tool provides
  implementation
• C and Java versions of interfaces are generated
  from the AID files.
• For more details on AIDA, see http//aida.freehep
  .org

14
AIDA Examples

• IHistogram1D
• IAxis axis()
• void fill(double x)
• void fill(double x, double w)
• double mean()
• double rms()
• double binEntries(int index)
• double binError(int index)
• .

• IAxis
• int bins()
• double lowerEdge()
• double upperEdge()
• double binLowerEdge(int index)
• double binUpperEdge(int index)
• double binWidth(int index)
• int coordinateToIndex(double x)
• .

15
JAIDA

• JAIDA Java implementation of AIDA
• Complete implementation of AIDA interfaces
• With a few exceptions
• limited support for plot styles currently
• Some missing features in ITree, cp, mv
• Uses Java service provider mechanism for
  dynamic loading of services at Runtime
• Can be used to add support for new fitters,
  stores etc.
• Have defined developer level interfaces for
  IOptimizer, IStore etc
• Uses Java Expression Library (JEL) for runtime
  compilation of expressions (for tuple evaluators
  and functions)

16
JAIDA Features continued

• Fitting
• support for multiple fitting engines (optimizers)
• UNCMIN (pure Java)
• Minuit (needs minuitAdapter native library)
• Support for multiple fitting methods
• LeastSquares, Chi2, BinnedMaximumLikelihood and
  UnbinnedMaximumLikelihood
• Stores
• Read/Write support for AIDA XML format (with
  built-in gzip)
• Read support for HBOOK/PAW files (needs
  hbookAdapter native library)
• Read support for Root files (write support
  hopefully coming soon)
• Future plans to add
• SQL interface, binary store format, text file
  support
• Plotting
• Two implementations of IAnalysisFactory
• BatchAnalysisFactory dummy plotter, ignore
  calls to IPlotter
• AnalysisFactory Full plotter functionality
• Multi-Threaded plots update in real-time as
  they are filled
• No need to call interact() or refresh()
• Support for embedding plotter

17
AIDA-JNI

• C interface uses AIDA-JNI

JAIDA (Java)

• AIDA-JNI partially generated by AID, partly by
  hand
• Does support full AIDA functionality

18
Using JAIDA with Java
19
Using JAIDA with Python
20
Using JAIDA with C
21
Future Plans for JAIDA

• Complete implementation of missing features
• Continue to track improvements to AIDA standard
• Add support for function arithmetic
• Web Interface
• Support for Remote AIDA
• Support for more storage formats
• Need good binary IO format .. either AIDA
  standard or roll our own
• Future AIDA developers meetings

22
JAS3 and AIDA

• JAS3 uses JAIDA - supports all AIDA
  functionality, including
• Histograms (includes arithmetic, projections,
  etc.)
• Clouds (unbinned histograms, scatterplots)
• Plotter
• Tuples
• Fitting AIDA interfaces allow for multiple
  fitters
• Uncmin -- pure java minimizer
• Minuit -- Fortran called by Java Native Interface
  (JNI)
• IO
• AIDA XML, PAW, Root
• JAS3 supports user interaction with AIDA in three
  ways
• Scripting (Pnuts, Python etc)
• Compiled (Java) code
• GUI Plotting, Fitting, Cuts etc.

23
Graphical Interface to AIDA
Histograms, Clouds, Tuples all presented in AIDA
tree
.aida files, .hbook files, .root files all
presented as AIDA objects
Drag items onto page, or use (popup) menus
24
Tuple Explorer - Plots
Histogram
Profile
Works with any tuple, read from file or
dynamically created
ScatterPlot
XY Data (More appropriate for smaller data sets)
25
Tuple Explorer Define Columns
26
Tuple Explorer - Cuts
27
Tuple Explorer - Tabulate
28
Tuple Explorer Record Source
To be used with record loop
29
Record Sources
Opening record (or event) based files causes the
run control toolbar to appear
Supports random access and tagged data sets
(mainly for event displays)
30
JAS3 Plugins

• JAS3 is based on the Freehep Application
  Framework, which provides a basic application
  framework into which extension modules, or
  "plugins" can be loaded.
• The application framework provides general
  purpose functionality, such as top level GUI
  (windows, menus etc.), help system, storing and
  retrieving user preferences etc.
• All of the data analysis specific functionality
  of JAS3 is provided by a set of extension
  modules.
• Modules communicate with each other via service
  interfaces and notification events. JAS3 comes
  with a standard set of service interfaces and
  notification events, but new plugins can also
  define their own interfaces and events.

31
JAS3 Plugin II

• Plugin functionality
• Do some plugin-specific task
• Use services provided by other plugins
• Provide services to other plugins
• Modify JAS3 menus and pop-up menus
• Core Plugins provide basic services
• Simple File Opener, Memory Status, Status Bar,
• More Complex Web Browser, Plotter, Console,
  XMLIO,
• Extensions separate packages
• Included with JAS3 distribution
• AIDA, TupleExplorer, JAVA Compiler and Editor,
  Spreadsheet,
• Specific to particular Experiment
• BaBar Ambient DB Browser AmbientPlugin, Geant4
  Plugin, etc.
• Partial Tutorial on writing a plugin
  http//jas.freehep.org/jas3/developers/tutorial.js
  p

32
JAS3 Plugin Demo I

• AmbientPlugin
• Access BaBar Ambient DB via CORBA
• Present Time History as an AIDA object
  (DataPointSet)

CORBA Server (C)
BaBar Ambient DB
JAS3 (Java)
AmbientPlugin (Java CORBA Client)
33
JAS3 Plugin Demo II

• Gean4 Plugin

Geant4 Application (C)
Java RMI Server
JNI Layer
JAS3 (Java)
Geant4 Plugin (Java RMI Client)
34
AIDA Web Interface
35
Goals

• Provide access to Histogram plots without
  requiring any software installation
• Putting results on to the web for collaborators
  etc.
• For example for viewing results of quality
  control plots
• Viewing live histograms on the web for
• Online monitoring
• Monitoring progress of batch jobs
• Add as much functionality as possible in a
  generic way so that it can be used by others
  (with minimal work)
• Complete out-of-the-box web application
• Packaged as a .war file for use with any J2EE app
  server
• (e.g. Tomcat)
• Configurable via a simple .ini file (or better
  via a web interface)

36
Architecture
.root
.paw
.aida
37
Features

• Current Feature set
• Display of AIDA Tree for selection of
  histograms
• Displays plots as .png or .gif images in browser
• Allows (limited) control of image properties
• Allows (limited) control over plot properties
• Allows comparison to reference plots (including
  Kolmagorov test)
• Allows export of plots to PDF, EPS, SVG, SWF,
  GIF, PNG, etc.
• Planned features
• Extend to support n-tuples as well as histograms
• Dynamic definition of new columns and cuts
  (evaluators, filters)
• Web based fitting (including function definition)
• Support multiple plots per page
• Thumbnail browser
• Extend which plot options can be configured
  through web
• Allow users to retain settings via cookies

38
Current Interface
Can try from the web http//glast02.slac.stanford
.edu8080/aida/welcome.html
39
Status

• Currently released JAS3 version 0.7.3 (0.7.4 next
  week)
• AIDA functionality is quite solid
• Compiler, Loader, Record Loop all quite recently
  added,
• Certainly still some rough edges
• Documentation limited but available
• Built-in example scripts and programs
• Tutorial on web
• If you used JAS2 you will find some functionality
  not yet ported to JAS3
• Remote (client/server) access to data.
• 3D Lego/Surface plots

40
JAS3 Conclusion

• JAS3 is a powerful and flexible Analysis Tool
• JAS3 uses Plugin Architecture
• Easy to add new functionality
• Easy to fix problems with existing modules do
  not have to wait for new release of all libraries
• JAS3 release 0.7.3 is out try it!
• It is still a beta version, so bugs happen
• We need your input to make JAS3 better!
• Report bugs, problems, suggestions at
  http//bugs.freehep.org
• Do not have to be on JAS team to be JAS developer
  ideas/code for general-purpose plugins are
  always welcome
• Try writing JAS3 plugins for your own needs for
  Experiment and Home ?

41
JAS3 Links, More Info

• Report Issues http//bugs.freehep.org/
• JAS Java Analysis Studio - http//jas.freehep.or
  g
• JAS3 http//jas.freehep.org/jas3
• JAIDA http//java.freehep.org/jaida/
• AIDA http//aida.freehep.org
• FreeHEP - http//www.freehep.org
• FreeHEP Java Libraries - http//java.freehep.org
• WIRED http//wired.freehep.org