Software for MAPS and MERLIN

1
Software for MAPS and MERLIN
T.G. Perring ISIS Facility, Rutherford Appleton
Laboratory
2
Outline

• Plan of talk
• Overview
• Brief description of the present MAPS/Het/MARI
  software
• Challenges of MERLIN/ARCS
• How we plan to re-write the software
• Our aims from this meeting
• Contribute to the plans for direction of ARCS
  software development
• Explore cooperation in software development
• Experience a working ARCS-like spectrometer
• Fresh view of the problems of handling huge
  inelastic datasets

3
MAPS spectrometer

• Specification
• 20meV
• lmod-chop 10m
• lsam-det 6m
• low angle bank 3?-20? high angle bank ?
  60?
• ?hbar?/EI 1- 5 (FWHH)
• 40,000 detector elements
• 2500 time channels
• ?108 pixels ? 0.4GB datasets

Background chopper
Monochromating chopper
Sample position
Position sensitive detector array
4
Overview

• Inelastic scattering from single crystals
• Triple axis spectrometer point-by-point serial
  operation
• Few kB of data
• Every group of users writes its own least-squares
  fitting algorithm from scratch
• Ad-hoc cooperation only
• MAPS/MERLIN/ARCS
• Parallel operation collecting on 3D surface in
  4D S(Q,w)-space
• MAPS raw data 108 pixels 0.4GB
• corrected data 1-2 x 107 0.1-0.2GB
  x 10 datasets
• MERLIN, ARCS 1-2 x 107 0.1-0.2GB x
  100 datasets
• Need Rietveld refinement for inelastic
  scattering
• Groups cannot afford to write own software
• Pointless anyway e.g. in diffraction GSAS, CCSL
  are trusted
• We should move to a similar model of trusted
  black boxes

5
Overview

• (Rietveld refinement cont.)
• Use entire data set to extract a limited number
  of parameters in a model
• Magnetic coupling constants
• Force constants
• Full inversion of the data if S(Q,w) in 4D
• Pragmatism
• In magnetism S(Q,w) (or ?''(Q,w)) is the
  quantity to test
• Qualitative features (are there antiferromagnetic
  fluctuations and if so, where ?)
• Unknown processes distorting measure of goodness
  of fit
• Rapid slicing and dicing, testing models on
  limited volumes of data, testing implications for
  other parts of the data volume, feeding back into
  operation of experiment
• seamless integration of simulation,
  visualisation and analysis programs
• More formal cooperation
• Agreed data structures for instrument
  information, counts/ S(Q,w)
• Definitions for input/output of algorithms
• Capitalise on the investment we individually make
  in algorithms

6
Current software on MAPS, HET, MARI
Planning experiment Tobyplot (reciprocal space
viewing) Chop (resolution, flux)
Data reduction HOMER Genie-II
Monitoring experiment Genie-II OpenGenie
Visualization MSLICE GUI driven Matlab
Analysis Tobyfit (single crystals) Ad-hoc
programs in Multi-frills MFIT MSCATT
7
Tobyplot
Ei450 meV Psi101.3

• Reciprocal space viewing
• Mostly important for 3D systems
• but very useful for 1D, 2D, to assess access in
  reciprocal space

Fortran77 PGPLOT VMS/Windows/Unix
8
CHOP

• Flux
• Resolution
• At elastic position
• As function of

Test flux/resolution compromises
Fortran77 PGPLOT VMS/Windows/Unix
9
Monitoring an Experiment
GENIE-II I(t) for single spectrum rebinning,
units conversion, integration Algebra on spectra
e.g. W1(0.3W2 W3)/W4 can call user-written
FORTRAN algorithms Good for quick checks during
and after experiments Still use today on MAPS VMS
only OpenGENIE New generation Used on many ISIS
instruments Windows, Unix, VMS Not the features
of Matlab, IDL Is free, however
10
Data reduction
HOMER (au. Ray Osborn) I(detno, t)
corrects Kf/ki, efficiency(kf)
S(detno, ?)
ASCII output file (or VMS binary)

• Encapsulates years of experience of the
  instruments
• Scaled very well even to MAPS
• homer/mapparpix_981.map/mask8900/van8850
  8900 100 -30 95 0.25

11
Imaging single crystal data on HET, MARI, MAPS,
and IRIS (Radu Coldea (ISIS / Oak Ridge now
Oxford)
MSLICE
2D slices in (Q,?)

• GUI interface
• run info.
• sample parameters
• 2D 1D cuts

1D cuts in (Q,?)
12
Least squares fitting of resolution broadened
cross-section models (Toby Perring, ISIS)
TOBYFIT
Simultaneous fitting to many 2D or 1D data sets

• Text-based interface for entering
• instrument
• sample parameters
• cross-section parameters

13
MSLICE and TOBYFIT
Integral part of the operation of the
spectrometer - Tertiary spectrometer MSLICE
Visualisation of 3D data in 2D slices, 1D
cuts Can generate backgrounds from selected
parts of the data MATLAB as front
end GUI Graphics manipulating data
structures, ad-hoc programming FORTRAN77 for
speed of operation of algorithms PC with 1GB
RAM, 500MHz necessary TOBYFIT Fitting and
simulation?test ideas by feeding S(detno, ?) to
MSLICE Hold an experiment in a parameter
file Fortran PGPLOT Runs on VMS, Windows,
UNIX Communicate via ASCII files 1 for data, 1
for detector parameters other sample
instrument parameters
14
Challenges offered by future instrumentation

• Physics a function of 4 variables
  C(x1,x2,x3,x4)
• Instrument gathers data on a 3D volume in that 4D
  space
• Data gathered on a fine non-Cartesian grid
• 0.1-0.2 Gbyte
• MAPS usually 10 settings in an experiment
• fine data on 3D surface, coarse in 4th dimension
• ARCS/MERLIN 100 settings
• fine in all 4 dimensions (scan Ei, or crystal
  orientation)
• 10-20GByte complete data set
• Data has low statistics - need techniques to pick
  out features in data
• will always need real-time slicing and dicing of
  data
• too many ways of being led astray or being
  deceived
• will be doing this after going back to home
  institution

15
Visualisation a hierarchy of views

• View 4D data
• ? How do that ?
• Define integration interval along any one
  dimension, and then
• View 3D data
• isosurfaces slider control for
• intensity levels rotation and viewpoint
• binning along the three axes smoothing, image
  processing control
• Move a plane through the 3D volume define
  integration interval along one of the remaining
  dimensions, and then
• View 2D data
• Contour plots, mountain plots, slider controls
  for
• contouring levels interval scaling (linear,
  log, sqrt )
• binning along the two axes
• Move a line across the plane, defining a
  thickness, and then
• View 1D data
• overplotting, fine comparison
• book-keeping of titles of the plots ...

On raw counts, white beam files, as well as
S(Q,?)
16
Instrument resolution, modelling

• Must be able to simulate results of experiments
  and view results in same way as data (number
  crunching)
• Must perform on-line analysis (resolution-convolut
  ed model fitting,multiple scattering) within
  framework of same package (even more number
  crunching)
• flexibility fit on limited volume of data,
  simulate for whole dataset, slice-and-dice in
  same way as data to try out ideas
• User wants one-stop shop

Compare, fit
I(det,t)
Sexp(Q,w) inst
S calc(Q,w)
Convolve with instrument
Visualisation, algebra on 1,2,3,4D
Tobyplot MkII
17
Issues

• Number crunching - more than the typical user
  institute will have
• huge storage requirements (not just the raw data)
• data management a real problem - we already
  create hundreds of cuts
• thumbnails when click on file
• database functions (select by date, temperature,
  field, scan of a parameter)
• history of analysis stored in file
• well-defined data structures needed
• ease interact seamlessly with other programs
• deconvolution, modelling
• user-written algorithms easy to write
• define appropriate methods and algebras
• (addition, subtraction, background generation,
  symmetrization)
• not just GUIs
• scripting must always be possible
• maximises flexibility

18
Our plans

• Existing programs need to rewritten
• F77, getting unmaintainable, monolithic,functional
  ity insufficient, grown organically, written
  independently
• Define NeXus files to hold all processed files
• 1D,2D,3D,4D data all relevant instrument
  information (detectors etc.)
• sufficient information for MCSTAS simulation
• include raw data files eventually
• Mirror the data structures in Fortran95 methods
• Fortran95 for speedy algorithms
• number crunching for visualisation (MSLICE-2),
  fitting (Tobyfit-2)
• MATLAB for graphics, language for manipulation
  and scripting - the glue
• access to all features of MATLAB for ad-hoc
  manipulations
• TGP, SMB inst. Scientists part effort from 2
  post-docs (25-50 effort from each)

19
(plans cont.)

• About to start working with E-Science centre at
  Rutherford Laboratory
• funding to implement grid based applications for
  science within the laboratory
• demonstration projects
• data portal to distributed data stores
• graphics processing and number-crunching
• ISIS projects chosen to focus the development for
  real applications
• aim to isolate user from where the work is done -
  no need for their own Beowulf cluster
• user will have a front end - in our case we want
  MATLAB
• but also web-based interface (oceanography, space
  science)
• Ensures that one version is maintained
• assumes high-speed networks
• GLOBUS toolkit to isolate user from location of
  resources
• ISIS full effort of ISIS computing staff member
  post-doc
• effort of E-Science centre