McStas Neutron Instrument Simulation on the TeraGrid

1
(No Transcript)
2
McStas Neutron Instrument Simulation on the
TeraGrid
TeraGrid 06

• Mei Li Chen, John Cobb, Garrett Granroth, Mark
  Hagen, Steve Miller, Jim Kohl
• Oak Ridge National Laboratory
• Spallation Neutron Source

June 14, 2006
3
Career Trajectories
First time presenting co-workers talk
First Keynote
First Invited Talk
After Dinner speaker
First Plenary session Talk
From John Brown, After Dinner Talk, ACNS 2005
4
PSoNI, the Parallel Simulations of Neutron
Instrument on Teragrid
Pronounciation Key The P is silent
5
Out line

• Motivation
• Software Environment PSoNI
• - Design Goals
• - What is PSoNI
• - Current status
• - Performance
• How to use PSoNI
• - PSoNI_ps for Parametric Simulations,
• - PSoNI_ms for Massive Simulations.

6
Motivation

• Fast instrument simulation is needed for
• Instrument design and enhancements/upgrades
• Virtual experiments
  Pre-experiment planning/verification/feasibility
  (for proposals!) During experiment
  for interpretation and on-the-fly planning
  For analysis, comparing models to data

I want fast instrument simulation of SEQUOIA, 3
hours to trace 108 neutrons is too slow for me!
7
Motivation (cont.)
Parallel computing on high-performance computers
can provide a good solution

• To parallelizing instrument simulation, we need
• Dedicated high-performance computing resources
  
• - We do have Teragrid and SNS resources
• A software environment
• - in which users serial simulation code can
  run in parallel with minors efforts.

8
PSoNI, A Software Environment for parallelizing
Neutron Instrument Simulations

• Goals of Software Design
• 1. The problems to be tackled
• Larger simulations Requires 109 or
  more rays
• Parametric simulations One program
  runs many times with different parameters
• 2. No culture shock to users
• Be able to run serial simulation code parallel
• Minimize user effort
• No parallel computing or batch job knowledge
  needed on user side

9
PSoNI, A Software Environment for Parallelizing
Neutron Instrument Simulations (cont.)

• What is PSoNI?
• PSoNI is designed for parallelizing instrument
  simulations at SNS.
• Could be extended to other uses..
• It consists of two packages,
• - PSoNI_ms for Massive Simulations.
• - PSoNI_ps for Parametric Simulations,
• MPI (Message Passing Interface) parallelization,
  and Perl scripts for automated data management
  and batch job submission.
• The output data can be visualized by means of an
  independent viz package.

10
PSoNI, A Software Environment for Parallelizing
Neutron Instrument Simulations (cont.)

• Current status
• What have been done?
• PSoNI has been developed on TG resources at
  ORNL.
• PSoNI_ms and _ps have been tested with McStas
  simulations of SEQUOIA and HYSPEC respectively.
• PSoNI_ms has been deployed onto a couple of
  remote TG super-computers and successfully tested
  with McStas simulation of SEQUOIA.
• PSoNI_ms is used by and _ps is ready for
  friendly-users to parallelize their serial McStas
  simulations.
• What to be done?
• PSoNI is designed to parallelize serial
  instrumentation simulations based on various
  simulation packages.
• The current version of PSoNI only hooked with
  McStas.
• Supporting of other packages (UNIX version) can
  be provided upon request.
• The rest of my presentation will be based on
  the current version of PSoNI.
• (The user interfaces will be basically the same
  for different simulation packages.)

11
PSoNI, A Software Environment for Parallelizing
Neutron Instrument Simulations (cont.)

• Performance
• PSoNI_ms run of SEQUOIA simulation with108
  neutrons took 3 min 21 sec on 21 processesr (3
  hours on personal workstation.)
• A PSoNI_ms test run of 1010 neutrons takes 5
  hours. (Not attempted on personal workstation.)
• A Campaign of higher resolution runs was used to
  assist in a late design optimization of an
  Instrument.
• Test results PSoNI_ps automatically submits
  HYSPEC parametric simulation jobs almost
  simultaneously. These jobs run parallel to
  significantly reduce the time of parametric
  simulations.

12
PSoNI_ms package

• 1. Who can use PSoNI_ms?
• As long as you
• - have your McStas serial simulation code ready
  
• - and provide event based output data (a format
  has been defined for SEQUOIA),
• you can use PSoNI_ms to parallelize your
  simulation.
• How to use PSoNI_ms?
• a). Setting environment
• - Log on and copy sample .tcshrc file into your
  home directory
• - Type tcsh.
• b). Creating serial simulation executable file
• - Prepare your simulation code as same as on
  your workstation. (If using existing PC code,
  some migration efforts may be needed. lt Help is
  available ?.)
• - Type psoni-mclink my-instr-file-name
• c). To run your serial simulation parallel
• - Copy and modify two configuration sample files
  (5 lines each).
• - Type psoni-m-qsub path-of-myPBSsub.conf
  
• (for direct PBS submission)

Can you make my life easy?
13
An example of running PSoNI_ms
Anticipates Portal Integration
1. Copy/modify the following configuration file
File 1 mpiRun.conf
baseDir/users/myHomeDir/SEQUOIA exeFileFermi_T
0 totalNeutrons1e8 parametersFermi_nu480
T0_nu60 nrad0.5801 nchans26 Edes60
T0_off0.0out FileType.event .tdat
File 2 autoPBSsub.conf
Oh, a piece of cake!
baseDir/users/myHomeDir/SEQUOIA
jobNamesequoia-2.0 wallTime001000 numCPUs
21 emailAddrmyUID_at_ornl.gov
2. Type psoni-m-qsub (The default
config file ./ autoPBSsub.conf is used here)
14
PSoNI_ps package
1. Who can use PSoNI_ps? As long as you -
have your serial McStas simulation code ready,
you can use PSoNI_ps to parallelize your
parametric simulations. 2. How to use PSoNI_ps?
a). Setting environment - Log on and copy
sample .tcshrc file into your home directory -
Type tcsh. b). Creating serial simulation
executable file - Prepare your simulation code
as same as on a PC. (If using existing PC
code, some migration efforts may be needed. lt
Help is available ?.) - Type psoni-mclink
my-instr-file-name c). To run your parametric
simulations parallel - Write your userScript
(see next page for an example) - Type
psoni-p-pbs config-file-path, if not default
15
An Example of PSoNI_ps configuration file

• The definition of each line in the configuration
  file
• Path of input data file
• Base directory the directory contains your
  executable
• Executable file of serial simulation
• Level1 directory name of output data, which will
  be located under your Base dir
• subdirectory name1, which will be located under
  Level1 directory
• running command of your executable file
• subdirectory name2, which will be located under
  Level1 directory
• ..

/users/myDir/HYSPEC_test/source_sct21a_td_05_1.dat
/users/myDir/HYSPEC_test_paral fcs01_mono Zyb7e
e_3p6meV ../../fcs01_mono -n 1e7
Edes3.6 E_min3.24 E_max3.96 toffset7.35e-5
eta_h48. eta_v48. wid_h0.3
wid_v0.204 nos_h1.0 nos_v1.0 ih0.0 iv0.0
e_5meV ../../fcs01_mono -n 1e7 Edes5.0
E_min4.5 E_max5.5 toffset6.32e-5 eta_h48.
eta_v48. wid_h0.3 wid_v0.204
nos_h1.0 nos_v1.0 ih0.0 iv0.0
e_xxmeV ..
16
(No Transcript)
17
Backup
18
Scientific user support

• A full user support package is available for you
• Setting up working environment on TG machines
• Migrating simulation code from PC
• Compiling and linking simulation code
• Running PSoNI
• User support contact information
• chenml_at_ornl.gov

19

Any suggestions/desires/questions regarding fast
instrument simulation?