GTC Tiger Team Update

1
GTC Tiger Team Update

• Shirley Moore
• shirley_at_eecs.utk.edu
• SC08 PERI Meeting
• November 17, 2008

2
GTC Status on Jaguar

• Before code stopped working because of seg fault
  in ADIOS routine, verified that load balance and
  runtime improved with Gabriel Marins correction
  to particle initialization (see next slide)
• Scott Klaskys version of gtc_yong with new
  version of ADIOS builds and runs with both MPI
  only and hybrid MPI/OpenMP
• MPI only version builds and runs instrumented
  with TAU
• Hybrid MPI/OpenMP version instrumented with most
  recent TAU builds but crashes with segmentation
  fault (with TAU_THROTTLE set to 0)

3
TAU Time Profiles of GTC with Different Particle
Initializations Showing Load Imbalances
Corrected particle initialization results in less
severe load imbalance
Load imbalance due to incorrect particle
initialization
128 process runs on Jaguar
Profiling helps ensure that a valid version is
used for modeling.
4
GTC Status on Intrepid

• Version from Stephane Ethier built and ran but he
  wanted it optimized further for BGP before we
  made measurements.
• Stephane also provided small and medium problem
  input sets (same kinetic electron problem we have
  been running on Jaguar) with instructions for
  scaling)
• I optimized the most time-consuming OpenMP loops
  using MASS vector routines (as suggested by
  Stephane) and got 15-20 improvement in runtime
  (show performance data)
• Current status
• Optimized version builds and runs without and
  with TAU on small size problem
• Optimized version runs without TAU but crashes
  with TAU on medium size problem
• GTC memory use on Intrepid is very close to the
  limit and even the uninstrumented code fails
  occasionally
• Random number generator causes different size
  arrays on different runs

5
GTS Status

• Files in /tmp/work on Jaguar for ADIOS version
  obtained from Scott Klasky wiped out
• Stephane Ethier has agreed to package up a
  non-ADIOS version that he uses on Jaguar
• Stephane has not yet ported GTS to BG/P
• Because GTS uses double precision, it uses much
  more memory than GTC and this will likely be a
  problem for the port.

6
What Next?

• Currently a Tiger Team of one, need help!!!
• GTC code is fragile
• Plan
• Get another copy of GTS
• Determine exactly what runs are needed
• Set a 2-week period when TAU team and other
  personnel are available to help (preferably
  GTC/GTS developers, but can we get that
  commitment?)
• Push the runs through the queues and check that
  the data collected are sufficient for modeling
  purposes