CGAM Running the Met Office Unified Model on HPCx

1
CGAMRunning the Met Office Unified Model on HPCx

• Paul Burton
• CGAM, University of Reading
• Paul_at_met.rdg.ac.uk
• www.cgam.nerc.ac.uk/paul

2
Overview

• CGAM Who, what, why and how
• The Met Office Unified Model
• Ensemble Climate Models
• High Resolution Climate Models
• Unified Model Performance
• Future Challenges and Directions

3
Who is CGAM?
4
What does CGAM do?

• Climate Science
• UK Centre of expertise for climate science
• Lead UK research in climate science
• Understand and simulate the highly non-linear
  dynamics and feedbacks of the climate system
• Earth System Modelling
• From seasonal to 100s of years
• Close links to Met Office
• Computational Science
• Support scientists using Unified Model
• Porting and optimisation
• Development of new tools

5
Why does CGAM exist?

• Will there be an El Nino this year?
• How severe will it be?
• Are we seeing increases in extreme weather events
  in the UK?
• 2000 Autumn floods
• Drought?
• Will the milder winters of the last decade
  continue?
• Can we reproduce and understand past abrupt
  changes in climate?

6
How does CGAM answer such questions?

• Models are our laboratory
• Investigate predictability
• Explore forcings and feedbacks
• Test hypothesis

7
Met Office Unified Model

• Standardise on using a single model
• Met Offices Hadley Centre recognised as world
  leader in climate research
• Two way collaboration with the Met Office
• Very flexible model
• Forecast
• Climate
• Global or Limited Area
• Coupled ocean model
• Easy configuration via a GUI
• User configurable diagnostic output

8
Unified Model Technical Details

• Climate configuration uses old vn4.5
• Vn5 has an updated dynamical core
• Next generation HadGEM climate configuration
  will use this
• Grid-point model
• Regular latitude/longitude grid
• Dynamics
• Split-explicit finite-difference scheme
• Diffusion and polar filtering
• Physical Parameterisation
• Almost all constrained to a vertical column

9
Unified Model Parallelisation

• Domain decomposition
• Atmosphere 2D regular decomposition
• Ocean 1D (latitude) decomposition
• GCOM library for communications
• Interface to selectable communications library
  MPI, SHMEM, ???
• Basic communication primitives
• Specialised communications for UM
• Communication Patterns
• Halo update (SWAPBOUNDS)
• Gather/scatter
• Global/partial summations
• Designed/optimised for Cray T3E!

10
Model Configurations

• Currently
• HadAM3 / HadCM3
• Low resolution (270km 96 x 73 x 19L)
• Running on 10-40 CPUs
• Turing (T3E1200), Green (O3800), Beowulf cluster
• Over the next year
• More of the same
• Ensembles
• Low resolution (HadAM3/HadCM3)
• 10-100 members
• High resolution
• 90km 288 x 217 x 30L
• 60km 432 x 325 x 40L

11
Ensemble Methods in Weather Forecasting

• Have been used operationally for many years (is.
  ECMWF)
• Perturbed starting conditions
• Reduced resolution
• Multi-model ensembles
• Perturbed starting conditions
• Different models
• Why are they used?
• Give some indication of predictability
• Allows objective assessment of weather-related
  risks
• More chance of seeing extreme events

12
(No Transcript)
13
Climate Ensembles

• Predictability
• What confidence do we have in climate change?
• What effect do different forcings have?
• CO2 different scenarios
• Volcano erruptions
• Deforestation
• How sensitive is the model
• Twiddle the knobs and see what happens
• How likely are extreme events?
• Allows governments to take defensive action now

14
Ensembles Implementation

• Setup
• Allow users to specify and design an ensemble
  exeperiment
• Runtime
• Allow the ensemble to run as a single job on the
  machine for easy management
• Analysis
• How to view and process vast amounts of data
  produced

15
Setup Normal UM workflow
16
Setup UM Ensemble workflow
17
UM Ensemble Runtime (1)

• poe called at top level calls a
  top_level_script
• Works out which CPU its on
• Hence which member it is
• Hence which directory/model SCRIPT to run
• Model scripts run in a separate directory for
  each member
• Each model script calls the executable

18
UM Ensemble Run time (2)

• Uses MPH to change the global communicator
• http//www.nersc.gov/research/SCG/acpi/MPH/
• Freely available tool from NERSC
• MPH designed for running coupled multi-model
  experiments
• Each member has a unique MPI communicator
  replacing the global communicator

19
UM Ensemble Future Work

• Run time tools
• Control and monitoring of ensemble members
• Real-time production of diagnostics
• Currently each member writes its own diagnostics
  files
• Lots of disk space
• I/O performance?
• Have a dedicated diagnostics process
• Only output statistical analysis

20
UK-HIGEM

• National Grand Challenge Programme for High
  Resolution Modelling of the Global Environment
• Collaboration between a number of academic groups
  and the Met Offices Hadley Centre
• Develop high resolution version of HadGEM ( 10
  atmosphere, 1/30 ocean)
• Better understanding and prediction of
• Extreme events
• Predictability
• Feedbacks and interactions
• Climate surprises
• Regional Impacts of climate change

21
UK HiGEM Status

• Project only just starting
• Plan to use Earth Simulator for production runs
• Preliminary runs carried out
• Earth Simulator
• Very encouraging results
• HPCx is a useful platform
• For development
• Possibly for some production runs

22
UM Performance

• Two configurations
• Low resolution 96x73x19L
• High resolution 288x217x30L
• Built in comprehensive timer diagnostics
• Wallclock time
• Communications
• Not yet implemented
• I/O, memory, hardware counters, ???
• Outputs an XML file
• Analysed using PHP web page

23
LowRes Scalability
24
LowRes Communication Time
25
LowRes Load Imbalance
26
LowRes Relative Costs
27
HiRes Scalability
28
HiRes Communication Time
29
HiRes Load Imbalance
30
HiRes Relative Costs
31
HiRes Exclusive Timer

• QT_POS has large Collective time
• Unexpected!
• Call to global_MAX routine in gather/scatter
• Not needed, so deleted!

32
HiRes After optimisation

• QT_POS reduced from 65s to 35s
• Improved scalability
• And repeat

33
Optimisation Strategy

• Low Res
• Aiming for 8 CPU runs as ensemble members
  (typically 50 members)
• Physics optimisation a priority
• Load Imbalance (SW radiation)
• Single processor optimisation
• Hi Res
• As many CPUs as is feasible
• Dynamics optimisation a priority
• Remove/optimise collective operations
• Increase average message length

34
Future Challenges

• Diagnostics and I/O
• UM does huge amounts of diagnostic I/O in a
  typical climate run
• All I/O through a single processor
• Cost of gather
• Non-parallel I/O
• Ocean models
• Only 1D decomposition, so limited scalability
• T3E optimised!
• Next generation UM5.x
• Much more expensive
• Better parallelisation for dynamics scheme