Climate modeling: Coupling Component Models by MPH for Distributed Multi-Component Environment

1
Climate modeling Coupling Component Models by
MPH for Distributed Multi-Component Environment

• Chris Ding and Yun (Helen) He
• NERSC Division
• Lawrence Berkeley National Laboratory

2
(No Transcript)
3
Motivation

• Application problems grow in scale complexity
• Effective organization of simulation software
  system ? a major issue
• Software lasts much longer than a computer!

4
Multi-Component Approach

• Build from (semi-)independent programs
• Coupled Climate System Atmosphere Ocean
  Sea-Ice Land-Surface Flux-Coupler
• Components developed by different groups at
  different institutions
• Maximum flexibility and independence
• Algorithm, implementation depends on individual
  groups, practicality, time-to-completion, etc.
• Components communicate through well-defined
  interface data structure.
• Software industry trend (CORBA, DCE, DCOM)
• Common Component Architecture (DOE project)
• ESMF

5
Distributed Components on HPC Systems

• Use MPI for high performance
• MPH establish a multi-component environment
• MPI Communicator for each components
• Component name registration
• Resource allocation for each component
• Support different job execution modes
• Stand-out / stand-in redirect
• Complete flexibility
• Similar to PVM, but much smaller, simpler,
  scalable, and support more execution /
  integration modes.

6
A climate simulation system consists of
many independently-developed componentson
distributed memory multi-processor computer

• Single-component executable
• Each component is a stand-alone executable
• Multi-component executable
• Several components compiled into an executable
• Multi-component system execution modes
• Single-Component executable Multi-Executable
  system (SCME)
• Multi-Component executable Single-Executable
  system (MCSE)
• Multi-Component executable Multi-Executable
  system (MCME)

7
Component Integration / Job Execution Modes

• Single-Component exec. Multi-Executable system
  (SCME)
• Each component is an independent executable image
• Components run on separate subsets of SMP nodes
• Max flexibility in language, data structures,
  etc.
• Industry standard approach
• Multi-Component exec. Single-Executable system
  (MCSE)
• Each component is a module
• All components compiled into a single executable
• Many issues name conflict, static allocations,
  etc.
• Stand-alone component
• Easy to understand and coordinate

8
Component Integration / Job Execution Modes

• Multi-Component exec. Multi-executable system
  (MCME)
• Several components compiled into one executable
• Multiple executables form a single system
• Different executables run on different processors
• Different components within same executable could
  run on separate subsets of processors
• Maximum flexibility
• Includes SCME and MCSE as special cases
• Easy to adopt for concurrent ensemble simulations

9
Multi-Component Single-Executable (MCSE)
master.F PCM call MPH_setup_MCSE (
atmosphere, ! atmosphere registered
ocean, ! ocean registered
coupler, ! coupler registered )
! Add more components if
(PE_in_component (ocean, comm)) call ocean_v1
(comm) if (PE_in_component (atmosphere,
comm)) call atmosphere (comm) if
(PE_in_component (coupler, comm)) call
coupler_v2 (comm) PROCESSOR_MAP
atmosphere 0 7 ocean
8 13 coupler 14 15
10
Single-Component Multi-Executable (SCME)
Coupled System Atmosphere Ocean
Flux-Coupler atm.F call MPH_setup
(atmosphere, atmosphere_World) ocean.F call
MPH_setup (ocean, ocean_World) coupler.F
call MPH_setup (coupler, coupler_World)
Component Registration File atmosphere
ocean
coupler
11
MPH3 Multi-Component Multi-Executable (MCME)
mpi_exe_world MPH_components (name1ocean,
name2ice,) Component Registration File
BEGIN coupler Multi_Comp_Start
2 ocean 0 3 ice
4 10 Multi_Comp_End
Multi_Comp_Start 3 atmosphere
0 10 land 11 13
chemistry 14 25
Multi_Comp_End END Launch parallel job on
IBM SP _at_ task_geometry (5,2)(1,3)(4,6,0)

12
Joining two components

• MPH_comm_join (atmosphere, ocean, comm_new)
  comm_new contains all nodes in atmosphere,
  ocean. atmosphere nodes rank 07, ocean
  nodes rank 811
• MPH_comm_join (ocean, atmosphere, comm_new)
  ocean nodes rank 03, atmosphere nodes
  rank 411
• Afterwards, data remapping with comm_new

Direct communication between two components

• MPI_send (, MPH_global_id (ocean, 3), )
• for query / probe tasks

13
MPH Inquiry Functions

• MPH_global_id()
• MPH_local_id()
• MPH_component_name()
• MPH_total_components()

14
Status and Users

• Completed MPH1, MPH2, MPH3
• Software available free online
  http//www.nersc.gov/research/SCG/acpi
• Complete user manual
• MPH runs on
• IBM SP
• SGI Origin
• HP Compaq clusters
• PC Linux clusters
• MPH users
• NCAR CCSM
• CSU Icosahedra Grid Coupled Climate Model
• People expressed clear interests in using MPH
• SGI/NASA Irene Carpenter / Jim Laft, on SGI for
  coupled models
• UK ECMWF, for ensemble simulations
• Johannes Diemer, Germany, for coupled model on HP
  clusters

15
Summary

• Multi-Component Approach for large complex
  application software
• MPH glues together distributed components
• Name registration, resource allocation,
  standard-out
• Single-Component Multi-Executable (SCME)
• Multi-Component Single-Executable (MCSE)
• Multi-Component Multi-Executable (MCME)