Parallel MM5 Simulation

1
Parallel MM5 Simulation ResultsforCentral
America
2
Objectives

• Parallel Programming Concepts 
• Set up MM5 for long simulations using parallel
  architecture
• Initialize MM5 using GCM output
• Overcoming challenges in running long simulations
• Discussion of results
• Use Web based tool to order data (sub divide data
  into regions)

3
What is Parallel Computing?

• parallel computing is the simultaneous use of
  multiple compute resources to solve a
  computational problem
• The compute resources can include
• - A single computer with multiple processors
• - An arbitrary number of computers connected by
  a network
• - A combination of both.

4
Why use Parallel Computing ?

• Save time - wall clock time
• Solve larger problems
• Taking advantage of non-local resources
• Cost savings
• Overcoming memory constraints

5
General Parallel Terminology

• What is a task ?
• -A logically discrete section of computational
  work
• What is a Parallel task ?
• -A task that can be executed by multiple
  processors safely (yields correct results)

6
General Parallel Terminology Continued

• Communications
• -Parallel tasks need to exchange data through
  shared memory bus or network
• Synchronization
• -Coordination of Parallel tasks associated with
  communications in real time

7
Parallel Overhead

• Task start-up time
• Synchronizations
• Data communications
• Software overhead imposed by parallel compilers,
  libraries, tools, operating system, etc.
• Task termination time

8
Parallel Computer Memory Architecture

• Shared Memory

9
Shared Memory
Advantages
Fast and Easy
Disadvantages
Scalability and Cost
10
Parallel Computer Memory Architecture

• Distributed Memory

11
Parallel Computer Memory Architecture

• Advantages
• -Memory is Scalable with number of processors
• -Cost Effectiveness
• Disadvantages
• -Responsibility for data communication between
  processors
• -Difficult to convert programs based on global
  memory to this memory organization

12
Parallel Computer Memory Architecture

• Hybrid Distributed-Shared Memory

13
Parallel Programming Models

• In the threads model of parallel programming, a
  single process can have multiple, concurrent
  execution paths

14
Parallel Programming Model

• Message Passing Model Implementation
• -Subroutines imbedded in the source code
• -MPI Message Passing Interface

15
Parallel Programming Models

• Thread Model Implementation
• -- Subroutines called from within Parallel source
  code
• -- Compiler directives in either serial or
  parallel code
• --OpenMP or POSIX Threads

16
Parallel Programming Models

• Message Passing Model

17
Cluster Components

• Parallel Applications
• Linux or Windows
• Fast/GigE Infiniband Myrinet
• Intel/AMD Processors

18
Cluster Software

• Message Passing Interface
• Easy to use interface for installation and
  Management
• Batch/queue Software (PBS)

19
Beowulf Cluster

• Massively Parallel Computer
• Runs a free operating system
• Connected by high speed interconnect
• Compute nodes are dedicated
• Everything in a Beowulf is open-source and open
  standard- easier to manage/upgrade

20
Linux Distributions

• CentOS - RHEL compatible
• Fedora - Red Hats free version
• Owl - Openwall GNU/Linux (security
  enhancements)
• Redhat - Supported(RHEL)
• SuSe - Novell
• Lindows - Runs Linux and MS/Windows software

21
Beowulf Interconnect

• The most important component
• Factors to consider
• Bandwidth
• Latency
• - Price
• Software Support
• Ethernet (Inexpensive,reasonably fast)
• Myrinet (low latency and high bandwidth-popular)

22
Cheetah at ORNL
23
Cheetahs Configuration

• IBM SP4 Supercomputer
• Each of 24 nodes has 32 processors
• Provides 4 Teraflops of computing power
• More than a terabyte of Memory
• 40 Terabytes of disk space
• Connected to the Mass Storage System by HPSS

24
Parallel MM5

• Runtime System Library (RSL)
• -Provides inter-domain communication
• -Irregular domain decomposition
• -Distributed I/O
• -Dynamic load balancing

25
Parallelism in MM5

• Runs on shared-memory architecture
• Runs on distributed memory architecture

26
Building MM5 on Parallel Architecture

• Download
• ftp//ftp.ucar.edu/mesouser/MM5V3/MM5.TAR.gz
• ftp//ftp.ucar.edu/mesouser/MM5V3/MPP.TAR.gz
• Unzip and untar
• gzip d c MM5.TAR.gz tar xf
• cd MM5
• gzip d c MPP.TAR.gz tar xf -

27
Building MM5 on Parallel Architecture

• Edit configure.user file for computer and
  configuration
• Find the subsection in section 7 to define your
  computer architecture
• Adjust PROCMIN_NS and PROCMIN_EW settings at top
  of section 7 for memory scaling
• Build the model make mpp
• Executable Run/mm5.mpp

28
Building MM5 on Parallel Architecture

• To remake the code in different configuration
• make mpclean
• To reinstall the code in different location
• make uninstall

29
MM5 Input/Output

• Overcoming file size limits of 2GB on LINUX
• -If you anticipate that your MM5 model output
  will be larger than 2GB (the file size limit on
  LINUX machines), add the following to the
  compiler loading flags
• LDFLAGS -L/usr/pgi/linux86/liblf

30
NCAR CCSM3.0

• IPCC runs made at T85 resolution (about 140 km in
  latitude and longitude)
• Output saved 4X daily
• Fortran program to convert to pregrid output
  format

31
CCSM3 Soil Levels

• Variable name levsoi in (m)
• levsoi10
• Soil levels at
• (0.007100635, 0.027925, 0.06225858, 0.1188651,
  0.2121934, 0.3660658, 0.6197585, 1.038027,
  1.727635, 2.864607)
• Soil Temperature and Soil water written at
  4,6,8,9 to match mm5 soil moisture levels
  10,40,100,20

32
CCSM3 Output

• Atmospheric hybrid sigma pressure coordinate
  lev26
• -lev
• 3.54463800000001, 7.38881350000001, 13.967214,
  23.944625, 37.2302900000001, 53.1146050000002,
  70.0591500000003, 85.4391150000003, 100.514695,
  118.250335, 139.115395, 163.66207, 192.539935,
  226.513265, 266.481155, 313.501265000001,
  368.817980000002, 433.895225000001,
  510.455255000002, 600.524200000003,
  696.796290000003, 787.702060000003,
  867.160760000001, 929.648875000002,
  970.554830000001, 992.5561

33
CCSM3 Output

• atm/ cpl/ ice/ lnd/ ocn/
• 411582540 6 2.4 Gb (approx.) (atm)
• b30.042e.cam2.h3.2005-06-05-21600.nc
• b30.042e.cam2.h3.2005-06-10-21600.nc
• b30.042e.cam2.h3.2005-06-15-21600.nc
• b30.042e.cam2.h3.2005-06-20-21600.nc
• b30.042e.cam2.h3.2005-06-25-21600.nc
• b30.042e.cam2.h3.2005-06-30-21600.nc
• 25169936 (lnd)
• b30.042e.clm2.h0.2005-06.nc

34
CCSM3 Output

• 'Skin Temperature (TS) in K'
• 'Source model terrain height m'
• '2-m Temperature (TREFHT) in K'
• 'Sea Level Pressure (PSL)'
• 'Surface zonal wind (usfc)'
• 'Surface meridional wind (vsfc)'
• 'Surface relative humidity ()'
• 'Total Precip (mm/day)'

35
CCSM3 Output

• 'Water equivalent snow depth (mm)'
• 'Upper Air Temperature (T)
• 'Upper Air Zonal Wind (U)'
• 'Upper Air Meridional Wind (V)'
• 'Upper Air Temperature (T) '
• 'Geopotential Height'
• 'volumetric soil water (mm3/mm3)
• 'Soil temperature (K)'

36
MM5 Input Files

• Four years of CCSM3 input chosen
  (2005,2010,2015,2025)
• REGRID and INTERPF Programs run to create initial
  and lower boundary files

37
MM5 Runtime Options

• FDDAGD0 no 4DDA
• FDDAOB0 no obs 4DDA
• MAXNES2 no. of domains
• MIX180 no. of grid points in I dir
• MJX255 no. of grid points in J dir.
• MKX 23 no. of grid points in K dir

38
MM5 Runtime Options

• Atmospheric Radiation scheme option
• IFRAD2 Cloud radiation scheme
• ISOIL2 Noah land-surface scheme
• ISHALLO0 no shallow convective scheme

39
MM5 Runtime Options

• Physics Option
• IMPHYS 4 Simple Dudhia
• MPHYSTBL 0 not using look-up table
• ICUPA 3 Grell cumulus parameterization
• IBLTYP 5 MRF PBL

40
Mmlif Options

• FORECAST TIME AND TIME STEP TIMAX 43200.,
• forecast length in minutesTISTEP 60.,
• coarse domain DT in model, use 3DX

41
Mmlif Options

• Default soil layers expected as input for ISOIL
  2 3 These values reflect the BOTTOM of the
  soil layer availableISTLYR 10,40,100,200,ISMLY
  R 10,40,100,200,

42
Mmlif options

• LEVIDN  0,   1,   2,  1,  1,  1,  1,  1,  1,  
  1,   level of nest for each domainNUMNC
   1,   1,   2,  1,  1,  1,  1,  1,  1,  1,   ID
  of mother domain for each nestNESTIX 165, 178,
  136, 181, 211, 221 domain size iNESTJX ,
  250, 253, 181, 196, 211, 221  domain size
  jNESTI    1, 56, 28, 35, 45, 50 
  start location iNESTJ    1, 86, 25, 65,
  55, 50start location iXSTNES 0.,  0., 900.,
   0., 0.,  0.,  0.,  0.,  0.,  0., domain
  initiationXENNES 99999,99999,99999.,720.,720.,72
  0.,720.,720.,720.,720. domain terminationIOVERW
     2, 2, 0, 0, 0, 0, 0, 0, 0, 0, overwrite
  nest input                  0interpolate from
  coarse mesh (for nest domains)                 
  1read in domain initial conditions             
       2read in nest terrain fileIACTIV 1, 1,
  0, 0, 0, 0, 0, 0, 0, 0, in case of restart
  is this domain active?

43
MM5 Input/Output
44
MM5 Input/Output
45
MM5 FAQ

• Can I get CCSM3.0 output to initialize MM5 ?
• Only By Permission

46
MM5 FAQ

• Where can I get a copy of the most recent version
  of MM5 source code
• -- NCAR anonymous ftp (ftp.ucar.edu)
• /mesouser/MM5V3/TERRAIN.TAR.gz
• /mesouser/MM5V3/REGRID.TAR.gz
• /mesouser/MM5V3/INTERPF.TAR.gz
• /mesouser/MM5V3/MM5.TAR.gz
• /mesouser/MM5V3/MPP.TAR.gz

47
MM5 FAQ

• Is it possible to run all MM5 modeling system
  programs on a Linux Machine ?
• Yes, it is possible. Compiler options provided
  by Portland Group Inc. is provided.

48
MM5 FAQ

• How long does it take to run a simulation with
  nested domain at 12 km and 36 km ?
• It takes about 36 hours of wall clock time on
  one node with 32 processors on an IBM sp2.

49
MM5 FAQ

• What are the fields available in the MM5 output
  file on the Web ?
• Output for the runs are stored on the archival
  system at Oak Ridge National Laboratory. Few
  fields are extracted and gif images of those
  fields are available on the SERVIR web page.

50
MM5 FAQ

• What is the URL of the Web site for viewing a few
  extracted variables from the simulation ?
• http//servir.nsstc.nasa.gov/climate_change/scenar
  ios.html

51
MM5

• How do I visualize the output of MM5?
• -There is a utility program called mm5tograds
  (MM5toGrADS.tar.gz)
• -There is also a program which will convert
  the output to vis5d (tovis5d.tar.gz) available
  from MM5 home

52
MM5 Output fields 2d

• ps pstar (Pa)
• tg ground temp (K)
• rc accum conv pcn (cm)
• rn accum non-c pcn (cm)
• ter ter elevation (m)
• xmf cross map factor
• dmf dot map factor
• cor coriolis (s-1)
• tr reservoir temp (K)
• xlat cross lat (degree)
• xlon cross lon (degree)
• lu land use
• sc snow cover
• pblh pbl height (m)
• pblr pbl regime
• shf sen heat flux (W/m2)

53
MM5 Output fields 2d

• lhf lat heat flux (W/m2)
• ust friction vel (m/s)
• swd down sw rad (W/m2)
• lwd down lw rad (W/m2)
• swo out sw rad (W/m2)
• lwo out lw rad (W/m2)
• sst sea sfc temp
• st1 soil temp 1 (K)
• st2 soil temp 2 (K)
• st3 soil temp 3 (K)
• st4 soil temp 4 (K)
• sm1 soil moisture 1 (m3/
• sm2 soil moisture 2 (m3/
• sm3 soil moisture 3 (m3/
• sm4 soil moisture 4 (m3/
• can canopy moisture (m)
• ssi seaice

54
MM5 Output Fields 2d

• sro surface runoff (mm)
• uro underground runoff (
• t2m 2 m temperature (K)
• q2m 2 m mixing ratio (kg
• u10 10 m u wind (m/sec)
• v10 10 m v wind (m/sec)
• pslv sea level prs (mb)
• cref composite refl (dbz)
• iclw integrat clw (cm)
• irnw integrat rnw (cm)
• pwat precipit water (cm)
• clfrlo low cloud fraction
• clfrmi mid cloud fraction
• clfrhi high cloud fraction

55
MM5 Output fields 3d

• u u wind (m/s)
• v v wind (m/s)
• w vertical vel (m/s)
• pp prs pert (Pa)
• t temperature (C)
• q mixing ratio (kg/kg)
• clw cloud water (kg/kg)
• rnw rain water (kg/kg)
• rtnd rad tend (K/day)
• z height AGL (m)
• h geopot height (m)
• td dewpoint temp (C)

56
MM5 Output fields 3d

• rh rel humidity ()
• th potential temp (K)
• the theta-e (K)
• prs pressure (Pa)
• vor vorticity (s-1)
• pv potential vort (pvu)
• dbz reflectivity (dbz)
• div divergence (s-1)
• dir dir (degrees)
• tadv temp advection (K/s)

57
DEMO

• Look at some extracted variables from the runs
• Vis5D demo
• Vis5D version 5.2 is available at
  ftp//ftp.ssec.wisc.edu/pub/vis5d-5.2.