L arge

1
L arge A tmospheric C omputation on the E arth
S imulator
RPN Michel Desgagné, Gilbert Brunet,
Michel Valin
ESC Wataru Ohfuchi
with MC2
McGill U. Peter Yau, John Gyakum,
Yosvany Martinez U. of Tokyo Hiroshi
Niino, Yuki Fukurawa U.
of Albany Ron McTaggart-Cowan Other Claude
Girard, Pierre Pellerin Robert Benoit, Mike
Montgomrey
2
EARTH SIMULATOR CENTER in YOKOHAMA JAPAN
In Operation Since March 2002
3
Sustained performances 36 TF on some benchmarks
(640 nodes) 27 TF with AFES (640 nodes) (10
km global climate simulation) 13 TF with MC2
(495 nodes) (1 km 11000x8640x51 grid)
Some Specs of the Earth Simulator - 640 nodes
of 8 vector processors - Processor peaks at 8
Gf/sec. - Shared memory/node 16
GB ----------------------------------------------
Whole system - 5120 processors - Peak 40
Tf/sec - Memory 10 TB - Interconnect 12.3
GB/sec x 2
4
LACES A Grand Challenge project on the Earth
Simulator
964 hPa
ET Phase
985 hPa
Modelling the Full Lifecycle of Hurricane Earl
(Sept 1998) at 1km Resolution with the Canadian
MC2 Model
Tropical Phase Class2 Hurricane
September 1998 Classified as a very active TC
period
5
COMPARE III Workshop Tokyo, Japan December
13-15, 1999
Typhoon FLO - Septembre 1990 2km 16-30 H Forecast
of Relative Vorticity at 20m
frame every hour
max 700-800 e-5 sec-1
6
Grid Strategy for EARL on the ES 3 Stages
Rotated Mercator grids
50 km 400 x 332 x 31 timestep - 240s, KFC
10 km 1600 x 1260 x 41 - timestep 45s, KFC-YAU
1 km 11007 x 8647 x 51 - timestep 6s, YAU
Memory 7 Tb CPUs 22 x 180 (3960) Steps 104400
X 6 sec. Wall clock 7-8 days
Outer 50 km grid Initial and BCs from
CMC analysis
7
Time strategy for the Simulation ofthe Full
Lifecycle of Hurricane EARL on the ES
31/08-18Z
04/09-00Z
Total of 174 Hours Forecast at 1 km minus 2x6
Hours overlap (6.75 days)
12Z
50 km
15Z
10 km
1 km - 78H Tropical phase
03/09-18Z
05/09-12Z
50 km
12Z
10 km
15Z
1 km - 42H
07/09-12Z
05/09-06Z
50 km
00Z
10 km
03Z
1 km - 54H ET phase
8
Main Challenges

• Elliptic iterative DM pressure solver (4.8 giga
  equations at once)
• Parallel I/O software
• Model output
• Restart files
• Startup of order 4000 MPI processes on a vector
  architecture
• Predictibility issues at 1 km reso. on a very
  large domain

9
(No Transcript)
10
(No Transcript)
11
1 km output
Full domain 8000 x 6000
Reduced domain 3000 x 2000
Full Domain Averaged 4 DX
Full Domain Averaged 10 DX
3D U,V,W,T,P,HU,QN,QP,QI,QG 3D U,V,W,T,P,HU,QN,QP,QI,QG
2D QR, PN, RT, PR, FC, FV
3D U,V,W,T,P,HU,QN,QP,QI,QG
10 min
15 min
30 min
8 x 64 files 30 GBytes
24 x 12 files 151 GBytes
4 x 64 files 29 GBytes
4 x 64 files 14 GBytes
Grand total 1300 files, 225 GBytes/4H ? 4.2 TB
for first 78H
All data compressed to 16 bits
12
(No Transcript)
13
(No Transcript)
14
(No Transcript)
15
(No Transcript)
16
Techical I/O - an Important Issue

• Preprocessor MC2NTR gone
• All computation perform within the DM main
  program
• Global distribute/collect removed

Block partitioning the PEs
Scaling up with a subdomain size of 500 x 50
on vector processor system
17
THE END
Thank You !
18
30 August to 9 September 1998
Mean Sea Level Pressure
Bonny
Danielle
Earl
19
50km and 10 km output
10 Km 1593 x 1253 x 41
50 Km 393 x 325 x 31
2D QR, PN, RT, PR
2D QR, PN, RT, PR
2D physics ?10 km 3D U,V,W,T,P,HU,QN 2D physics ?1 km 3D U,V,W,T,P,HU,QN
10 min
15 min
60 min
Total 12 GBytes For 192 H forecast
Total 218 GBytes For 183 H forecast
All data compressed to 16 bits
20
12 UTC 0109
00 UTC 0209
Bonny
Danielle 1000
Danielle 1004
Earl 998
Earl 999
12 UTC 0209
00 UTC 0309
Danielle 994
Danielle 999
Earl 985
Earl 994
CMC MSL Analysis
21
12 UTC 0309
00 UTC 0409
Danielle 994
Danielle 994
Earl 994
Earl 990
00 UTC 0509
12 UTC 0409
Earl 1000
Earl 998
CMC MSL Analysis
22
12 UTC 0509
00 UTC 0609
Earl 979
Earl 990
00 UTC 0709
12 UTC 0609
Earl 968
Earl 964
CMC MSL Analysis
23
45H Forecast valid 02 Sept 12 UTC
Tropical phase MSL Forecast on 10 km grid
Produced on IBM Power4
978
994
966
999
8 x 20 processor topology 6480 timestep(45s) 20
restarts 30 Hours Wall Clock 80 Gbytes of memory
994
985
81H Forecast valid 04 Sept 00 UTC