Running NEMO under prepIFS at ECMWF

1
Running NEMO under prepIFS at ECMWF

• Kristian S. Mogensen
• Last revised 20061107

2
Outline

• Launching NEMO experiments in prepIFS
• prepIFS menus
• suite structure
• directory layout on the supercomputer
• Scripts for NEMO
• Making changes to the code for a running
  experiment

3
Launching NEMO experiments in prepIFS basic steps

1. Start prepIFS using the prepIFS local command
   on ecgate
2. Create a new experiment as CY31R2
3. Select type opa
4. Review the menus (see walk through later).
5. Possibly change the execution host using the
   right mouse button
6. Possibly change the resolution using the right
   mouse button
7. Remark the CY31R2 release version is not related
   to IFS in any way, but used by me as a way to
   have different menus for different development
   stages

4
Submission of NEMO run creating experiment
5
Submission of NEMO run selecting NEMO
OPAVAR is the default NEMO is NOT
6
Submission of NEMO run controlling scripts and
NEMO version
Scripts not in perforce
Your changes
7
Submission of NEMO run fluxes, initial data and
observations
The OASIS3 interface will be ignored in this
presentation
8
Submission of NEMO run date control
Selecting a different resolution will
automatically change the time step (will be
discussed later).
9
Submission of NEMO run directories
Only needed for special non default runs
10
Submission of NEMO run experiment submission
Special submit script used for type OPA
11
Submission of NEMO run job submission
requirements
Tasks/memory allocation will change when
selecting a different resolution
SCHOST is changeable by a menu
12
Submission of NEMO run changing the
supercomputer host
Change execution host using this menu
13
Submission of NEMO run changing resolution
Change ORCA grid using this menu. This will
change the time step
14
Submission of NEMO run submitting the experiment
15
Use of sms/xcdp to run experiments

• SMS (Supervisor Monitor Scheduler) is a tool to
  control a suite consisting of tasks with
  dependencies
• A task is typically a small well defined part of
  an experiment. An example is to retrieve the
  observation files.
• A task is defined by as SMS wrapper which is an
  ascii file containing a script with additional
  SMS directives. When the task is submitted the
  wrapper is pre-processed into a job script and
  submitted onto the execution host.
• The most common directives can be SMS variables
  (syntax SMSVAR) and include statements
  (syntax include ltconfig.hgt)
• The xcdp program is a graphical interface used
  to monitor the progress of running experiments
• Xcdp can be used to change a running experiment
  as well
• More information (manuals etc.) can be found on
• http//www.ecmwf.int/products/data/software/sms.ht
  ml

16
Example of script use OPA convention

• Check for errors
• if errflg -ne 0 exedir unknown
  expnam unknown \\
• rundir unknown then
• print usage
• exit 1
• fi
• cd rundir
• pwd
• ls -l
• TRACEBK0
• exenamnemo
• exedir/exenam
• cat ocean.output

• !/bin/ksh
• run the model
• original 02-06 (Arthur)
• Set error trapping on
• set -eux
• Document command line options
• usage"Usage\n0 -r ltworking directorygt -e name
  of the experimentgt\\
• -d ltbinary directorygt"
• Set defaults for command line options
• rundirunknown
• expnamunknown

17
Example of SMS wrapper

• manual
• runmod Run NEMO
• end
• include ltqsub.hgt
• QSUB -lT CPUTIME_NEMO
• QSUB -lM MEM_NEMOMb
• QSUB -lh 1
• QSUB -lPv NEMO_NPROC
• include ltstep1.hgt
• include ltconfig.hgt
• include ltrcp.hgt
• include lttrap.hgt
• include ltsetup.hgt
• include ltnemo.hgt
• include ltopavardates.hgt
• include lttmpdir.hgt

• if OASIS3NEWWEIGHTS yes OCNINDAT
  OCCURDAT then
• weights"-w -W OCNDATA -n N80_OPA_GRID_mask"
  
• else
• weights"-W OCNDATA -n N80_OPA_GRID_mask"
• Fi
• if OASIS3DEBUG0 1 then
• oasis3debug-O
• else
• oasis3debug
• fi
• if OASIS3FLUXES yes then
• nrunmod_oasis3 -r WDIR/nemo_model/ -d BINS -e
  EXPVER \\
• p NEMO_NPROC -g
  OPA_GRID -t OCRUNDAYS \\
• -i WINSTART
  weights oasis3debug
• else
• nrunmod -r WDIR/nemo_model/ -d BINS -e
  EXPVER
• fi

18
Script conventions used by NEMO/OPAVAR

• We are aiming of having a system which can/will
  also be used outside SMS by our partners
• The basic idea is that the scripts is SMS
  independent with all information needed passed
  through as arguments
• No environment variables are used with the
  scripts
• The SMS wrappers provides the link between the
  SMS environment and the scripts
• Rather that using e.g. ECFS commands directly in
  the scripts generic scripts (e.g. CP_ARCH) are
  used instead. The content of these scripts is of
  course site dependent

19
The OPA/NEMO suite in xcdp
20
Main directories (internal ECMWF user)

• On disk
• /fdb/EXPVER/data/ninput
• Contains date independent data and one set of
  restart files
• /fdb/EXPVER/YMD/opa/nemo_model
• Directory for running the model
• In ecfs
• ec/RDX/prepIFS/EXPVER/log
• Logfiles
• ec/RDX/prepIFS/EXPVER/output/opa/RESTART
• Restart files
• ec/RDX/prepIFS/EXPVER/output/opa/OUTPUT
• Model output

21
Main directories (member state user)

• On disk
• /msx_fdb/USER/EXPVER/data/ninput
• Contains date independent data and one set of
  restart files
• /msx_fdb/USER/EXPVER/YMD/opa/nemo_model
• Directory for running the model
• In ecfs
• ec/xfr,xgb,../public/EXPVER/log
• Logfiles
• ec/xfr,xgb,../public/EXPVER/output/opa/RESTART
  
• Restart files
• ec/xfr,xgb,../public/EXPVER/output/opa/OUTPUT
• Model output

22
Exercise/demonstration

1. Start prepIFS and load an opa experiment type
2. Change the RUNOPAVAR/RUNNEMO variables to run a
   nemo run
3. Set up a 3 month run with 1 month chunks
4. Change the domain decomposition to use 4x2 MPI
   tasks
5. Launch the experiment
6. Start xcdp to monitor the experiment
7. Login to the supercomputer and change to the
   directory where the model run
8. Have a look at the file ocean.output

23
Version control issues for NEMO

• To use your NEMO branch change NEMO_BRANCH in
  the Libraries / Master directory prepIFS menu.
• Last committed version of all files are transfer
  to the supercomputer and compiled
• So remember to commit all changes before
  launching the experiment
• Scripts and SMS wrapper is not version controlled
  (yet).
• You can copy the whole directory structure from
  the default OPAMASTER directory to a directory of
  your choice and set OPAMASTER accordingly

24
ECMWFs make system for NEMO under SMS

• Steps in the make process in SMS
• p4setup_nemo sets up the NEMO perforce build
  client
• p4diff_nemo syncs the build client and does a
  q2 diffbranch
• setupnemomake makes a directory for objects etc
• linknemomake Links the following files from
  Q2CLIENT_ROOT/nemo/build to the object directory
• Makefile.nemo -gt Makefile
• .make -gt .make
• ltARCHgt.config -gt compiler_settings
• CPP_keys. -gt CPP_keys.
• nemomake runs make in the object directory
• Different possible MPP configurations and CPP
  directives so make is always done

25
Make corrections to the NEMO code after the
launch the experiment

• Assumption you have created a branch and
  submitted it, and now you want to make changes to
  branch and recompile. The following steps are
  needed
• Make your changes in perforce and remember to
  submit your changes (using e.g. p4 submit)
• Rerun the p4diff_nemo task from xcdp. This
  synchronize the source code files on the
  supercomputer
• Rerun the nemomake tasks. This will work for most
  modifications of files, but not for additions of
  files
• If you want to be sure that everything is
  recompiled you can requeue the whole
  opa/nemo/make family. This will wipe out any
  previous objects/libraries

26
Exercise/demonstration 2

1. Create a perforce branch of NEMO and insert a I
   was here print statement in step.F90 in a place
   where the code get executed.
2. Set up a new short experiment with the above
   branch.
3. Launch the experiment.
4. Bring the teacher a cop of coffee.
5. Extract the log files from the ECFS and confirm
   that your print statement was executed.