Notes on the Oct-2013 CODE.tar Version

These are notes on building and running the version of NEMO configured for the Salish Sea as received in the CODE.tar archive prepared by J-P Paquin and downloaded at UBC on 2-Oct-2013.

The “Notes on NEMO/OPA Usage” document in the DOCUMENTATION.tar archive provided guidance for the creation of the notes below.

Todo

Add links to “Notes on NEMO/OPA Usage” .doc and PDF

What’s in CODE.tar

The CODE.tar tarball unpacks to a directory called CONCEPTS110_WCSD_OW_NOBC_tide which apparently indicates:

  • CONCEPTS110: Canadian Operational Network of Coupled Environmental PredicTion Systems model at the version 110 level

  • WCSD: MEOPAR West Coast Sub-Domain configuration

  • OW: Open West boundary

  • NOBC: ?? maybe refers to reading OBC file data instead of initial conditions ??

  • tide: focus of the model runs is to calculate tides in the domain

CONCEPTS110_WCSD_OW_NOBC_tide contains jcompile.sh, as bash script that runs the build tool chain described below, and the modipsl/ directory.

modipsl (http://forge.ipsl.fr/igcmg/wiki/platform/en/documentation/J_outils#modipsl) is the framework that was used to obtain and build the NEMO code prior to NEMO up to v3.2. modipsl is available via anonymous svn checkout from http://forge.ipsl.fr/igcmg/svn/modipsl/. However, it is unclear what revision/tag of modipsl the tarball contains.

The documented (in “Notes on NEMO/OPA Usage” and in the “NEMO Quick Start Guide”: https://www.nemo-ocean.eu/Using-NEMO/User-Guides/Basics/NEMO-Quick-Start-Guide) means of obtaining the NEMO code is to run:

cd modipsl/util
model NEMO

which automates the process of doing checkouts of code from a collection of cvs and svn repositories.

Examination of the modipsl/util/log file that the modipsl/util/model script creates and the the modipsl/util/mod.def definitions file for the script indicate that model was run on 18-Mar-2010 and that the nemo_v3_1 tag was used to obtain the NEMO code.

Build Notes

Note

The build process described here was attempted on several platforms with the following results:

  • MacBook running OS/X 10.8.5 with Xcode installed: fait_AA_make reported sed: RE error: illegal byte sequence numerous times,

  • salish: fait_AA_make failed because ksh was not installed

  • jasper: build was successful

The build NEMO 3.1 for a new configuration the following steps are required:

  • modeles/NEMO/OPA_SRC/par_oce.F90 must be edited to set the number of processors to be used, and include configuration parameters via a .h90 file. The edits are done in C-preprocessor (CPP) blocks. J-P Paquin did this, using the CPP key key_wc3 to specify an 8x8 processor layout and parameters in par_WC3.h90. The edits are flagged with comments that start with !JPP.

  • Based on modeles/NEMO/OPA_SRC/par_ORCA_R2.h90, create the configuration parameters header file included via the new CPP key in modeles/NEMO/OPA_SRC/par_oce.F90. The file created by J-P Paquin is par_WC3.h90.

  • Edit modeles/UTIL/fait_config to define the NEMO source files that are required for the build. The configuration name, WC3 in the present case, must be added to the LIST shell variable, a newline-separated list of configuration names. A set -A statement must also be added to the collection that follows the LIST. The configuration is referenced in the set -A statement as DIR_configname; i.e. DIR_WC3 for the present case. The set -A statement lists the source code directories that are to be included in the build. Those edits are present in fait_config in the CODE.tar tarball.

  • Run:

    cd modeles/UTIL
./fait_config WC3

    That results in the creation of the modeles/NEMO/WORK/ directory in which the source files from the directories given in the set -A DRI_WC3 ... statement in fait_config are symlinked so as to appear to be all in the same directory. The modipsl/config/WC3/ directory is also created. Its scripts/ directory contains the BB_make and BB_make.ldef files.

  • Edit BB_make.ldef to activate/deactivate CPP keys for the configuration. The keys are listed in a single, space-separated line as the value of the P_P variable. modipsl/config/WC3/scripts/BB_make.ldef in CODE.tar contains three P_P lists identified with #-- JPP comments. The uncommented one (and so presumably most recently used) is:

    #-- JPP 20130717 Run compilation keys for TEST1
P_P = key_wc3 key_dtatem key_dtasal key_flx_core key_vvl key_zrefsurf key_zdftke key_traldf_c2d key_dynldf_c3d key_mpp_mpi key_ldfslp key_dynspg_ts2 key_dtatem_month key_dtasal_month key_obc_mer key_tide key_diaharm

    Also ensure that there is a prefix for preprocessing line for the build/run target, for example:

    #-Q- jasper  prefix = -D

  • Ensure that there is an appropriate set of definitions in modipsl/util/AA_make.gdef for the build/run target, for example:

    #-Q- jasper     #- Global definitions for jasper.westgrid.ca using Linux Compiler Intel v8
#-Q- jasper     LIB_MPI = MPI2
#-Q- jasper     LIB_MPI_BIS =
#-Q- jasper     M_K = make
#-Q- jasper     P_C = cpp
#-Q- jasper     P_O = -P -C -traditional $(P_P)
#-Q- jasper     F_C = mpiifort -c -fpp
#-Q- jasper     #-D- MD    F_D = -g
#-Q- jasper     #-D- MN    F_D =
#-Q- jasper     #-P- I4R4  F_P = -i4
#-Q- jasper     #-P- I4R8  F_P = -i4 -r8
#-Q- jasper     #-P- I8R8  F_P = -i8 -r8
#-Q- jasper     #-P- ??    F_P = -i4 -r8
#-Q- jasper     F_O = -O3 $(F_P)  -I$(MODDIR) -I$(MODDIR)/oce -module $(MODDIR) -assume byterecl -convert big_endian -I $(NCDF_INC)
#-Q- jasper     F_F = $(F_O) -extend_source
#-Q- jasper     F_L = mpiifort
#-Q- jasper     L_O =
#-Q- jasper     A_C = ar -r
#-Q- jasper     A_G = ar -x
#-Q- jasper     C_C = gcc -c
#-Q- jasper     C_O =
#-Q- jasper     C_L = gcc
#-Q- jasper     #-
#-Q- jasper     NCDF_INC = -I/lustre/jasper/software/netcdf/netcdf-4.1.3/include -lhdf5_hl -lhdf5 -lz -lsz
#-Q- jasper     NCDF_LIB = -L/lustre/jasper/software/netcdf/netcdf-4.1.3/lib -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz

  • Run:

    cd modeles/NEMO
../UTIL/fait_AA_make

    to calculate compilation rules, options, and build dependencies so as to create NEMO/WORK/AA_make (which is symlinked to modipsl/config/WC3/scripts/BB_make)

    Note

    fait_AA_make must be run from the modeles/NEMO/ directory.

  • Run:

    cd modipsl/util
./clr_make
./ins_make -t target

    to remove existing Makefiles and create new ones. The target argument to ins_make specifies a compiler or host name defined in modipsl/util/AA_make.gdef and modipsl/config/WC3/scripts/BB_make.ldef.

  • Run:

    cd modipsl/config/WC3
make clean
make

    to compile and link the code.

The results of a successful build are:

  • a ../../bin/opa executable

  • a ../../lib/libioipsl.a library

  • a ../../lib/oce/libopa.a library

Problems

With the CPP keys above in BB_make.ldef the make command on jasper completes with these messages:

dynadv_ppm.F90(76): warning #6843: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [PHTRA_ADV]
   SUBROUTINE adv_ppm_hor ( kt, pun, pvn, tra, traa, phtra_adv, z2, sort )
-----------------------------------------------------^
./ldfdyn_c3d.h90(148): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
               IF(lwp) WRITE(numout,'(34x,E7.2,8x,i3)') zcoef(jk) * ahm0, jk
-------------------------------------------^
dynzdf_imp.F90(20): remark #6536: All symbols from this module are already visible due to another USE; the ONLY clause will have no effect. Rename clauses, if any, will be honored.   [OCE]
   USE oce             ! ocean dynamics and tracers
-------^

The library is up-to-date

mpiifort  -o ../../../bin/opa model.o ../../../lib/oce/libopa.a  ../../../lib/libioipsl.a -L/lustre/jasper/software/netcdf/netcdf-4.1.3/lib -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz
/lustre/jasper/software/intel/l_ics_2012.0.032/composer_xe_2011_sp1.10.319/compiler/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
OPA model is OK

Run Notes

The WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY.tar tarball contain the namelist and scripts to setup and run on the BIO HPC cluster:

  • linkfile.sh links the intial conditions, forcing, etc. files into the run directory with the file names that NEMO expects

  • namelist is the NEMO namelist for the run

  • submit_64.sh is the file containing PBS directives and shell commands that is submitted to the TORQUE resource manager via qsub

The meaning of WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY (from J-P’s README.txt) is:

WCSD   : West Coast Sub Domain (398x345)
M2     : Run with only M2 tides from WebTide
OW_ON  : Open West & OpenNorth boundaries
file   : reading OBC file (not initial conditions)
DAMP   : increased horizontal eddy viscosity
ANALY  : Analytical forcing (namsbc_ana) - no atm-ocean fluxes
         or atmospheric forcing

The WCSD_PREP.tar tarball contains the intial conditions, forcing, etc. files for the WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY case.

With those two tarballs unpacked beside each other one the dirPREP variable in linkfile.sh need to be set to:

dirPREP=../WCSD_PREP

and linkfile.sh run in WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY to prepare for the run.

submit_64.sh is tailored to the BIO HPC cluster. To run on jasper, the following script was used:

#!/bin/bash

#PBS -N WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY
#PBS -S /bin/bash
#PBS -l procs=64
# memory per processor
#PBS -l pmem=2gb
#PBS -l walltime=1:00:00
# email  when the job [b]egins and [e]nds, or is [a]borted
#PBS -m bea
#PBS -M dlatornell@eos.ubc.ca
#PBS -o OPA.output
#PBS -e OPA.output.error


cd $PBS_O_WORKDIR
echo working dir: $(pwd)

module load compiler/intel/12.1
module load library/intelmpi/4.0.3.008
module load library/netcdf/4.1.3
module load library/szip/2.1

mpiexec ./opa

If that script is stored as jasper.pbs, a run is submitted with the command:

qsub jasper.pbs

As an initial test, the run duration was set to 720 time steps via the &namrun.nitend namelist item. The run completed in just over 2 minutes. A subsequent 4320 time step run took about 17 minutes.

Post-Processing

The results of the runs described above are groups of 64 netCDF files (one for each processor) for each of the calculated quantities:

  • U, V, W, and T

  • output.init: initial time step output ??

  • restart and open boundary condition restart

  • 2D slice timeseries results

  • tidal harmonics disagnostic results

NOCSCOMBINE

Google lead to the NOCSCOMBINE tool at ftp://ftp.soc.soton.ac.uk/omfftp/NEMO/NOCSCOMBINE.tar. Building it on jasper required creation of a new makefile with NCHOME and LIBS variable set to:

NCHOME = /lustre/jasper/software/netcdf/netcdf-4.1.3
LIBS = -L$(NCHOME)/lib -I$(NCHOME)/include -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz

Commands like:

cd WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY/
../../NOCSCOMBINE/nocscombine -f WC3_CU60_20020102_20020104_grid_U_0000.nc

result in the 64 pre-processor files of u velocity results being combined into a single WC3_CU60_20020102_20020104_grid_U.nc file. The process takes over 10 minutes per quantity for U, V, and T for the 72 hour run, and nearly 30 minutes for W.