Notes on the Oct-2013 CODE.tar Version

These are notes on building and running the version of NEMO configured for the Salish Sea as received in the CODE.tar archive prepared by J-P Paquin and downloaded at UBC on 2-Oct-2013.

The “Notes on NEMO/OPA Usage” document in the DOCUMENTATION.tar archive provided guidance for the creation of the notes below.


Add links to “Notes on NEMO/OPA Usage” .doc and PDF

What’s in CODE.tar

The CODE.tar tarball unpacks to a directory called CONCEPTS110_WCSD_OW_NOBC_tide which apparently indicates:

  • CONCEPTS110: Canadian Operational Network of Coupled Environmental PredicTion Systems model at the version 110 level

  • WCSD: MEOPAR West Coast Sub-Domain configuration

  • OW: Open West boundary

  • NOBC: ?? maybe refers to reading OBC file data instead of initial conditions ??

  • tide: focus of the model runs is to calculate tides in the domain

CONCEPTS110_WCSD_OW_NOBC_tide contains, as bash script that runs the build tool chain described below, and the modipsl/ directory.

modipsl is the framework that was used to obtain and build the NEMO code prior to NEMO up to v3.2. modipsl is available via anonymous svn checkout from However, it is unclear what revision/tag of modipsl the tarball contains.

The documented (in “Notes on NEMO/OPA Usage” and in the NEMO Quick Start Guide) means of obtaining the NEMO code is to run:

cd modipsl/util
model NEMO

which automates the process of doing checkouts of code from a collection of cvs and svn repositories.

Examination of the modipsl/util/log file that the modipsl/util/model script creates and the the modipsl/util/mod.def definitions file for the script indicate that model was run on 18-Mar-2010 and that the nemo_v3_1 tag was used to obtain the NEMO code.

Build Notes


The build process described here was attempted on several platforms with the following results:

  • MacBook running OS/X 10.8.5 with Xcode installed: fait_AA_make reported sed: RE error: illegal byte sequence numerous times,

  • salish: fait_AA_make failed because ksh was not installed

  • jasper: build was successful

The build NEMO 3.1 for a new configuration the following steps are required:

  • modeles/NEMO/OPA_SRC/par_oce.F90 must be edited to set the number of processors to be used, and include configuration parameters via a .h90 file. The edits are done in C-preprocessor (CPP) blocks. J-P Paquin did this, using the CPP key key_wc3 to specify an 8x8 processor layout and parameters in par_WC3.h90. The edits are flagged with comments that start with !JPP.

  • Based on modeles/NEMO/OPA_SRC/par_ORCA_R2.h90, create the configuration parameters header file included via the new CPP key in modeles/NEMO/OPA_SRC/par_oce.F90. The file created by J-P Paquin is par_WC3.h90.

  • Edit modeles/UTIL/fait_config to define the NEMO source files that are required for the build. The configuration name, WC3 in the present case, must be added to the LIST shell variable, a newline-separated list of configuration names. A set -A statement must also be added to the collection that follows the LIST. The configuration is referenced in the set -A statement as DIR_configname; i.e. DIR_WC3 for the present case. The set -A statement lists the source code directories that are to be included in the build. Those edits are present in fait_config in the CODE.tar tarball.

  • Run:

    cd modeles/UTIL
    ./fait_config WC3

    That results in the creation of the modeles/NEMO/WORK/ directory in which the source files from the directories given in the set -A DRI_WC3 ... statement in fait_config are symlinked so as to appear to be all in the same directory. The modipsl/config/WC3/ directory is also created. Its scripts/ directory contains the BB_make and BB_make.ldef files.

  • Edit BB_make.ldef to activate/deactivate CPP keys for the configuration. The keys are listed in a single, space-separated line as the value of the P_P variable. modipsl/config/WC3/scripts/BB_make.ldef in CODE.tar contains three P_P lists identified with #-- JPP comments. The uncommented one (and so presumably most recently used) is:

    #-- JPP 20130717 Run compilation keys for TEST1
    P_P = key_wc3 key_dtatem key_dtasal key_flx_core key_vvl key_zrefsurf key_zdftke key_traldf_c2d key_dynldf_c3d key_mpp_mpi key_ldfslp key_dynspg_ts2 key_dtatem_month key_dtasal_month key_obc_mer key_tide key_diaharm

    Also ensure that there is a prefix for preprocessing line for the build/run target, for example:

    #-Q- jasper  prefix = -D
  • Ensure that there is an appropriate set of definitions in modipsl/util/AA_make.gdef for the build/run target, for example:

    #-Q- jasper     #- Global definitions for using Linux Compiler Intel v8
    #-Q- jasper     LIB_MPI = MPI2
    #-Q- jasper     LIB_MPI_BIS =
    #-Q- jasper     M_K = make
    #-Q- jasper     P_C = cpp
    #-Q- jasper     P_O = -P -C -traditional $(P_P)
    #-Q- jasper     F_C = mpiifort -c -fpp
    #-Q- jasper     #-D- MD    F_D = -g
    #-Q- jasper     #-D- MN    F_D =
    #-Q- jasper     #-P- I4R4  F_P = -i4
    #-Q- jasper     #-P- I4R8  F_P = -i4 -r8
    #-Q- jasper     #-P- I8R8  F_P = -i8 -r8
    #-Q- jasper     #-P- ??    F_P = -i4 -r8
    #-Q- jasper     F_O = -O3 $(F_P)  -I$(MODDIR) -I$(MODDIR)/oce -module $(MODDIR) -assume byterecl -convert big_endian -I $(NCDF_INC)
    #-Q- jasper     F_F = $(F_O) -extend_source
    #-Q- jasper     F_L = mpiifort
    #-Q- jasper     L_O =
    #-Q- jasper     A_C = ar -r
    #-Q- jasper     A_G = ar -x
    #-Q- jasper     C_C = gcc -c
    #-Q- jasper     C_O =
    #-Q- jasper     C_L = gcc
    #-Q- jasper     #-
    #-Q- jasper     NCDF_INC = -I/lustre/jasper/software/netcdf/netcdf-4.1.3/include -lhdf5_hl -lhdf5 -lz -lsz
    #-Q- jasper     NCDF_LIB = -L/lustre/jasper/software/netcdf/netcdf-4.1.3/lib -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz
  • Run:

    cd modeles/NEMO

    to calculate compilation rules, options, and build dependencies so as to create NEMO/WORK/AA_make (which is symlinked to modipsl/config/WC3/scripts/BB_make)


    fait_AA_make must be run from the modeles/NEMO/ directory.

  • Run:

    cd modipsl/util
    ./ins_make -t target

    to remove existing Makefiles and create new ones. The target argument to ins_make specifies a compiler or host name defined in modipsl/util/AA_make.gdef and modipsl/config/WC3/scripts/BB_make.ldef.

  • Run:

    cd modipsl/config/WC3
    make clean

    to compile and link the code.

The results of a successful build are:

  • a ../../bin/opa executable

  • a ../../lib/libioipsl.a library

  • a ../../lib/oce/libopa.a library


With the CPP keys above in BB_make.ldef the make command on jasper completes with these messages:

dynadv_ppm.F90(76): warning #6843: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.   [PHTRA_ADV]
   SUBROUTINE adv_ppm_hor ( kt, pun, pvn, tra, traa, phtra_adv, z2, sort )
./ldfdyn_c3d.h90(148): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'.
               IF(lwp) WRITE(numout,'(34x,E7.2,8x,i3)') zcoef(jk) * ahm0, jk
dynzdf_imp.F90(20): remark #6536: All symbols from this module are already visible due to another USE; the ONLY clause will have no effect. Rename clauses, if any, will be honored.   [OCE]
   USE oce             ! ocean dynamics and tracers

The library is up-to-date

mpiifort  -o ../../../bin/opa model.o ../../../lib/oce/libopa.a  ../../../lib/libioipsl.a -L/lustre/jasper/software/netcdf/netcdf-4.1.3/lib -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz
/lustre/jasper/software/intel/l_ics_2012.0.032/composer_xe_2011_sp1.10.319/compiler/lib/intel64/ warning: warning: feupdateenv is not implemented and will always fail
OPA model is OK

Run Notes

The WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY.tar tarball contain the namelist and scripts to setup and run on the BIO HPC cluster:

  • links the intial conditions, forcing, etc. files into the run directory with the file names that NEMO expects

  • namelist is the NEMO namelist for the run

  • is the file containing PBS directives and shell commands that is submitted to the TORQUE resource manager via qsub

The meaning of WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY (from J-P’s README.txt) is:

WCSD   : West Coast Sub Domain (398x345)
M2     : Run with only M2 tides from WebTide
OW_ON  : Open West & OpenNorth boundaries
file   : reading OBC file (not initial conditions)
DAMP   : increased horizontal eddy viscosity
ANALY  : Analytical forcing (namsbc_ana) - no atm-ocean fluxes
         or atmospheric forcing

The WCSD_PREP.tar tarball contains the intial conditions, forcing, etc. files for the WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY case.

With those two tarballs unpacked beside each other one the dirPREP variable in need to be set to:


and run in WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY to prepare for the run. is tailored to the BIO HPC cluster. To run on jasper, the following script was used:


#PBS -S /bin/bash
#PBS -l procs=64
# memory per processor
#PBS -l pmem=2gb
#PBS -l walltime=1:00:00
# email  when the job [b]egins and [e]nds, or is [a]borted
#PBS -m bea
#PBS -o OPA.output
#PBS -e OPA.output.error

echo working dir: $(pwd)

module load compiler/intel/12.1
module load library/intelmpi/
module load library/netcdf/4.1.3
module load library/szip/2.1

mpiexec ./opa

If that script is stored as jasper.pbs, a run is submitted with the command:

qsub jasper.pbs

As an initial test, the run duration was set to 720 time steps via the &namrun.nitend namelist item. The run completed in just over 2 minutes. A subsequent 4320 time step run took about 17 minutes.


The results of the runs described above are groups of 64 netCDF files (one for each processor) for each of the calculated quantities:

  • U, V, W, and T

  • output.init: initial time step output ??

  • restart and open boundary condition restart

  • 2D slice timeseries results

  • tidal harmonics disagnostic results


Google lead to the NOCSCOMBINE tool at Building it on jasper required creation of a new makefile with NCHOME and LIBS variable set to:

NCHOME = /lustre/jasper/software/netcdf/netcdf-4.1.3
LIBS = -L$(NCHOME)/lib -I$(NCHOME)/include -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz

Commands like:

cd WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY/
../../NOCSCOMBINE/nocscombine -f

result in the 64 pre-processor files of u velocity results being combined into a single file. The process takes over 10 minutes per quantity for U, V, and T for the 72 hour run, and nearly 30 minutes for W.