NEMO-code Repository¶
These notes describe the Salish Sea MEOPAR project NEMO-code repository and its maintenance. They are a narrative guide to the Mercurial log and diffs that can be obtained from the repository itself or via the Bitbucket interface.
The NEMO-code repo is a Mercurial repository in which is maintained the merger of the trunk of the main NEMO svn repository and the changes made by the Salish Sea MEOPAR project team.
Note
The NEMO-code repository is a private repository for members of the Salish Sea MEOPAR project team. That is because it contains parts of the NEMO codebase. Although that codebase is openly licensed its developers require registration to access the code.
If you have completed that registration and would like access to the NEMO-code, please contact Susan Allen, the Salish Sea MEOPAR project leader.
Getting the Code¶
Team members using SSH key authentication on Bitbucket may clone the NEMO-code repo with:
hg clone ssh://hg@bitbucket.org/salishsea/nemo-code NEMO-code
For password authentication use:
hg clone https://<you>@bitbucket.org/salishsea/nemo-code NEMO-code
where <you> is your Bitbucket user id.
Managing Configurations¶
Building an Existing Configuration¶
To build the executable code for an already defined configurations, for example, SalishSea, use:
cd NEMO-code/NEMOGCM/CONFIG
./makenemo -n SalishSea -m salish -j8
That will compile the full domain Salish Sea NEMO configuration and the IOM output server with the salish architecture definitions with the compilation distributed over 8 cores.
The resulting executables are located in NEMO-code/NEMOGCM/CONFIG/SalishSea/BLD/bin/
.
Available Configurations¶
In addition to the configurations from the NEMO svn checkout, the repo contains:
SalishSea
: The initial full domain Salish Sea development configurationSalishSea_TRC
: SalishSea above plus passive tracersSalishSea_no_IOM
: A copy of theSalishSea
configuration compiled without the key_iom_put CPP keySALISH_amm
: A very early stepping-stone configuration created during the learning process of creating a Salish Sea configuration based on theAMM12
configurationSALISH_JPP
: The Salish Sea sub-domain configuration used to compare with CONCEPTS-110
Available Architecture Definitions¶
In addition to the collection of architecture definitions that the NEMO consortium provides, the NEMO-code repo includes definitions for:
mpif90_jasper: OpenMPI builds on jasper.westgrid.ca
ifort_jasper: Intel MPI builds on jasper.westgrid.ca
salish: MPI builds on salish.eos.ubc.ca
ocean: single processor builds on UBC-EOAS ocean cluster workstations and salish
gfortran_osx: single processor builds on Mac-OSX
Creating a New Configuration¶
To create a new configuration based on, for example, AMM12 use:
cd NEMO-code/NEMOGCM/CONFIG
./makenemo -r AMM12 -n MY_AMM12 -m salish -j8 add_key "key_netcdf4 key_nosignedzero"
That will use the existing AMM12 configuration as a basis to build a new configuration called MY_AMM12 with the salish architecture definitions and with compilation distributed over 8 cores.
The C Pre-Processor (CPP) keys key_netcdf4 and key_nosignedzero will be added to configurations.
The resulting configuration,
including a compiled and link NEMO executable,
is located in NEMO-code/NEMOGCM/CONFIG/MY_AMM12
.
See ./makenemo -h for details of options and sub-commands.
Running the Model¶
We don’t want to clutter the NEMO-code repo with files from development and exploration run-sets
(aka experiments),
run results,
etc.,
so runs are done in directories outside the NEMO-code/
tree.
The Salish Sea Run Sets Files repo contains example run description, namelist, and output server control files that define a run. Run description files define the names and locations of files and directories that the Salish Sea NEMO Command Processor tool uses to manage Salish Sea NEMO runs and their results. See Run Description File Structure for details of the run description file syntax.
After copying and/or editing the run description, namelist, and output server control files to define a run use the salishsea prepare Sub-command to set up a temporary run directory from which to execute the run. For example:
salishsea prepare SalishSea.yaml iodef.xml
Use the run description and output server definitions files names that are appropriate for your run. See the prepare Sub-command docs for details of the files and symbolic links that are created in the run directory. The path to the run directory is printed upon completion of the command.
Go to the run directory and start the run with a command like:
mpiexec -n 16 ./nemo.exe > stdout 2> stderr &
That command runs the model in the background on 16 processors,
redirecting the stdout and stderr streams to stdout
and stderr
files.
The number of processors to run is must match the domain decomposition defined in the namelist.compute
file.
A convenient command to monitor the memory use of a run and its time step progress is:
watch -n 5 "(free -m; cat time.step)"
The salishsea gather sub-command collects the run definition and results files from a Salish Sea NEMO run into a results directory. The collection process includes combining the per-processors netCDF results files into files in the results directory. It has a number of option flags to define how it works; see salishsea gather -h for details. A typical use on salish is:
salishsea gather --no-compress SalishSea.yaml ../results/15-21Sep2002
The files that define the run, and the non-netCDF results files produced by the run are also moved to the results directory by salishsea gather.
NEMO svn Repo Mirror Maintenance¶
The /ocean/sallen/hg_repos/NEMO-hg-mirror
repository is an svn checkout of http://forge.ipsl.jussieu.fr/nemo/svn/branches/2012/dev_v3_4_STABLE_2012 and also a read-only Mercurial repository.
It was initialized with:
$ cd /ocean/sallen/hg_repos
$ svn --username "dlatornell" co -r 3819 http://forge.ipsl.jussieu.fr/nemo/svn/branches/2012/dev_v3_4_STABLE_2012 NEMO-hg-mirror
$ hg init NEMO-hg-mirror
$ cd NEMO-hg-mirror
$ cat > .hgignore
.svn
DOC/NEMO_book.pdf
ctrl-d
$ hg add
$ hg ci -m"Initialize NEMO svn mirror at r3819 of ^/branches/2012/dev_v3_4_STABLE_2012."
svn v1.7.5 was used on salish for the svn part of the initialization.

NEMO code repositories and workflow to update and merge SVN and local changes¶
Workflow to Pull Changes from NEMO svn Repo¶
The workflow to pull changes from the master NEMO svn repo and commit them to our NEMO-hg-mirror
repo is somewhat automated by the Marlin – Salish Sea NEMO svn-hg Maintenance Tool.
Review the upstream changes in the source browser at https://forge.ipsl.jussieu.fr/nemo/log/branches/2012/dev_v3_4_STABLE_2012 to select a range of changes to be pulled into our
NEMO-hg-mirror
repo.Note
Pay special attention to changes in the
OPA_SRC/
tree that involve files that have been copied intoNEMOGCM/CONFIG/SalishSea/MY_SRC/
or team members’MY_SRC/
directories. Those files must be manually merged with theirMY_SRC/
counterparts.Working on salish in the
/ocean/sallen/hg_repos/NEMO-hg-mirror
repo with an activated virtualenv in which marlin is installed:$ ssh salish $ workon marlin (marlin)$ cd /ocean/sallen/hg_repos/NEMO-hg-mirror
Use marlin incoming information about the next SVN revision that will be pulled from upstream and confirm that it is the expected revision:
(marlin)$ marlin incoming r3843 2013-03-20 09:29:58 UTC correct ice output filename for limwri.F90
The --limit option can be used to see more incoming revisions; see marlin help incoming for details.
Use marlin update to update the working copy to the next upstream commit and commit the SVN update as a Mercurial changeset with the SVN commit message as the body of the Mercurial commit message and echo that message:
(marlin)$ marlin update Update to svn r3843. correct ice output filename for limwri.F90
The --to-rev option can be used to apply a series of upstream updates, committing them to Mercurial one at a time; see marlin help update for details.
Workflow to Merge NEMO svn Repo and Salish Sea Revisions¶
Merging changes from NEMO svn and the Salish Sea central NEMO-code repo on Bitbucket is done in a repo that is used for only that purpose.
Doug does the merges on his laptop.
The repo in which the merging is done was created by cloning the /ocean/sallen/hg_repos/NEMO-hg-mirror
repo:
hg clone ssh://sable.eos.ubc.ca//ocean/sallen/hg_repos/NEMO-hg-mirror NEMO-mirror-merge
and setting the paths in its .hg/hgrc
to:
[paths]
bb = ssh://hg@bitbucket.org/salishsea/nemo-code
default-push = ssh://hg@bitbucket.org/salishsea/nemo-code
mirror = ssh://sable.eos.ubc.ca//ocean/sallen/hg_repos/NEMO-hg-mirror
Those paths mean that the repo for hg pull and hg incoming commands must be specified explicitly.
The bb and mirror paths are provided to facilitate pulling from NEMO-code on Bitbucket and /ocean/sallen/hg_repos/NEMO-hg-mirror
,
respectively.
hg push and hg outgoing commands will act on the NEMO-code repo,
unless otherwise specified.
After the Workflow to Pull Changes from NEMO svn Repo has been completed the workflow to merge those changes with Salish Sea MEOPAR project revisions is:
Pull and update recent changes from NEMO-code into NEMO-mirror-merge:
cd NEMO-mirror-merge hg pull --update bb
Pull and update the changes from
/ocean/sallen/hg_repos/NEMO-hg-mirror
into NEMO-mirror-merge:hg pull mirror
Because the changesets pulled from NEMO-code are public a branch merge is necessary:
hg merge hg commit -m"Merge svn updates."
Manually merge and commit changes that involve files that have been copied into
NEMOGCM/CONFIG/SalishSea/MY_SRC/
or team members’MY_SRC/
directories. Those files are most likely to be inOPA_SRC/
.Push the result of the updates and merges to NEMO-code:
hg push bb
If other users have pushed changes to NEMO-code while merge conflicts were being handled hg pull --rebase can be used to bring in those changes and deal with any additional merge conflicts.
Notify team members of the upstream merge, especially if manual merges of
MY_SRC/
files were required, so that they can manage merging changes into any untrackedMY_SRC/
files they may have.
Migration to NEMO-3.6¶
In February 2015 the process of migrating the Salish Sea NEMO model from NEMO-3.4 to NEMO-3.6 was started. A collection of mirror repos similar to that described in NEMO svn Repo Mirror Maintenance was created. NEMO-3.6 uses a separately distributed output server package called XIOS so the maintenance of Mercurial mirror repos for the Salish Sea NEMO project is expanded to deal with 2 upstream SVN repos. For NEMO:
/ocean/sallen/hg_repos/NEMO-3.6-hg-mirror
/Users/doug/MEOPAR/NEMO-3.6-mirror-merge
https://bitbucket.org/salishsea/nemo-3.6-code
and for XIOS:
/ocean/sallen/hg_repos/XIOS-hg-mirror
/Users/doug/MEOPAR/XIOS-mirror-merge
https://bitbucket.org/salishsea/xios
The /ocean/sallen/hg_repos/NEMO-hg-mirror
repository is an svn checkout of http://forge.ipsl.jussieu.fr/nemo/svn/trunk and also a read-only Mercurial repository.
It was initialized with:
$ cd /ocean/sallen/hg_repos
$ svn --username "dlatornell" co http://forge.ipsl.jussieu.fr/nemo/svn/trunk NEMO-3.6-hg-mirror
$ hg init NEMO-3.6-hg-mirror
$ cd NEMO-3.6-hg-mirror
$ cat > .hgignore
.svn
DOC/NEMO_book.pdf
ctrl-d
$ hg add
$ hg ci -m"Initialize NEMO-3.6 svn mirror at r5072 of ^/trunk."
The /ocean/sallen/hg_repos/XIOS-hg-mirror
repository is an svn checkout of http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 and also a read-only Mercurial repository.
It was initialized with:
$ cd /ocean/sallen/hg_repos
$ http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 XIOS-hg-mirror
$ hg init XIOS-hg-mirror
$ cd XIOS-hg-mirror
$ cat > .hgignore
.svn
ctrl-d
$ hg add
$ hg ci -m"Initialize XIOS svn mirror at r549 of ^/branchs/xios-1.0."
svn v1.8.8 was used on salish for the svn part of the initialization.
On 30-Jun-2015 the v3_6_STABLE release of NEMO was announced.
The following process
(not quite as simple as suggested in the 3.6 Quick Start Guide)
was used to update the /ocean/sallen/hg_repos/NEMO-hg-mirror
repository to the HEAD of the http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE branch:
The svn revisions from r5300, through the creation of the
^/branches/2015/nemo_v3_6_STABLE
branch, and onward to r5628 were reviewed and annotated in our tracking spreadsheetThe
/ocean/sallen/hg_repos/NEMO-hg-mirror
repository was Mercurial-tagged with b4_nemo_v3_6_stable.The svn revisions to r5518 were applied a few at a time using the Marlin – Salish Sea NEMO svn-hg Maintenance Tool and the process described in Workflow to Pull Changes from NEMO svn Repo. Revisions that add or delete files, or copy files in from another svn branch were handled individually, using hg add, or hg rm as necessary to add and remove files, followed by hg commit --amend to update the changeset to reflect that additions/deletions.
The commands:
$ svn switch http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE $ svn update -r 5519
were used to change the svn URL to point at the 3.6-stable branch and update to the earliest revision on that branch, r5519
The svn revisions to r5628 were applied a few at a time using the procedure described above for early revisions.
The svn updates were merged with our NEMO-3.6-code and the result tagged as NEMO-3.6r5628.
Initial testing revealed that NEMO-3.6r5628 could not be compiled against XIOS-1.0r548. That issue was resolved by updating and merging XIOS (XML I/O Server) to svn revision r648. The result was confirmed to enable a successful build of NEMO-3.6r5628 and was tagged as XIOS-1.0r648.
The following sections are in-process notes about getting to a running Salish Sea NEMO-3.6 model on various platforms.
Building and Testing XIOS¶
Building on salish¶
On salish,
XIOS was successfully built with the following arch/arch-*
files:
An arch/*.env
file is not required for salish.
arch/arch-GCC_SALISH.path
:
NETCDF_LIB="-lnetcdff -lnetcdf"
HDF5_LIB="-lz"
arch/arch-GCC_SALISH.fcm
:
%CCOMPILER mpicc
%FCOMPILER mpif90
%LINKER mpif90
%BASE_CFLAGS -ansi -w
%PROD_CFLAGS -O3 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -O2
%DEBUG_CFLAGS -g
%BASE_FFLAGS -D__NONE__
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2
%DEBUG_FFLAGS -g
%BASE_INC -D__NONE__
%BASE_LD -lstdc++
%CPP cpp
%FPP cpp -P
%MAKE make
using the command:
$ ./make_xios --arch GCC_SALISH --netcdf_lib netcdf4_seq --job 8
As salish has only 16 physical cores,
running multiple xios_server.exe
processes that use parallel output is unnecessary,
so the --netcdf_lib netcdf4_seq option is used.
Building on jasper¶
On jasper,
XIOS was successfully built with the following arch/arch-*
files:
arch/arch-X64_JASPER.env
:
module load library/openmpi/1.6.4-intel
module load library/netcdf/4.1.3
module load library/hdf5/1.8.8
arch/arch-X64_JASPER.path
:
NETCDF_LIB="-lnetcdf"
HDF5_LIB="-lhdf5_hl -lhdf5 -lz"
arch/arch-X64_JASPER.fcm
:
%CCOMPILER mpicc
%FCOMPILER mpif90
%LINKER mpif90
%BASE_CFLAGS -diag-disable 1125 -diag-disable 279
%PROD_CFLAGS -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -traceback
%DEBUG_CFLAGS -DBZ_DEBUG -g -traceback -fno-inline
%BASE_FFLAGS -D__NONE__
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2 -traceback
%DEBUG_FFLAGS -g -traceback
%BASE_INC -D__NONE__
%BASE_LD -lstdc++
%CPP mpicc -EP
%FPP cpp -P
%MAKE gmake
using the command:
$ ./make_xios --arch X64_JASPER --netcdf_lib netcdf4_seq --job 4
At present, jasper lacks the parallel versions of the netCDF4 library that is required to build XIOS so that it produces a single output file, hence the --netcdf_lib netcdf4_seq option.
Building on orcinus¶
Note
These build notes for orcinus are provisional. They include hard-coded library version numbers and paths for new netCDF and HDF5 library builds that were created in early October 2015. The details below will change when those new libraries have been integrated into modules on orcinus.
On orcinus,
XIOS was successfully built with the following arch/arch-*
files:
arch/arch-X64_ORCINUS.env
:
module load intel
arch/arch-X64_ORCINUS.path
:
NETCDF_INC_DIR="/global/software/lib64/intel/ncsa-tools/netcdf-4.3.3.1/include"
NETCDF_INCDIR="-I $NETCDF_INC_DIR"
NETCDF_LIB_DIR="/global/software/lib64/intel/ncsa-tools/netcdf-4.3.3.1/lib"
NETCDF_LIB="-L$NETCDF_LIB_DIR -lirc -lnetcdf"
HDF5_LIB_DIR="/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/lib"
HDF5_LIB="-L$HDF5_LIB_DIR -lhdf5_hl -lhdf5 -lz"
arch/arch-X64_ORCINUS.fcm
:
%CCOMPILER mpicc
%FCOMPILER mpif90
%LINKER mpif90
%BASE_CFLAGS -diag-disable 1125 -diag-disable 279
%PROD_CFLAGS -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS -g -traceback
%DEBUG_CFLAGS -DBZ_DEBUG -g -traceback -fno-inline
%BASE_FFLAGS -D__NONE__
%PROD_FFLAGS -O3
%DEV_FFLAGS -g -O2 -traceback
%DEBUG_FFLAGS -g -traceback
%BASE_INC -D__NONE__
%BASE_LD -lstdc++ -lirc -shared-intel
%CPP mpicc -EP
%FPP cpp -P
%MAKE make
using the command:
$ ./make_xios --arch X64_ORCINUS --netcdf_lib netcdf4_seq --job 4
Although orcinus has the parallel versions of the netCDF4 library that is required to build XIOS so that it produces a single output file, only the output file per processor has been tested at time of writing, hence the --netcdf_lib netcdf4_seq option.
Testing¶
The test_client.exe
executable that is built with xios_server.exe
and the inputs/iodef.xml
file can be used to test XIOS.
This was done by creating a test-XIOS/
directory,
copying inputs/iodef.xml
into it,
and symlinking xios_server.exe
and test_client.exe
into it.
In contrast to NEMO-3.4 where the IO server configuration is specified in the xmlio_server.def
file,
the configuration for XIOS is included as a stanza in iodef.xml
.
As copied,
iodef.xml
configures XIOS to run in “attached” mode,
similar to how the IO server is used in NEMO-3.4.
The relevant stanza is:
<context id="xios">
<variable_definition>
<variable_group id="buffer">
buffer_size = 80000000
buffer_server_factor_size = 2
</variable_group>
<variable_group id="parameters" >
<variable id="using_server" type="boolean">false</variable>
<variable id="info_level" type="int">50</variable>
</variable_group>
</variable_definition>
</context>
and the line:
<variable id="using_server" type="boolean">false</variable>
sets “attached” mode.
Using qsub to submit a file containing the following shell script with PBS directives runs test_client.exe
on 10 processors and produces 10 netCDF4 output files,
output_0.nc
through output_9.nc
:
#!/bin/bash
#PBS -N test-XIOS
#PBS -S /bin/bash
#PBS -l procs=10
#PBS -l walltime=0:10:00
#PBS -m bea
#PBS -M dlatornell@eos.ubc.ca
#PBS -o stdout
#PBS -e stderr
cd $PBS_O_WORKDIR
echo working dir: $(pwd)
module load library/netcdf/4.1.3
module load library/hdf5/1.8.8
mpirun -np 10 ./test_client.exe
echo done!
Changing the XIOS server configuration in iodef.xml
to:
<variable id="using_server" type="boolean">true</variable>
creating an MPI application file (let’s call it test-XIOS.app) containing:
-np 8 ./test_client.exe
-np 2 ./xios_server.exe
and submitting a PBS script with the mpirun line changed to:
$ mpirun --app ./test-XIOS.app
results in test_client.exe
running on 8 processors and xios_server.exe
running on 2 and produces 2 netCDF4 output files,
output_0.nc
and output_1.nc
.
The netCDF4 files that XIOS produces are not deflated. Running:
$ ncks -4 -L4 output_0.nc output_0.nc
on one of the files produces by the above test reduces the file size to 33% or its original size. Note that the present build of NCO on jasper is against the netCDF3 library so it cannot be used to do this deflation.
Building and Testing NEMO-3.6¶
Building on salish¶
On salish,
NEMO-3.6 was successfully built with the following NEMOGCM/ARCH/
file:
NEMOGCM/ARCH/UBC_EOAS/arch-GCC_SALISH.fcm
:
%XIOS_HOME /data/$USER/MEOPAR/XIOS
%NCDF_INC -I/usr/include
%NCDF_LIB -L/usr/lib -lnetcdff -lnetcdf
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios -lstdc++
%CPP cpp
%FC mpif90
%FCFLAGS -cpp -O3 -fdefault-real-8 -funroll-all-loops -fcray-pointer -ffree-line-length-none
%FFLAGS %FCFLAGS
%LD mpif90
%LDFLAGS
%FPPFLAGS -P -C -traditional
%AR ar
%ARFLAGS -rs
%MK make
%USER_INC %XIOS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %NCDF_LIB
using the commands:
$ cd NEMO-3.6-code/NEMOGCM/CONFIG
$ ./makenemo -n SalishSea -m GCC_SALISH -j4
Important things to note:
Our arch files are contained in the institution-specific directory
NEMOGCM/ARCH/UBC_EOAS/
The %XIOS_HOME build variable uses the
USER
environment variable to locate the XIOS library to link with NEMO. It is assumed that XIOS is installed and built in/data/userid/MEOPAR/XIOS/
on salish.The -lstdc++ library option must follow -lxios otherwise a truly astonishing volume of unresolved reference errors will be generated and the build will fail.
Building on jasper¶
On jasper,
NEMO-3.6 was successfully built with the following NEMOGCM/ARCH/
file:
NEMOGCM/ARCH/UBC_EOAS/arch-X64_JASPER.fcm
:
%NCDF_HOME /global/software/netcdf/netcdf-4.1.3
%HDF5_HOME /global/software/hdf5/hdf5-1.8.9
%XIOS_HOME $HOME/MEOPAR/XIOS
%NCDF_INC -I%NCDF_HOME/include
%NCDF_LIB -L%NCDF_HOME/lib -lnetcdff -lnetcdf -L%HDF5_HOME/lib -lhdf5_hl -lhdf5 -lhdf5
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios
%CPP cpp
%FC mpif90
%FCFLAGS -c -fpp -r8 -O3 -assume byterecl -convert big_endian -heap-arrays
%FFLAGS %FCFLAGS
%LD mpif90
%LDFLAGS -lstdc++
%FPPFLAGS -P -C -traditional
%AR ar
%ARFLAGS -r
%MK make
%USER_INC %XIOS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %NCDF_LIB
using the commands:
$ cd NEMO-3.6-code/NEMOGCM/CONFIG
$ ./makenemo -n SalishSea -m X64_JASPER -j8
Important things to note:
Building on orcinus¶
Note
These build notes for orcinus are provisional. They include hard-coded library version numbers and paths for new netCDF and HDF5 library builds that were created in early October 2015. The details below will change when those new libraries have been integrated into modules on orcinus.
On orcinus,
NEMO-3.6 was successfully built with the following NEMOGCM/ARCH/
file:
NEMOGCM/ARCH/UBC_EOAS/arch-X64_ORCINUS.fcm
:
%NCDF_HOME /global/software/lib64/intel/ncsa-tools/netcdf-4.3.3.1
%HDF5_HOME /global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1
%XIOS_HOME $HOME/MEOPAR/XIOS
%NCDF_INC -I%NCDF_HOME/include -I /global/software/lib64/intel/ncsa-tools/netcdf-fortran-4.4.0/include
%NCDF_LIB -L%NCDF_HOME/lib -lnetcdf -L /global/software/lib64/intel/ncsa-tools/netcdf-fortran-4.4.0/lib -lnetcdff -L%HDF5_HOME/lib -lhdf5_hl -lhdf5
%XIOS_INC -I%XIOS_HOME/inc
%XIOS_LIB -L%XIOS_HOME/lib -lxios
%CPP cpp
%FC mpif90
%FCFLAGS -c -fpp -r8 -O3 -assume byterecl -heap-arrays
%FFLAGS %FCFLAGS
%LD mpif90
%LDFLAGS -lstdc++ -shared-intel
%FPPFLAGS -P -C -traditional
%AR ar
%ARFLAGS rcs
%MK make
%USER_INC %XIOS_INC %NCDF_INC
%USER_LIB %XIOS_LIB %NCDF_LIB
using the commands:
$ cd $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SalishSea
$ source orcinus_build.sh
Important things to note:
The rebuild_nemo.exe tool should also be built with the commands:
Testing the GYRE Configuration on salish¶
A new NEMO-3.6 GYRE configuration was created on salish with:
$ cd NEMOGCM/CONFIG/
$ ./makenemo -n my_GYRE -r GYRE -m GCC_SALISH -j8
After the build completed successfully,
a test directory was created outside the NEMO-3.6-code
repo:
$ cd /data/dlatorne/MEOPAR
$ mkdir GYRE-3.6-test
$ cd GYRE-3.6-test/
and the shared XIOS configurations files, reference namelist, and executables were symlinked into the test directory:
$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/domain_def.xml
$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml
$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/namelist_ref
$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/my_GYRE/BLD/bin/nemo.exe
$ ln -s /data/dlatorne/MEOPAR/XIOS/bin/xios_server.exe
The iodef.xml
and namelist_cfg
files generated for the my_GYRE configuration were copied into the test directory:
$ cp /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/my_GYRE/EXP00/iodef.xml ./
$ cp /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/my_GYRE/EXP00/namelist_cfg ./
The xios context stanza in iodef.xml
was edited to set the using_server variable to true so that the XIOS server will run in a separate process:
<context id="xios">
<variable_definition>
<!-- We must have buffer_size > jpi*jpj*jpk*8 (with jpi and jpj the subdomain size) -->
<variable id="buffer_size" type="integer">10000000</variable>
<variable id="buffer_server_factor_size" type="integer">2</variable>
<variable id="info_level" type="integer">0</variable>
<variable id="using_server" type="boolean">true</variable>
<variable id="using_oasis" type="boolean">false</variable>
<variable id="oasis_codes_id" type="string" >oceanx</variable>
</variable_definition>
</context>
The namelist_cfg
file was edited to add the following 2 lines to the nammpp namelist:
jpni = 3 ! jpni: number of processors following i (set automatically if < 1)
jpnj = 5 ! jpnj: number of processors following j (set automatically if < 1)
to set the MPI decomposition for the run to 3x5.
An MPI app file called nemo.app
was created containing:
-np 15 ./nemo.exe
-np 1 ./xios_server.exe
to define how many cores to run each executable on.
The -np value for nemo.exe
must equal the product of the jpni and jpnj values in the nammpp namelist.
Finally, we’re ready to run the model with:
$ mpirun -app nemo.app >stout 2>stderr
The run produces a single netCDF4 file for the grid_T, grid_U, grid_V, and grid_W variables, and, sadly, 14 per-processor restart files that have to be gathered into a run restart file. The netCDF4 files do not use Limpel-Ziv deflation compression at the variable level so they can be reduces in size by ~50% with commands like:
$ ncks -4 -L4 GYRE_5d_00010101_00011230_grid_T.nc GYRE_5d_00010101_00011230_grid_T.nc
Testing the SalishSea Configuration on jasper¶
Note
This section describes how to manually set up a run directory for the NEMO-3.6 SalishSea configuration and submit the run to the job queue on jasper. The steps described here will eventually be incorporated into the Salish Sea NEMO Command Processor. Once the command processor is capable of working with NEMO-3.6 the recommendation is that it be used rather than doing manual run setups.
After successfully Building on jasper,
create a test directory outside the NEMO-3.6-code
repo:
$ cd $HOME/MEOPAR
$ mkdir NEMO-3.6-test
$ cd NEMO-3.6-test/
Symlink the shared XIOS configurations files, reference namelist, and NEMO & XIOS executables into the test directory:
$ ln -s $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml
$ ln -s $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/namelist_ref
$ ln -s $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SalishSea/BLD/bin/nemo.exe
$ ln -s $HOME/MEOPAR/XIOS/bin/xios_server.exe
Symlink the SalishSea output domain definition file:
$ ln -s $HOME/MEOPAR/SS-run-sets/SalishSea/nemo3.6/domain_def.xml
You also required an iodef.xml
file. A basic one generated for the SalishSea configuration can be copied into the test directory:
$ cp $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SalishSea/EXP00/iodef.xml ./
If you wish, edit that file to adjust the output interval(s) for various variables. A more complicated example, including 1/2 hour tidal-point output can be copied from the tides SS-run-sets instead:
$ cp $HOME/MEOPAR/SS-run-sets/SalishSea/nemo3.6/tides/iodef.xml ./
Create a namelist_cfg
configuration namelist file for the run by running:
$ bash $HOME/MEOPAR/SS-run-sets/SalishSea/nemo3.6/make_namelist_cfg
which concatenates the default namelist section files from MEOPAR/SS-run-sets/SalishSea/nemo3.6/
into a namelist_cfg
file in the present working directory,
then roars like a big cat.
Edit namelist_cfg
to configure your run.
In particular,
set the MPI decomposition values,
jpni and jpnj,
in the nammpp namelist at the end of the file;
e.g.
jpni = 8 ! number of processors in the i direction
jpnj = 18 ! number of processors in the j direction
Symlink the SalishSea domain bathymetry and coordinates files into the test directory:
$ ln -s $HOME/MEOPAR/NEMO-forcing/grid/bathy_meter_SalishSea2.nc bathy_meter.nc
$ ln -s $HOME/MEOPAR/NEMO-forcing/grid/coordinates_seagrid_SalishSea.nc coordinates.nc
Symlink the open boundaries, river, and atmospheric forcing directories into the test directory:
$ ln -s $HOME/MEOPAR/NEMO-forcing/open_boundaries
$ ln -s $HOME/MEOPAR/NEMO-forcing/rivers
$ ln -s /home/sallen/MEOPAR/GEM2.5/ops/NEMO-atmos
Symlink the directory containing the initial stratification files into the test directory. In this case we use results from one of the spin-up runs:
$ ln -s /home/dlatorne/MEOPAR/SalishSea/results/spinup/7dec16dec initial_strat
Create a nemo.pbs
file to define the run environment on jasper and to execute the run:
#!/bin/bash
#PBS -N 8x18+6-regression-1d-6h
#PBS -S /bin/bash
#PBS -l nodes=13:ppn=12
#PBS -l pmem=2000mb
#PBS -l feature=X5675
#PBS -l walltime=0:30:00
#PBS -m bea
#PBS -M dlatornell@eos.ubc.ca
#PBS -o stdout
#PBS -e stderr
cd ${PBS_O_WORKDIR}
echo working dir: $(pwd)
module load library/netcdf/4.1.3
module load library/hdf5/1.8.9
mpirun -np 144 ./nemo.exe : -np 6 ./xios_server.exe
echo done!
Ensure that the MPI decomposition values in your namelist,
The PBS nodes/ppn directive,
and the mpirun statement are consistent and account for the total number of processors required
(NEMO plus XIOS).
Note that jasper gives higher priority to jobs that fully occupy nodes,
but it is not necessary to use all of the processors that you request;
in the nemo.pbs
example above we request 13 nodes with 12 processors each
(156 processors)
but run NEMO with an 8x18 MPI decomposition and 6 XIOS servers
(144 + 6 = 150 processors).
Submit the run to the queue manager:
$ qsub nemo.pbs
Note
One very annoying “feature” of NEMO-3.6 and XIOS is that fatal errors in on seems to cause segmentation fault failures in the other.
For example,
if an input file is missing,
XIOS will fail to read it and the run will fail.
The stderr
file will be full of many uninformative segmentation fault errors and tracebacks that give no clue as to the root cause of the problem but make you think that something is very seriously messed up.
Be sure to also search for E R R O R messages in ocean.output
before you get too worried about the segmentation faults.
Another common mistake that results in a huge list of errors in stderr
is having mismatches in the number of processors among the MPI decomposition specified in the namelist_cfg
,
the PBS directive in your run script,
and the .app
file.