NEMO-code Repository

These notes describe the Salish Sea MEOPAR project NEMO-code repository and its maintenance. They are a narrative guide to the Mercurial log and diffs that can be obtained from the repository itself or via the Bitbucket interface.

The NEMO-code repo is a Mercurial repository in which is maintained the merger of the trunk of the main NEMO svn repository and the changes made by the Salish Sea MEOPAR project team.

Note

The NEMO-code repository is a private repository for members of the Salish Sea MEOPAR project team. That is because it contains parts of the NEMO codebase. Although that codebase is openly licensed its developers require registration to access the code.

If you have completed that registration and would like access to the NEMO-code, please contact Susan Allen, the Salish Sea MEOPAR project leader.

Getting the Code

Team members using SSH key authentication on Bitbucket may clone the NEMO-code repo with:

hg clone ssh://hg@bitbucket.org/salishsea/nemo-code NEMO-code

For password authentication use:

hg clone https://<you>@bitbucket.org/salishsea/nemo-code NEMO-code

where <you> is your Bitbucket user id.

Managing Configurations

Building an Existing Configuration

To build the executable code for an already defined configurations, for example, SalishSea, use:

cd NEMO-code/NEMOGCM/CONFIG
./makenemo -n SalishSea -m salish -j8

That will compile the full domain Salish Sea NEMO configuration and the IOM output server with the salish architecture definitions with the compilation distributed over 8 cores. The resulting executables are located in NEMO-code/NEMOGCM/CONFIG/SalishSea/BLD/bin/.

Available Configurations

In addition to the configurations from the NEMO svn checkout, the repo contains:

  • SalishSea: The initial full domain Salish Sea development configuration

  • SalishSea_TRC: SalishSea above plus passive tracers

  • SalishSea_no_IOM: A copy of the SalishSea configuration compiled without the key_iom_put CPP key

  • SALISH_amm: A very early stepping-stone configuration created during the learning process of creating a Salish Sea configuration based on the AMM12 configuration

  • SALISH_JPP: The Salish Sea sub-domain configuration used to compare with CONCEPTS-110

Available Architecture Definitions

In addition to the collection of architecture definitions that the NEMO consortium provides, the NEMO-code repo includes definitions for:

  • mpif90_jasper: OpenMPI builds on jasper.westgrid.ca

  • ifort_jasper: Intel MPI builds on jasper.westgrid.ca

  • salish: MPI builds on salish.eos.ubc.ca

  • ocean: single processor builds on UBC-EOAS ocean cluster workstations and salish

  • gfortran_osx: single processor builds on Mac-OSX

Creating a New Configuration

To create a new configuration based on, for example, AMM12 use:

cd NEMO-code/NEMOGCM/CONFIG
./makenemo -r AMM12 -n MY_AMM12 -m salish -j8 add_key "key_netcdf4 key_nosignedzero"

That will use the existing AMM12 configuration as a basis to build a new configuration called MY_AMM12 with the salish architecture definitions and with compilation distributed over 8 cores. The C Pre-Processor (CPP) keys key_netcdf4 and key_nosignedzero will be added to configurations. The resulting configuration, including a compiled and link NEMO executable, is located in NEMO-code/NEMOGCM/CONFIG/MY_AMM12.

See ./makenemo -h for details of options and sub-commands.

Running the Model

We don’t want to clutter the NEMO-code repo with files from development and exploration run-sets (aka experiments), run results, etc., so runs are done in directories outside the NEMO-code/ tree.

The Salish Sea Run Sets Files repo contains example run description, namelist, and output server control files that define a run. Run description files define the names and locations of files and directories that the Salish Sea NEMO Command Processor tool uses to manage Salish Sea NEMO runs and their results. See Run Description File Structure for details of the run description file syntax.

After copying and/or editing the run description, namelist, and output server control files to define a run use the salishsea prepare Sub-command to set up a temporary run directory from which to execute the run. For example:

salishsea prepare SalishSea.yaml iodef.xml

Use the run description and output server definitions files names that are appropriate for your run. See the prepare Sub-command docs for details of the files and symbolic links that are created in the run directory. The path to the run directory is printed upon completion of the command.

Go to the run directory and start the run with a command like:

mpiexec -n 16 ./nemo.exe > stdout 2> stderr &

That command runs the model in the background on 16 processors, redirecting the stdout and stderr streams to stdout and stderr files. The number of processors to run is must match the domain decomposition defined in the namelist.compute file.

A convenient command to monitor the memory use of a run and its time step progress is:

watch -n 5 "(free -m; cat time.step)"

The salishsea gather sub-command collects the run definition and results files from a Salish Sea NEMO run into a results directory. The collection process includes combining the per-processors netCDF results files into files in the results directory. It has a number of option flags to define how it works; see salishsea gather -h for details. A typical use on salish is:

salishsea gather --no-compress SalishSea.yaml ../results/15-21Sep2002

The files that define the run, and the non-netCDF results files produced by the run are also moved to the results directory by salishsea gather.

NEMO svn Repo Mirror Maintenance

The /ocean/sallen/hg_repos/NEMO-hg-mirror repository is an svn checkout of http://forge.ipsl.jussieu.fr/nemo/svn/branches/2012/dev_v3_4_STABLE_2012 and also a read-only Mercurial repository. It was initialized with:

$ cd /ocean/sallen/hg_repos
$ svn --username "dlatornell" co -r 3819 http://forge.ipsl.jussieu.fr/nemo/svn/branches/2012/dev_v3_4_STABLE_2012 NEMO-hg-mirror
$ hg init NEMO-hg-mirror
$ cd NEMO-hg-mirror
$ cat > .hgignore
.svn
DOC/NEMO_book.pdf
ctrl-d
$ hg add
$ hg ci -m"Initialize NEMO svn mirror at r3819 of ^/branches/2012/dev_v3_4_STABLE_2012."

svn v1.7.5 was used on salish for the svn part of the initialization.

../../_images/NEMO-CodeReposMaint.png

NEMO code repositories and workflow to update and merge SVN and local changes

Workflow to Pull Changes from NEMO svn Repo

The workflow to pull changes from the master NEMO svn repo and commit them to our NEMO-hg-mirror repo is somewhat automated by the Marlin – Salish Sea NEMO svn-hg Maintenance Tool.

  1. Review the upstream changes in the source browser at https://forge.ipsl.jussieu.fr/nemo/log/branches/2012/dev_v3_4_STABLE_2012 to select a range of changes to be pulled into our NEMO-hg-mirror repo.

    Note

    Pay special attention to changes in the OPA_SRC/ tree that involve files that have been copied into NEMOGCM/CONFIG/SalishSea/MY_SRC/ or team members’ MY_SRC/ directories. Those files must be manually merged with their MY_SRC/ counterparts.

  2. Working on salish in the /ocean/sallen/hg_repos/NEMO-hg-mirror repo with an activated virtualenv in which marlin is installed:

    $ ssh salish
$ workon marlin
(marlin)$ cd /ocean/sallen/hg_repos/NEMO-hg-mirror

  3. Use marlin incoming information about the next SVN revision that will be pulled from upstream and confirm that it is the expected revision:

    (marlin)$ marlin incoming
r3843 2013-03-20 09:29:58 UTC
  correct ice output filename for limwri.F90

    The --limit option can be used to see more incoming revisions; see marlin help incoming for details.

  4. Use marlin update to update the working copy to the next upstream commit and commit the SVN update as a Mercurial changeset with the SVN commit message as the body of the Mercurial commit message and echo that message:

    (marlin)$ marlin update
Update to svn r3843.

correct ice output filename for limwri.F90

    The --to-rev option can be used to apply a series of upstream updates, committing them to Mercurial one at a time; see marlin help update for details.

Workflow to Merge NEMO svn Repo and Salish Sea Revisions

Merging changes from NEMO svn and the Salish Sea central NEMO-code repo on Bitbucket is done in a repo that is used for only that purpose. Doug does the merges on his laptop. The repo in which the merging is done was created by cloning the /ocean/sallen/hg_repos/NEMO-hg-mirror repo:

hg clone ssh://sable.eos.ubc.ca//ocean/sallen/hg_repos/NEMO-hg-mirror NEMO-mirror-merge

and setting the paths in its .hg/hgrc to:

[paths]
bb = ssh://hg@bitbucket.org/salishsea/nemo-code
default-push = ssh://hg@bitbucket.org/salishsea/nemo-code
mirror = ssh://sable.eos.ubc.ca//ocean/sallen/hg_repos/NEMO-hg-mirror

Those paths mean that the repo for hg pull and hg incoming commands must be specified explicitly. The bb and mirror paths are provided to facilitate pulling from NEMO-code on Bitbucket and /ocean/sallen/hg_repos/NEMO-hg-mirror, respectively. hg push and hg outgoing commands will act on the NEMO-code repo, unless otherwise specified.

After the Workflow to Pull Changes from NEMO svn Repo has been completed the workflow to merge those changes with Salish Sea MEOPAR project revisions is:

  1. Pull and update recent changes from NEMO-code into NEMO-mirror-merge:

    cd NEMO-mirror-merge
hg pull --update bb

  2. Pull and update the changes from /ocean/sallen/hg_repos/NEMO-hg-mirror into NEMO-mirror-merge:

    hg pull mirror

  3. Because the changesets pulled from NEMO-code are public a branch merge is necessary:

    hg merge
hg commit -m"Merge svn updates."

  4. Manually merge and commit changes that involve files that have been copied into NEMOGCM/CONFIG/SalishSea/MY_SRC/ or team members’ MY_SRC/ directories. Those files are most likely to be in OPA_SRC/.

  5. Push the result of the updates and merges to NEMO-code:

    hg push bb

    If other users have pushed changes to NEMO-code while merge conflicts were being handled hg pull --rebase can be used to bring in those changes and deal with any additional merge conflicts.

  6. Notify team members of the upstream merge, especially if manual merges of MY_SRC/ files were required, so that they can manage merging changes into any untracked MY_SRC/ files they may have.

Migration to NEMO-3.6

In February 2015 the process of migrating the Salish Sea NEMO model from NEMO-3.4 to NEMO-3.6 was started. A collection of mirror repos similar to that described in NEMO svn Repo Mirror Maintenance was created. NEMO-3.6 uses a separately distributed output server package called XIOS so the maintenance of Mercurial mirror repos for the Salish Sea NEMO project is expanded to deal with 2 upstream SVN repos. For NEMO:

  • /ocean/sallen/hg_repos/NEMO-3.6-hg-mirror

  • /Users/doug/MEOPAR/NEMO-3.6-mirror-merge

  • https://bitbucket.org/salishsea/nemo-3.6-code

and for XIOS:

  • /ocean/sallen/hg_repos/XIOS-hg-mirror

  • /Users/doug/MEOPAR/XIOS-mirror-merge

  • https://bitbucket.org/salishsea/xios

The /ocean/sallen/hg_repos/NEMO-hg-mirror repository is an svn checkout of http://forge.ipsl.jussieu.fr/nemo/svn/trunk and also a read-only Mercurial repository. It was initialized with:

$ cd /ocean/sallen/hg_repos
$ svn --username "dlatornell" co http://forge.ipsl.jussieu.fr/nemo/svn/trunk NEMO-3.6-hg-mirror
$ hg init NEMO-3.6-hg-mirror
$ cd NEMO-3.6-hg-mirror
$ cat > .hgignore
.svn
DOC/NEMO_book.pdf
ctrl-d
$ hg add
$ hg ci -m"Initialize NEMO-3.6 svn mirror at r5072 of ^/trunk."

The /ocean/sallen/hg_repos/XIOS-hg-mirror repository is an svn checkout of http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 and also a read-only Mercurial repository. It was initialized with:

$ cd /ocean/sallen/hg_repos
$ http://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-1.0 XIOS-hg-mirror
$ hg init XIOS-hg-mirror
$ cd XIOS-hg-mirror
$ cat > .hgignore
.svn
ctrl-d
$ hg add
$ hg ci -m"Initialize XIOS svn mirror at r549 of ^/branchs/xios-1.0."

svn v1.8.8 was used on salish for the svn part of the initialization.

On 30-Jun-2015 the v3_6_STABLE release of NEMO was announced. The following process (not quite as simple as suggested in the 3.6 Quick Start Guide) was used to update the /ocean/sallen/hg_repos/NEMO-hg-mirror repository to the HEAD of the http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE branch:

  1. The svn revisions from r5300, through the creation of the ^/branches/2015/nemo_v3_6_STABLE branch, and onward to r5628 were reviewed and annotated in our tracking spreadsheet

  2. The /ocean/sallen/hg_repos/NEMO-hg-mirror repository was Mercurial-tagged with b4_nemo_v3_6_stable.

  3. The svn revisions to r5518 were applied a few at a time using the Marlin – Salish Sea NEMO svn-hg Maintenance Tool and the process described in Workflow to Pull Changes from NEMO svn Repo. Revisions that add or delete files, or copy files in from another svn branch were handled individually, using hg add, or hg rm as necessary to add and remove files, followed by hg commit --amend to update the changeset to reflect that additions/deletions.

  4. The commands:

    $ svn switch http://forge.ipsl.jussieu.fr/nemo/svn/branches/2015/nemo_v3_6_STABLE
$ svn update -r 5519

    were used to change the svn URL to point at the 3.6-stable branch and update to the earliest revision on that branch, r5519

  5. The svn revisions to r5628 were applied a few at a time using the procedure described above for early revisions.

  6. The svn updates were merged with our NEMO-3.6-code and the result tagged as NEMO-3.6r5628.

Initial testing revealed that NEMO-3.6r5628 could not be compiled against XIOS-1.0r548. That issue was resolved by updating and merging XIOS (XML I/O Server) to svn revision r648. The result was confirmed to enable a successful build of NEMO-3.6r5628 and was tagged as XIOS-1.0r648.

The following sections are in-process notes about getting to a running Salish Sea NEMO-3.6 model on various platforms.

Building and Testing XIOS

Building on salish

On salish, XIOS was successfully built with the following arch/arch-* files:

An arch/*.env file is not required for salish.

arch/arch-GCC_SALISH.path:

NETCDF_LIB="-lnetcdff -lnetcdf"
HDF5_LIB="-lz"

arch/arch-GCC_SALISH.fcm:

%CCOMPILER      mpicc
%FCOMPILER      mpif90
%LINKER         mpif90

%BASE_CFLAGS    -ansi -w
%PROD_CFLAGS    -O3 -DBOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g -O2
%DEBUG_CFLAGS   -g

%BASE_FFLAGS    -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2
%DEBUG_FFLAGS   -g

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++

%CPP            cpp
%FPP            cpp -P
%MAKE           make

using the command:

$ ./make_xios --arch GCC_SALISH --netcdf_lib netcdf4_seq --job 8

As salish has only 16 physical cores, running multiple xios_server.exe processes that use parallel output is unnecessary, so the --netcdf_lib netcdf4_seq option is used.

Building on jasper

On jasper, XIOS was successfully built with the following arch/arch-* files:

arch/arch-X64_JASPER.env:

module load library/openmpi/1.6.4-intel
module load library/netcdf/4.1.3
module load library/hdf5/1.8.8

arch/arch-X64_JASPER.path:

NETCDF_LIB="-lnetcdf"
HDF5_LIB="-lhdf5_hl -lhdf5 -lz"

arch/arch-X64_JASPER.fcm:

%CCOMPILER      mpicc
%FCOMPILER      mpif90
%LINKER         mpif90

%BASE_CFLAGS    -diag-disable 1125 -diag-disable 279
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g -traceback
%DEBUG_CFLAGS   -DBZ_DEBUG -g -traceback -fno-inline

%BASE_FFLAGS    -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2 -traceback
%DEBUG_FFLAGS   -g -traceback

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++

%CPP            mpicc -EP
%FPP            cpp -P
%MAKE           gmake

using the command:

$ ./make_xios --arch X64_JASPER --netcdf_lib netcdf4_seq --job 4

At present, jasper lacks the parallel versions of the netCDF4 library that is required to build XIOS so that it produces a single output file, hence the --netcdf_lib netcdf4_seq option.

Building on orcinus

Note

These build notes for orcinus are provisional. They include hard-coded library version numbers and paths for new netCDF and HDF5 library builds that were created in early October 2015. The details below will change when those new libraries have been integrated into modules on orcinus.

On orcinus, XIOS was successfully built with the following arch/arch-* files:

arch/arch-X64_ORCINUS.env:

module load intel

arch/arch-X64_ORCINUS.path:

NETCDF_INC_DIR="/global/software/lib64/intel/ncsa-tools/netcdf-4.3.3.1/include"
NETCDF_INCDIR="-I $NETCDF_INC_DIR"
NETCDF_LIB_DIR="/global/software/lib64/intel/ncsa-tools/netcdf-4.3.3.1/lib"
NETCDF_LIB="-L$NETCDF_LIB_DIR -lirc -lnetcdf"

HDF5_LIB_DIR="/global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1/lib"
HDF5_LIB="-L$HDF5_LIB_DIR -lhdf5_hl -lhdf5 -lz"

arch/arch-X64_ORCINUS.fcm:

%CCOMPILER      mpicc
%FCOMPILER      mpif90
%LINKER         mpif90

%BASE_CFLAGS    -diag-disable 1125 -diag-disable 279
%PROD_CFLAGS    -O3 -D BOOST_DISABLE_ASSERTS
%DEV_CFLAGS     -g -traceback
%DEBUG_CFLAGS   -DBZ_DEBUG -g -traceback -fno-inline

%BASE_FFLAGS    -D__NONE__
%PROD_FFLAGS    -O3
%DEV_FFLAGS     -g -O2 -traceback
%DEBUG_FFLAGS   -g -traceback

%BASE_INC       -D__NONE__
%BASE_LD        -lstdc++ -lirc -shared-intel

%CPP            mpicc -EP
%FPP            cpp -P
%MAKE           make

using the command:

$ ./make_xios --arch X64_ORCINUS --netcdf_lib netcdf4_seq --job 4

Although orcinus has the parallel versions of the netCDF4 library that is required to build XIOS so that it produces a single output file, only the output file per processor has been tested at time of writing, hence the --netcdf_lib netcdf4_seq option.

Testing

The test_client.exe executable that is built with xios_server.exe and the inputs/iodef.xml file can be used to test XIOS. This was done by creating a test-XIOS/ directory, copying inputs/iodef.xml into it, and symlinking xios_server.exe and test_client.exe into it.

In contrast to NEMO-3.4 where the IO server configuration is specified in the xmlio_server.def file, the configuration for XIOS is included as a stanza in iodef.xml. As copied, iodef.xml configures XIOS to run in “attached” mode, similar to how the IO server is used in NEMO-3.4. The relevant stanza is:

<context id="xios">
    <variable_definition>
      <variable_group id="buffer">
          buffer_size = 80000000
          buffer_server_factor_size = 2
       </variable_group>

      <variable_group id="parameters" >
        <variable id="using_server" type="boolean">false</variable>
        <variable id="info_level" type="int">50</variable>
      </variable_group>
    </variable_definition>
</context>

and the line:

<variable id="using_server" type="boolean">false</variable>

sets “attached” mode.

Using qsub to submit a file containing the following shell script with PBS directives runs test_client.exe on 10 processors and produces 10 netCDF4 output files, output_0.nc through output_9.nc:

#!/bin/bash

#PBS -N test-XIOS
#PBS -S /bin/bash
#PBS -l procs=10
#PBS -l walltime=0:10:00
#PBS -m bea
#PBS -M dlatornell@eos.ubc.ca
#PBS -o stdout
#PBS -e stderr

cd $PBS_O_WORKDIR
echo working dir: $(pwd)

module load library/netcdf/4.1.3
module load library/hdf5/1.8.8

mpirun -np 10 ./test_client.exe
echo done!

Changing the XIOS server configuration in iodef.xml to:

<variable id="using_server" type="boolean">true</variable>

creating an MPI application file (let’s call it test-XIOS.app) containing:

-np 8 ./test_client.exe
-np 2 ./xios_server.exe

and submitting a PBS script with the mpirun line changed to:

$ mpirun --app ./test-XIOS.app

results in test_client.exe running on 8 processors and xios_server.exe running on 2 and produces 2 netCDF4 output files, output_0.nc and output_1.nc.

The netCDF4 files that XIOS produces are not deflated. Running:

$ ncks -4 -L4 output_0.nc output_0.nc

on one of the files produces by the above test reduces the file size to 33% or its original size. Note that the present build of NCO on jasper is against the netCDF3 library so it cannot be used to do this deflation.

Building and Testing NEMO-3.6

Building on salish

On salish, NEMO-3.6 was successfully built with the following NEMOGCM/ARCH/ file:

NEMOGCM/ARCH/UBC_EOAS/arch-GCC_SALISH.fcm:

%XIOS_HOME           /data/$USER/MEOPAR/XIOS

%NCDF_INC            -I/usr/include
%NCDF_LIB            -L/usr/lib -lnetcdff -lnetcdf

%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios -lstdc++

%CPP                 cpp
%FC                  mpif90
%FCFLAGS             -cpp -O3 -fdefault-real-8 -funroll-all-loops -fcray-pointer -ffree-line-length-none
%FFLAGS              %FCFLAGS
%LD                  mpif90
%LDFLAGS
%FPPFLAGS            -P -C -traditional
%AR                  ar
%ARFLAGS             -rs
%MK                  make
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

using the commands:

$ cd NEMO-3.6-code/NEMOGCM/CONFIG
$ ./makenemo -n SalishSea -m GCC_SALISH -j4

Important things to note:

  • Our arch files are contained in the institution-specific directory NEMOGCM/ARCH/UBC_EOAS/

  • The %XIOS_HOME build variable uses the USER environment variable to locate the XIOS library to link with NEMO. It is assumed that XIOS is installed and built in /data/userid/MEOPAR/XIOS/ on salish.

  • The -lstdc++ library option must follow -lxios otherwise a truly astonishing volume of unresolved reference errors will be generated and the build will fail.

Building on jasper

On jasper, NEMO-3.6 was successfully built with the following NEMOGCM/ARCH/ file:

NEMOGCM/ARCH/UBC_EOAS/arch-X64_JASPER.fcm:

%NCDF_HOME           /global/software/netcdf/netcdf-4.1.3
%HDF5_HOME           /global/software/hdf5/hdf5-1.8.9
%XIOS_HOME           $HOME/MEOPAR/XIOS

%NCDF_INC            -I%NCDF_HOME/include
%NCDF_LIB            -L%NCDF_HOME/lib -lnetcdff -lnetcdf -L%HDF5_HOME/lib -lhdf5_hl -lhdf5 -lhdf5
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios

%CPP                 cpp
%FC                  mpif90
%FCFLAGS             -c -fpp -r8 -O3 -assume byterecl -convert big_endian -heap-arrays
%FFLAGS              %FCFLAGS
%LD                  mpif90
%LDFLAGS             -lstdc++
%FPPFLAGS            -P -C -traditional
%AR                  ar
%ARFLAGS             -r
%MK                  make
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

using the commands:

$ cd NEMO-3.6-code/NEMOGCM/CONFIG
$ ./makenemo -n SalishSea -m X64_JASPER -j8

Important things to note:

  • Our arch files are contained in the institution-specific directory NEMOGCM/ARCH/UBC_EOAS/

  • The %XIOS_HOME build variable uses the HOME environment variable to locate the XIOS library to link with NEMO. It is assumed that XIOS is installed and built in $HOME/MEOPAR/XIOS/ on jasper.

Building on orcinus

Note

These build notes for orcinus are provisional. They include hard-coded library version numbers and paths for new netCDF and HDF5 library builds that were created in early October 2015. The details below will change when those new libraries have been integrated into modules on orcinus.

On orcinus, NEMO-3.6 was successfully built with the following NEMOGCM/ARCH/ file:

NEMOGCM/ARCH/UBC_EOAS/arch-X64_ORCINUS.fcm:

%NCDF_HOME           /global/software/lib64/intel/ncsa-tools/netcdf-4.3.3.1
%HDF5_HOME           /global/software/lib64/intel/ncsa-tools/hdf5-1.8.15p1
%XIOS_HOME           $HOME/MEOPAR/XIOS

%NCDF_INC            -I%NCDF_HOME/include -I /global/software/lib64/intel/ncsa-tools/netcdf-fortran-4.4.0/include
%NCDF_LIB            -L%NCDF_HOME/lib -lnetcdf -L /global/software/lib64/intel/ncsa-tools/netcdf-fortran-4.4.0/lib -lnetcdff -L%HDF5_HOME/lib -lhdf5_hl -lhdf5
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios

%CPP               cpp
%FC                  mpif90
%FCFLAGS             -c -fpp -r8 -O3 -assume byterecl -heap-arrays
%FFLAGS              %FCFLAGS
%LD                  mpif90
%LDFLAGS             -lstdc++ -shared-intel
%FPPFLAGS            -P -C -traditional
%AR                  ar
%ARFLAGS             rcs
%MK                  make
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

using the commands:

$ cd $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SalishSea
$ source orcinus_build.sh

Important things to note:

  • Our arch files are contained in the institution-specific directory NEMOGCM/ARCH/UBC_EOAS/

  • The %XIOS_HOME build variable uses the HOME environment variable to locate the XIOS library to link with NEMO. It is assumed that XIOS is installed and built in $HOME/MEOPAR/XIOS/ on orcinus.

The rebuild_nemo.exe tool should also be built with the commands:

Testing the GYRE Configuration on salish

A new NEMO-3.6 GYRE configuration was created on salish with:

$ cd NEMOGCM/CONFIG/
$ ./makenemo -n my_GYRE -r GYRE -m GCC_SALISH -j8

After the build completed successfully, a test directory was created outside the NEMO-3.6-code repo:

$ cd /data/dlatorne/MEOPAR
$ mkdir GYRE-3.6-test
$ cd GYRE-3.6-test/

and the shared XIOS configurations files, reference namelist, and executables were symlinked into the test directory:

$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/domain_def.xml
$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml
$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/namelist_ref
$ ln -s /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/my_GYRE/BLD/bin/nemo.exe
$ ln -s /data/dlatorne/MEOPAR/XIOS/bin/xios_server.exe

The iodef.xml and namelist_cfg files generated for the my_GYRE configuration were copied into the test directory:

$ cp /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/my_GYRE/EXP00/iodef.xml ./
$ cp /data/dlatorne/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/my_GYRE/EXP00/namelist_cfg ./

The xios context stanza in iodef.xml was edited to set the using_server variable to true so that the XIOS server will run in a separate process:

 <context id="xios">
   <variable_definition>
     <!-- We must have buffer_size > jpi*jpj*jpk*8 (with jpi and jpj the subdomain size) -->
     <variable id="buffer_size" type="integer">10000000</variable>
     <variable id="buffer_server_factor_size" type="integer">2</variable>
     <variable id="info_level" type="integer">0</variable>
     <variable id="using_server" type="boolean">true</variable>
     <variable id="using_oasis" type="boolean">false</variable>
     <variable id="oasis_codes_id" type="string" >oceanx</variable>
   </variable_definition>
 </context>

The namelist_cfg file was edited to add the following 2 lines to the nammpp namelist:

jpni = 3  ! jpni: number of processors following i (set automatically if < 1)
jpnj = 5  ! jpnj: number of processors following j (set automatically if < 1)

to set the MPI decomposition for the run to 3x5.

An MPI app file called nemo.app was created containing:

-np 15 ./nemo.exe
-np 1 ./xios_server.exe

to define how many cores to run each executable on. The -np value for nemo.exe must equal the product of the jpni and jpnj values in the nammpp namelist.

Finally, we’re ready to run the model with:

$ mpirun -app nemo.app >stout 2>stderr

The run produces a single netCDF4 file for the grid_T, grid_U, grid_V, and grid_W variables, and, sadly, 14 per-processor restart files that have to be gathered into a run restart file. The netCDF4 files do not use Limpel-Ziv deflation compression at the variable level so they can be reduces in size by ~50% with commands like:

$ ncks -4 -L4 GYRE_5d_00010101_00011230_grid_T.nc GYRE_5d_00010101_00011230_grid_T.nc

Testing the SalishSea Configuration on jasper

Note

This section describes how to manually set up a run directory for the NEMO-3.6 SalishSea configuration and submit the run to the job queue on jasper. The steps described here will eventually be incorporated into the Salish Sea NEMO Command Processor. Once the command processor is capable of working with NEMO-3.6 the recommendation is that it be used rather than doing manual run setups.

After successfully Building on jasper, create a test directory outside the NEMO-3.6-code repo:

$ cd $HOME/MEOPAR
$ mkdir NEMO-3.6-test
$ cd NEMO-3.6-test/

Symlink the shared XIOS configurations files, reference namelist, and NEMO & XIOS executables into the test directory:

$ ln -s $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/field_def.xml
$ ln -s $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SHARED/namelist_ref
$ ln -s $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SalishSea/BLD/bin/nemo.exe
$ ln -s $HOME/MEOPAR/XIOS/bin/xios_server.exe

Symlink the SalishSea output domain definition file:

$ ln -s $HOME/MEOPAR/SS-run-sets/SalishSea/nemo3.6/domain_def.xml

You also required an iodef.xml file. A basic one generated for the SalishSea configuration can be copied into the test directory:

$ cp $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG/SalishSea/EXP00/iodef.xml ./

If you wish, edit that file to adjust the output interval(s) for various variables. A more complicated example, including 1/2 hour tidal-point output can be copied from the tides SS-run-sets instead:

$ cp $HOME/MEOPAR/SS-run-sets/SalishSea/nemo3.6/tides/iodef.xml ./

Create a namelist_cfg configuration namelist file for the run by running:

$ bash $HOME/MEOPAR/SS-run-sets/SalishSea/nemo3.6/make_namelist_cfg

which concatenates the default namelist section files from MEOPAR/SS-run-sets/SalishSea/nemo3.6/ into a namelist_cfg file in the present working directory, then roars like a big cat.

Edit namelist_cfg to configure your run. In particular, set the MPI decomposition values, jpni and jpnj, in the nammpp namelist at the end of the file; e.g.

jpni        =   8       !  number of processors in the i direction
jpnj        =  18       !  number of processors in the j direction

Symlink the SalishSea domain bathymetry and coordinates files into the test directory:

$ ln -s $HOME/MEOPAR/NEMO-forcing/grid/bathy_meter_SalishSea2.nc bathy_meter.nc
$ ln -s $HOME/MEOPAR/NEMO-forcing/grid/coordinates_seagrid_SalishSea.nc coordinates.nc

Symlink the open boundaries, river, and atmospheric forcing directories into the test directory:

$ ln -s $HOME/MEOPAR/NEMO-forcing/open_boundaries
$ ln -s $HOME/MEOPAR/NEMO-forcing/rivers
$ ln -s /home/sallen/MEOPAR/GEM2.5/ops/NEMO-atmos

Symlink the directory containing the initial stratification files into the test directory. In this case we use results from one of the spin-up runs:

$ ln -s /home/dlatorne/MEOPAR/SalishSea/results/spinup/7dec16dec initial_strat

Create a nemo.pbs file to define the run environment on jasper and to execute the run:

#!/bin/bash

#PBS -N 8x18+6-regression-1d-6h
#PBS -S /bin/bash
#PBS -l nodes=13:ppn=12
#PBS -l pmem=2000mb
#PBS -l feature=X5675
#PBS -l walltime=0:30:00
#PBS -m bea
#PBS -M dlatornell@eos.ubc.ca
#PBS -o stdout
#PBS -e stderr

cd ${PBS_O_WORKDIR}
echo working dir: $(pwd)

module load library/netcdf/4.1.3
module load library/hdf5/1.8.9

mpirun -np 144 ./nemo.exe : -np 6 ./xios_server.exe
echo done!

Ensure that the MPI decomposition values in your namelist, The PBS nodes/ppn directive, and the mpirun statement are consistent and account for the total number of processors required (NEMO plus XIOS). Note that jasper gives higher priority to jobs that fully occupy nodes, but it is not necessary to use all of the processors that you request; in the nemo.pbs example above we request 13 nodes with 12 processors each (156 processors) but run NEMO with an 8x18 MPI decomposition and 6 XIOS servers (144 + 6 = 150 processors).

Submit the run to the queue manager:

$ qsub nemo.pbs

Note

One very annoying “feature” of NEMO-3.6 and XIOS is that fatal errors in on seems to cause segmentation fault failures in the other. For example, if an input file is missing, XIOS will fail to read it and the run will fail. The stderr file will be full of many uninformative segmentation fault errors and tracebacks that give no clue as to the root cause of the problem but make you think that something is very seriously messed up. Be sure to also search for E R R O R messages in ocean.output before you get too worried about the segmentation faults.

Another common mistake that results in a huge list of errors in stderr is having mismatches in the number of processors among the MPI decomposition specified in the namelist_cfg, the PBS directive in your run script, and the .app file.