Working on computecanada: NEMO v3.6

This section describes the steps to set up and run the Salish Sea NEMO version 3.6 code on the ComputeCanada machines graham.computecanada.ca and cedar.computecanada.ca.

This guide assumes that your Working Environment is set up and that you are familiar with Working on salish : NEMO v3.6.

Modules setup

When working on the ComputeCanada clusters, the module load command must be used to load extra software components.

You can manually load the modules each time you log in, or you can add the lines to your $HOME/.bashrc file so that they are automatically loaded upon login.

At present, beluga, cedar, and graham are configured similarly. The modules needed are:

module load netcdf-fortran-mpi/4.5.2
module load perl/5.30.2
module load python/3.8.2

Create a Workspace and Clone the Repositories

cedar and graham provide several different types of file storage. We use project space for our working environments because it is large, high performance, and backed up. Scratch space is even larger, also high performance, but not backed up, so we use that as the space to execute NEMO runs in, but generally move the run results to project space.

Both systems provide environment variables that are more convenient that remembering full paths to access your project and scratch spaces:

  • Your project space is at $PROJECT/$USER/

  • Your scratch space is at $SCRATCH/

  • Daily atmospheric, river, and west boundary forcing files are in the $PROJECT/SalishSea/forcing/ tree

Create a MEOPAR/ directory tree in your project space:

beluga

mkdir -p ~/projects/def-allen/$USER/MEOPAR/SalishSea/results

cedar

mkdir -p $PROJECT/$USER/MEOPAR/SalishSea/results

graham

mkdir -p $PROJECT/$USER/MEOPAR/SalishSea/results

Clone the repos needed to run the model:

cd $PROJECT/$USER/MEOPAR
git clone git@github.com:SalishSeaCast/grid.git
git clone git@github.com:SalishSeaCast/NEMO-Cmd.git
git clone git@github.com:SalishSeaCast/SalishSeaCmd.git
git clone git@github.com:SalishSeaCast/tides.git
git clone git@github.com:SalishSeaCast/tracers.git
git clone git@github.com:SalishSeaCast/rivers-climatology.git
git clone git@github.com:SalishSeaCast/NEMO-3.6-code.git
git clone git@github.com:SalishSeaCast/XIOS-ARCH.git
git clone git@github.com:SalishSeaCast/XIOS-2.git
git clone git@github.com:SalishSeaCast/SS-run-sets.git

Install the Command Processor Packages

Install the NEMO Command Processor and SalishSeaCast NEMO Command Processor Python packages:

cd $PROJECT/$USER/MEOPAR/
python3 -m pip install --user --editable NEMO-Cmd
python3 -m pip install --user --editable SalishSeaCmd

Compile XIOS-2

Please see the XIOS-2 section of the UBC EOAS MOAD Group Documentation.

Compile NEMO-3.6

Compile the SalishSeaCast NEMO configuration and link it to XIOS-2, and compile the rebuild_nemo tool. The NEMO ARCH files use the XIOS_HOME environment variable to find the XIOS-2 library you built above. XIOS_HOME must be an absolute path to your XIOS-2 clone directory. You can set XIOS_HOME on the command-line before the makenemo and maketools commands as shown below, or you can set and export the value of XIOS_HOME in your $HOME/.bashrc file.

beluga

cd $PROJECT/$USER/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2/ ./makenemo -n SalishSeaCast -m X64_BELUGA -j 8
cd $PROJECT/$USER/MEOPAR/NEMO-3.6-code/NEMOGCM/TOOLS
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2/ ./maketools -n REBUILD_NEMO -m X64_BELUGA

cedar

cd $PROJECT/$USER/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2/ ./makenemo -n SalishSeaCast -m X64_CEDAR -j 8
cd $PROJECT/$USER/MEOPAR/NEMO-3.6-code/NEMOGCM/TOOLS
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2/ ./maketools -n REBUILD_NEMO -m X64_CEDAR

graham

cd $PROJECT/$USER/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2/ ./makenemo -n SalishSeaCast -m X64_GRAHAM -j 8
cd $PROJECT/$USER/MEOPAR/NEMO-3.6-code/NEMOGCM/TOOLS
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2/ ./maketools -n REBUILD_NEMO -m X64_GRAHAM

To build a configuration other than SalishSeaCast, replace SalishSeaCast with the name of the configuration to be built, e.g. SMELT:

cd $PROJECT/$USER/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
XIOS_HOME=$PROJECT/$USER/MEOPAR/XIOS-2/ ./makenemo -n SMELT -m X64_CEDAR -j 8

Prepare and Execute Runs

The SS-run-sets/v201702/ directory in the SalishSea/ Directory repo contains version controlled sample run description files and namelist segment files. In your own directory in that repo copy, edit, and version control those files to define the runs that you want to execute.

The run description file is described in the Run Description File Structure section of the project tools documentation. The namelists are described in the NEMO-3.6 Book.

Please see the sections below for details of using forcing climatology and shared daily forcing files in your runs, and examples of run description and namelist file sections.

Use salishsea run Sub-command to prepare, execute, and gather the results for a run:

salishsea run SalishSea.yaml $PROJECT/$USER/MEOPAR/SalishSea/results/my_excellent_results

salishsea run returns the path and name of the temporary run directory, and the job identifier assigned by the queue manager, something like:

salishsea_cmd.run INFO: Created run directory /scratch/dlatorne/20mar17nowcast16x34_2017-10-06T101548.694389-0700
salishsea_cmd.run INFO: Submitted batch job 1578481

salishsea run has a number of command-line option flags that are useful for controlling details of how runs are executed, and for debugging your YAML files and the symlinks in the temporary run directory. Please see salishsea help run or the SalishSeaCmd package docs.

You can use the batch job number with squeue --job and sacct --job to monitor the execution status of your job.

When the job completes the results should have been gathered in the directory you specified in the salishsea run command and the temporary run directory should have been deleted.

To view and analyze the run results copy them to your EOAS /data/$USER/results/ workspace with scp, sftp or rsync.

Forcing Climatology and Daily Files

Model runs use a mixture of climatologies and daily forcing from other operational models or observations:

  • Tides are, by definition, climatological.

  • Most of the river run-offs are climatological, but daily average discharge and turbidity for the Fraser River may also be used.

  • Tracers at the northern boundary in Johnstone Strait are climatological. At the western boundary at the mouth of the Juan de Fuca Strait we have hourly tracer fields from the University of Washington LiveOcean model since 4-Feb-2017 as well as climatologies.