Working on westgrid: NEMO v3.6

This section describes the steps to set up and run the Salish Sea NEMO version 3.6 code on the WestGrid machine orcinus.westgrid.ca.

This guide assumes that your Working Environment is set up and that you are familiar with Working on salish : NEMO v3.6.

Modules setup

Similar to when working on ComputeCandada clusters the module load command must be used to load extra software components on orcinus.

You can manually load the modules each time you log in, or you can add the lines to your .bashrc file so that they are automatically loaded upon login.

The modules are:

module load python
module load intel/14.0.2
module load intel/14.0/netcdf-4.3.3.1_mpi
module load intel/14.0/netcdf-fortran-4.4.0_mpi
module load intel/14.0/hdf5-1.8.15p1_mpi
module load intel/14.0/nco-4.5.2

PATH Additions

Lastly, you need to modify your search path such that your shell can find python scripts installed with pip --user. Change the lpath line in the modify search path section of .bash_profile to include $HOME/.local/bin and $HOME/bin in your search path:

lpath=$HOME/.local/bin:$HOME/bin

Create a Workspace and Clone the Repositories

mkdir -p $HOME/MEOPAR/SalishSea/results

Clone the repos needed to run the model:

cd $HOME/MEOPAR
git clone git@github.com:SalishSeaCast/grid.git
git clone git@github.com:SalishSeaCast/tides.git
git clone git@github.com:SalishSeaCast/tracers.git
git clone git@github.com:SalishSeaCast/rivers-climatology.git
git clone git@github.com:SalishSeaCast/NEMO-3.6-code.git
git clone git@github.com:SalishSeaCast/XIOS-ARCH.git
git clone git@github.com:SalishSeaCast/XIOS-2.git
git clone git@github.com:SalishSeaCast/SS-run-sets.git
git clone git@github.com:SalishSeaCast/NEMO-Cmd.git
git clone git@github.com:SalishSeaCast/SalishSeaCmd.git

There is no need to clone the docs or analysis repos at WestGrid.

Install Tools and Command Processor Packages

Install the SalishSeaTools Package and SalishSeaCast NEMO Command Processor Python packages:

mkdir -p $HOME/.local
cd $HOME/MEOPAR/
python3 -m pip install --user --editable NEMO-Cmd
python3 -m pip install --user --editable SalishSeaCmd

Compile XIOS-2

Please see the XIOS-2 section of the UBC EOAS MOAD Group Documentation.

Compile NEMO-3.6

Compile the SalishSea NEMO configuration and link it to XIOS-2, and compile the rebuild_nemo tool. The NEMO ARCH files use the XIOS_HOME environment variable to find the XIOS-2 library you built above. XIOS_HOME must be an absolute path to your XIOS-2 clone directory. You can set XIOS_HOME on the command-line before the makenemo and maketools commands as shown below, or you can set and export the value of XIOS_HOME in your $HOME/.bashrc file.

cd $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
XIOS_HOME=$HOME/MEOPAR/XIOS-2/ ./makenemo -n SalishSea -m X64_ORCINUS -j 8
cd $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/TOOLS
XIOS_HOME=$HOME/MEOPAR/XIOS-2/ ./maketools -n REBUILD_NEMO -m X64_ORCINUS

To build a configuration other than SalishSea, replace SalishSea with the name of the configuration to be built, e.g. SMELT:

cd $HOME/MEOPAR/NEMO-3.6-code/NEMOGCM/CONFIG
XIOS_HOME=$HOME/MEOPAR/XIOS-2/ ./makenemo -n SMELT -m X64_ORCINUS -j 8

Prepare and Execute Runs

The SS-run-sets/v201702/ directory in the SalishSea/ Directory repo contains version controlled sample run description files and namelist segment files. In your own directory in that repo copy, edit, and version control those files to define the runs that you want to execute.

The run description file is described in the Run Description File Structure section of the project tools documentation. The namelists are described in the NEMO-3.6 Book.

Use salishsea run Sub-command to prepare, execute, and gather the results for a run:

salishsea run SalishSea.yaml $HOME/MEOPAR/SalishSea/results/my_excellent_results

salishsea run returns the path and name of the temporary run directory, and the job identifier assigned by the queue manager, something like:

salishsea_cmd.prepare INFO: Created run directory /home/dlatorne/MEOPAR/SalishSea/a90d391c-0e1e-11e4-aa4e-6431504adba6
salishsea_cmd.run INFO: 3544250.orca2.ibb

salishsea run has a number of command-line option flags that are useful for controlling details of how runs are executed, and for debugging your YAML files and the symlinks in the temporary run directory. Please see salishsea help run or the SalishSeaCmd package docs.

You can use the job identifier with qstat, showstart, and checkjob to monitor the execution status of your job.

When the job completes the results should have been gathered in the directory you specified in the salishsea run command and the temporary run directory should have been deleted.

To view and analyze the run results copy them to your EOAS /data/$USER/results/ workspace with scp, sftp or rsync.