Working on `salish`

This section describes very briefly the steps to set up and run the Salish Sea NEMO code. Details of what this all means and why the steps below are what they are can be found in subsequent sections.

Create a Workspace

You can call your workspace directory whatever you want but for consistency across systems we’re going to call it MEOPAR.

salish has a large local storage partition mounted at /data/ which is where we will put the code and run configuration file repos:

mkdir -p /data/$USER/MEOPAR

Create directories for the temporary directories that are created to run the model, and to receive results files from runs. We’ll call them SalishSea/ and SalishSea/results/:

mkdir -p /data/$USER/MEOPAR/SalishSea/results

The ocean.eos.ubc.ca storage storage server space that you use on your Waterhole workstation is mounted at /ocean/ on salish so you should be able to see your Salish Sea Repos and Packages at /ocean/$USER/MEOPAR/.

You may want to open an EOAS help desk ticket requesting that the salish /data/ partition be mounted on your Waterhole workstation so that you can easily view and copy files when you are not logged into salish without having to use scp or sftp.

Clone the Repos

Assuming that you are using SSH key authentication on Bitbucket (see Version Control), clone the NEMO-3.4-Code, NEMO-forcing, and SS-run-sets repos into your workspace on /data/:

cd /data/$USER/MEOPAR/
hg clone ssh://hg@bitbucket.org/salishsea/nemo-code NEMO-code
hg clone ssh://hg@bitbucket.org/salishsea/nemo-forcing NEMO-forcing
hg clone ssh://hg@bitbucket.org/salishsea/ss-run-sets SS-run-sets

Compile the Code

Compile and link the full domain Salish Sea NEMO configuration and the IOM output server with the salish architecture definitions with the compilation distributed over 8 cores.

cd NEMO-code/NEMOGCM/CONFIG
./makenemo -n SalishSea -m salish -j8

The resulting executables are located in NEMO-code/NEMOGCM/CONFIG/SalishSea/BLD/bin/.

Compile and link the rebuild_nemo tool:

cd NEMO-code/NEMOGCM/TOOLS
./maketools -m salish -n REBUILD_NEMO

See REBUILD_NEMO for more information about it.

Prepare and Execute Runs

The SS-run-sets/ SalishSea/ Directory contains version controlled sample run description files and namelist segment files. Create your own directory in that repo where you can copy, edit, and version control those files to define the run that you want to execute.

The run description file is described in the Run Description File Structure section of the project tools documentation. The namelists are described in the “NEMO-3.4 Book” (https://www.nemo-ocean.eu/content/download/21612/97924/file/NEMO_book_3_4.pdf).

Use salishsea run Sub-command to prepare, execute, and gather the results for a run:

salishsea run --nemo3.4 SalishSea.yaml iodef.xml /data/$USER/MEOPAR/SalishSea/results/my_excellent_results

salishsea run returns the path and name of the temporary run directory, and the job identifier assigned by the queue manager, something like:

salishsea_cmd.prepare INFO: Created run directory ../../SalishSea/38e87e0c-472d-11e3-9c8e-0025909a8461
salishsea_cmd.run INFO: 57.master

salishsea run has a number of command-line option flags that are useful for controlling details of how runs are executed, and for debugging your YAML files and the symlinks in the temporary run directory. Please see salishsea help run or the SalishSeaCmd package docs.

You can use the qstat command to monitor the execution status of your job.

A convenient command to monitor the memory use of a run and its time step progress is:

watch -n 5 "(free -m; cat time.step)"

When the job completes the results should have been gathered in the directory you specified in the salishsea run command and the temporary run directory should have been deleted.

Look at the Results

A number of notebooks that look at NetCDF files are available in analysis/analysis_tools/. To start these, go to the top level directory of the analysis repo on your local machine (not on salish) and type:

jupyter notebook

At this points alot of information will appear in your terminal. This terminal session is now running a server and cannot be used for anything else until you are finished with the notebooks. At that point you need to CTRL-C to get out.

At the same time a window should open in your browser. If it doesn’t, look at your terminal, find the ip address, something like:

The Jupyter Notebook is running at: http://127.0.0.1:8888/

and put that number in your browser. From this initial window you can open the notebooks in analysis_tools directory and look around. The links to the various files will probably not work. Change them to point to your file space. You will probably want to build your own notebook but these notebooks give you lots of examples to copy from.

Working on salish