Notes on the Oct-2013
These are notes on building and running the version of NEMO configured for the Salish Sea as received in the
CODE.tar archive prepared by J-P Paquin and downloaded at UBC on 2-Oct-2013.
The “Notes on NEMO/OPA Usage” document in the
DOCUMENTATION.tar archive provided guidance for the creation of the notes below.
Add links to “Notes on NEMO/OPA Usage” .doc and PDF
CODE.tar tarball unpacks to a directory called
CONCEPTS110_WCSD_OW_NOBC_tide which apparently indicates:
CONCEPTS110: Canadian Operational Network of Coupled Environmental PredicTion Systems model at the version 110 level
WCSD: MEOPAR West Coast Sub-Domain configuration
OW: Open West boundary
NOBC: ?? maybe refers to reading OBC file data instead of initial conditions ??
tide: focus of the model runs is to calculate tides in the domain
as bash script that runs the build tool chain described below,
modipsl is the framework that was used to obtain and build the NEMO code prior to NEMO up to v3.2. modipsl is available via anonymous svn checkout from http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/. However, it is unclear what revision/tag of modipsl the tarball contains.
The documented (in “Notes on NEMO/OPA Usage” and in the NEMO Quick Start Guide) means of obtaining the NEMO code is to run:
cd modipsl/util model NEMO
which automates the process of doing checkouts of code from a collection of cvs and svn repositories.
Examination of the
modipsl/util/log file that the
modipsl/util/model script creates and the the
modipsl/util/mod.def definitions file for the script indicate that
model was run on 18-Mar-2010 and that the nemo_v3_1 tag was used to obtain the NEMO code.
The build process described here was attempted on several platforms with the following results:
MacBook running OS/X 10.8.5 with Xcode installed:
fait_AA_makereported sed: RE error: illegal byte sequence numerous times,
fait_AA_makefailed because ksh was not installed
jasper: build was successful
The build NEMO 3.1 for a new configuration the following steps are required:
modeles/NEMO/OPA_SRC/par_oce.F90must be edited to set the number of processors to be used, and include configuration parameters via a
.h90file. The edits are done in C-preprocessor (CPP) blocks. J-P Paquin did this, using the CPP key key_wc3 to specify an 8x8 processor layout and parameters in
par_WC3.h90. The edits are flagged with comments that start with !JPP.
modeles/NEMO/OPA_SRC/par_ORCA_R2.h90, create the configuration parameters header file included via the new CPP key in
modeles/NEMO/OPA_SRC/par_oce.F90. The file created by J-P Paquin is
modeles/UTIL/fait_configto define the NEMO source files that are required for the build. The configuration name, WC3 in the present case, must be added to the LIST shell variable, a newline-separated list of configuration names. A set -A statement must also be added to the collection that follows the LIST. The configuration is referenced in the set -A statement as DIR_configname; i.e. DIR_WC3 for the present case. The set -A statement lists the source code directories that are to be included in the build. Those edits are present in
cd modeles/UTIL ./fait_config WC3
That results in the creation of the
modeles/NEMO/WORK/directory in which the source files from the directories given in the set -A DRI_WC3 ... statement in
fait_configare symlinked so as to appear to be all in the same directory. The
modipsl/config/WC3/directory is also created. Its
scripts/directory contains the
BB_make.ldefto activate/deactivate CPP keys for the configuration. The keys are listed in a single, space-separated line as the value of the P_P variable.
CODE.tarcontains three P_P lists identified with #-- JPP comments. The uncommented one (and so presumably most recently used) is:
#-- JPP 20130717 Run compilation keys for TEST1 P_P = key_wc3 key_dtatem key_dtasal key_flx_core key_vvl key_zrefsurf key_zdftke key_traldf_c2d key_dynldf_c3d key_mpp_mpi key_ldfslp key_dynspg_ts2 key_dtatem_month key_dtasal_month key_obc_mer key_tide key_diaharm
Also ensure that there is a prefix for preprocessing line for the build/run target, for example:
#-Q- jasper prefix = -D
Ensure that there is an appropriate set of definitions in
modipsl/util/AA_make.gdeffor the build/run target, for example:
#-Q- jasper #- Global definitions for jasper.westgrid.ca using Linux Compiler Intel v8 #-Q- jasper LIB_MPI = MPI2 #-Q- jasper LIB_MPI_BIS = #-Q- jasper M_K = make #-Q- jasper P_C = cpp #-Q- jasper P_O = -P -C -traditional $(P_P) #-Q- jasper F_C = mpiifort -c -fpp #-Q- jasper #-D- MD F_D = -g #-Q- jasper #-D- MN F_D = #-Q- jasper #-P- I4R4 F_P = -i4 #-Q- jasper #-P- I4R8 F_P = -i4 -r8 #-Q- jasper #-P- I8R8 F_P = -i8 -r8 #-Q- jasper #-P- ?? F_P = -i4 -r8 #-Q- jasper F_O = -O3 $(F_P) -I$(MODDIR) -I$(MODDIR)/oce -module $(MODDIR) -assume byterecl -convert big_endian -I $(NCDF_INC) #-Q- jasper F_F = $(F_O) -extend_source #-Q- jasper F_L = mpiifort #-Q- jasper L_O = #-Q- jasper A_C = ar -r #-Q- jasper A_G = ar -x #-Q- jasper C_C = gcc -c #-Q- jasper C_O = #-Q- jasper C_L = gcc #-Q- jasper #- #-Q- jasper NCDF_INC = -I/lustre/jasper/software/netcdf/netcdf-4.1.3/include -lhdf5_hl -lhdf5 -lz -lsz #-Q- jasper NCDF_LIB = -L/lustre/jasper/software/netcdf/netcdf-4.1.3/lib -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz
cd modeles/NEMO ../UTIL/fait_AA_make
to calculate compilation rules, options, and build dependencies so as to create
NEMO/WORK/AA_make(which is symlinked to
fait_AA_makemust be run from the
cd modipsl/util ./clr_make ./ins_make -t target
to remove existing
Makefilesand create new ones. The target argument to
ins_makespecifies a compiler or host name defined in
cd modipsl/config/WC3 make clean make
to compile and link the code.
The results of a successful build are:
With the CPP keys above in
BB_make.ldef the make command on jasper completes with these messages:
dynadv_ppm.F90(76): warning #6843: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value. [PHTRA_ADV] SUBROUTINE adv_ppm_hor ( kt, pun, pvn, tra, traa, phtra_adv, z2, sort ) -----------------------------------------------------^ ./ldfdyn_c3d.h90(148): remark #8291: Recommended relationship between field width 'W' and the number of fractional digits 'D' in this edit descriptor is 'W>=D+7'. IF(lwp) WRITE(numout,'(34x,E7.2,8x,i3)') zcoef(jk) * ahm0, jk -------------------------------------------^ dynzdf_imp.F90(20): remark #6536: All symbols from this module are already visible due to another USE; the ONLY clause will have no effect. Rename clauses, if any, will be honored. [OCE] USE oce ! ocean dynamics and tracers -------^ The library is up-to-date mpiifort -o ../../../bin/opa model.o ../../../lib/oce/libopa.a ../../../lib/libioipsl.a -L/lustre/jasper/software/netcdf/netcdf-4.1.3/lib -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz /lustre/jasper/software/intel/l_ics_2012.0.032/composer_xe_2011_sp1.10.319/compiler/lib/intel64/libimf.so: warning: warning: feupdateenv is not implemented and will always fail OPA model is OK
WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY.tar tarball contain the
namelist and scripts to setup and run on the BIO HPC cluster:
linkfile.shlinks the intial conditions, forcing, etc. files into the run directory with the file names that NEMO expects
namelistis the NEMO namelist for the run
submit_64.shis the file containing PBS directives and shell commands that is submitted to the TORQUE resource manager via qsub
The meaning of WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY
WCSD : West Coast Sub Domain (398x345) M2 : Run with only M2 tides from WebTide OW_ON : Open West & OpenNorth boundaries file : reading OBC file (not initial conditions) DAMP : increased horizontal eddy viscosity ANALY : Analytical forcing (namsbc_ana) - no atm-ocean fluxes or atmospheric forcing
WCSD_PREP.tar tarball contains the intial conditions,
files for the WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY case.
With those two tarballs unpacked beside each other one the dirPREP variable in
linkfile.sh need to be set to:
linkfile.sh run in
WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY to prepare for the run.
submit_64.sh is tailored to the BIO HPC cluster.
To run on jasper,
the following script was used:
#!/bin/bash #PBS -N WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY #PBS -S /bin/bash #PBS -l procs=64 # memory per processor #PBS -l pmem=2gb #PBS -l walltime=1:00:00 # email when the job [b]egins and [e]nds, or is [a]borted #PBS -m bea #PBS -M email@example.com #PBS -o OPA.output #PBS -e OPA.output.error cd $PBS_O_WORKDIR echo working dir: $(pwd) module load compiler/intel/12.1 module load library/intelmpi/4.0.3.008 module load library/netcdf/4.1.3 module load library/szip/2.1 mpiexec ./opa
If that script is stored as
a run is submitted with the command:
As an initial test, the run duration was set to 720 time steps via the &namrun.nitend namelist item. The run completed in just over 2 minutes. A subsequent 4320 time step run took about 17 minutes.
The results of the runs described above are groups of 64 netCDF files (one for each processor) for each of the calculated quantities:
U, V, W, and T
output.init: initial time step output ??
restart and open boundary condition restart
2D slice timeseries results
tidal harmonics disagnostic results
Google lead to the NOCSCOMBINE tool at ftp://ftp.soc.soton.ac.uk/omfftp/NEMO/NOCSCOMBINE.tar.
Building it on jasper required creation of a new
makefile with NCHOME and LIBS variable set to:
NCHOME = /lustre/jasper/software/netcdf/netcdf-4.1.3 LIBS = -L$(NCHOME)/lib -I$(NCHOME)/include -lnetcdf -lnetcdff -lhdf5_hl -lhdf5 -lz -lsz
cd WCSD_RUN_tide_M2_OW_ON_file_DAMP_ANALY/ ../../NOCSCOMBINE/nocscombine -f WC3_CU60_20020102_20020104_grid_U_0000.nc
result in the 64 pre-processor files of u velocity results being combined into a single
The process takes over 10 minutes per quantity for
U, V, and T
for the 72 hour run,
and nearly 30 minutes for W.