Notes on Downloading/Running NEMO 3.4 on Jasper

Set-up for runs on one processor or multiprocessors included.

Getting your Jasper Shell Ready

  • make sure your shell is bash (echo $SHELL), if its not, write to Westgrid support and get it changed.

  • follow the instructions in Loading Modules on HPC Clusters to manually load the necessary software component modules or edit your jasper $HOME/.bashrc to make them load automatically when you ssh into jasper.

Getting the Code

see Getting the Code in Notes on Downloading/Running NEMO 3.4 on Ocean Cluster including Salish

Making a Project

  • NEMO uses an ARCH (architecture) file to determine compiler, maker, netcdf library location.

    cd dev_v3_4_STABLE_2012/NEMOGCM/ARCH
    
  • NEMO ships with a number of different arch files. After changes to Jasper’s operating system and updated arch file is needed:

    # makefile definitions for mpif90 on jasper.westgrid.ca; based on ifort file from P. Myers group
    #
    # NCDF_INC    netcdf include file
    # NCDF_LIB    netcdf library
    # FC          Fortran compiler command
    # FCFLAGS     Fortran compiler flags
    # LD          linker
    # LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
    # FPPFLAGS    pre-processing flags
    # AR          assembler
    # ARFLAGS     assembler flags
    # MK          make
    # USER_INC    additional include files for the compiler,  e.g. -I<include dir>
    # USER_LIB    additional libraries to pass to the linker, e.g. -l<library>
    
    %NCDF_HOME    /global/software/netcdf/netcdf-4.1.3
    %HDF5_HOME    /global/software/hdf5/hdf5-1.8.9
    %NCDF_INC     -I%NCDF_HOME/include
    %NCDF_LIB     -L%NCDF_HOME/lib -lnetcdff -lnetcdf -L%HDF5_HOME/lib -lhdf5_hl -lhdf5 -lhdf5
    %CPP          cpp
    %FC           mpif90
    %FCFLAGS      -c -fpp -r8 -O3 -assume byterecl -convert big_endian -heap-arrays
    %LD           mpif90
    %FFLAGS       %FCFLAGS
    %LDFLAGS      -lstdc++
    %FPPFLAGS     -P -C -traditional
    %AR           ar
    %ARFLAGS      -r
    %MK           make
    %USER_INC     %NCDF_INC
    %USER_LIB     %NCDF_LIB
    
  • then change directory and make a project, e.g. then for a new GYRE configuration using your new arch file ocean

  • Add the netcdf4 key to use netcdf4 capabilities

  • Exactly the same (except GYRE -> AMM12) for AMM12

    cd ../CONFIG
    ./makenemo -m mpif90_jasper -r GYRE -n MY_GYRE add_key "key_netcdf4"
    

Running the Code: GYRE

  • Go to your version (where you want the results to end up)

    cd MY_GYRE/EXP00
    
  • Created a .pbs run file. For a simple run of GYRE that could be

  • PBS file: GYRE.pbs containing:

    # Script for running simple GYRE configuration
    
    #PBS -l procs=1
    #PBS -l pmem=500mb
    #PBS -l walltime=00:05:00
    
    module load compiler/intel/12.1
    module load library/intelmpi/4.0.3.008
    module load library/netcdf/4.1.3
    module load library/szip/2.1
    
    module list
    echo "Current working directory is `pwd`"
    cd dev_v3_4_STABLE_2012/NEMOGCM/CONFIG/MY_GYRE/EXP00
    echo "Current working directory is `pwd`"
    
    echo "Starting run at: `date`"
    ./opa
    echo "Program opa finished with exit code $? at: `date`"
    
  • and run

    qsub GYRE.pbs
    

Running the CODE: AMM12: 32 Processors

  • Need to get the AMM12 forcing and initialization files, untar and unzip

    curl -LO http://dodsp.idris.fr/reee512/NEMO/amm12_inputs_v3_4.tar
    cd dev_v3_4_STABLE_2012/NEMOGCM/CONFIG/MY_AMM12/EXP00/
    tar xvf ~/amm12_inputs_v3_4.tar
    gunzip *.gz
    rm ~/amm12_input_v3_4.tar
    
  • To make AMM12 run on multiple processors, edit the namelist file, changing the following lines

    jpni = 8
    jpnj = 4
    jnpij = 32
    
  • Need a .pbs file for multiple core run PBS file: AMM_multi.pbs containing:

    # Script for running multiple processor AMM12  configuration
    
    #PBS -l procs=32
    #PBS -l pmem=500mb
    #PBS -l walltime=00:15:00
    
    module load compiler/intel/12.1
    module load library/intelmpi/4.0.3.008
    module load library/netcdf/4.1.3
    module load library/szip/2.1
    
    module list
    echo "Current working directory is `pwd`"
    cd dev_v3_4_STABLE_2012/NEMOGCM/CONFIG/MY_AMM12/EXP00
    echo "Current working directory is `pwd`"
    
    echo "Starting run at: `date`"
    mpiexec ./opa
    
  • and run

    qsub AMM12_multi.pbs