Christopher Bull bio photo

Christopher Bull

Aspiring Oceanographer at Northumbria University. Big data python enthusiast. Outdoor adventurer.

Email Twitter Github

Building and Running NEMO4 on ARCHER2 full system.

We take the first step of downloading, compiling and testing NEMO4 code on ARCHER2 on the full system (December 2021).

Pre-requisites:

Suggested reading:

Installing NEMO

Get NEMO4, in my case I’m using a 4.0.4 branch..

svn -r 15194 co https://forge.ipsl.jussieu.fr/nemo/svn/NEMO/branches/UKMO/NEMO_4.0.4_momentum_trends NEMO404_MO

Build NEMO

The following is based on NEMO4 ARCHER2 help page. Log-into ARCHER2 (full system). Set-up your environment with:

module restore
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load xpmem
module load perftools-base

Originally from ARCHER2 4c: /work/n01/shared/acc/arch-X86_ARCHER2-Cray.fcm put into your ARCH folder, i.e., /work/n02/n02/chbull/temp/NEMO404_MO/arch/arch-X86_ARCHER2-Cray.fcm

%NCDF_HOME           $NETCDF_DIR
%HDF5_HOME           $HDF5_DIR
%XIOS_HOME           /work/n02/shared/chbull/xios-2.5

%NCDF_INC            -I%NCDF_HOME/include -I%HDF5_HOME/include
%NCDF_LIB            -L%HDF5_HOME/lib -L%NCDF_HOME/lib -lnetcdff -lnetcdf -lhdf5_hl -lhdf5 -lz
%XIOS_INC            -I%XIOS_HOME/inc
%XIOS_LIB            -L%XIOS_HOME/lib -lxios

%CPP	             cpp
%FC                  ftn
%FCFLAGS             -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%FFLAGS              -em -s integer32 -s real64 -O1 -hflex_mp=intolerant
%LD                  CC -Wl,"--allow-multiple-definition"
%FPPFLAGS            -P -traditional
%LDFLAGS             -lmpifort_cray
%AR                  ar
%ARFLAGS             -r
%MK                  gmake
%USER_INC            %XIOS_INC %NCDF_INC
%USER_LIB            %XIOS_LIB %NCDF_LIB

%CC                  cc
%CFLAGS              -O0

Add -J flag to FC_MODSEARCH in /work/n02/n02/chbull/temp/NEMO404_MO/ext/FCM/lib/Fcm/Config.pm, should look like

FC_MODSEARCH => '-J',                # FC flag, specify "module" path

Add key_nosignedzero to /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/cpp_ISOMIPY.fcm, should have:

chbull@uan01:/work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY> cat cpp_ISOMIPY.fcm 
 bld::tool::fppkeys   key_iomput key_mpp_mpi key_nosignedzero

Test compile works, i.e.,

cd /work/n02/n02/chbull/temp/NEMO404_MO/
./makenemo -n 'ISOMIPY' -a 'ISOMIP' -m X86_ARCHER2-Cray -j 16
ls -lah tests/ISOMIPY/BLD/bin/nemo.exe
# beer time ?

Run NEMO

We use Andrew’s script to create a template for a run file (saving to something like /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/EXP00/runnemo.sh; which will make sense in a moment)..

#!/bin/bash
#SBATCH --job-name=nemo_test
#SBATCH --time=00:10:00
#SBATCH --nodes=1
#SBATCH --ntasks=24
#SBATCH --account=n02-PROPHET
#SBATCH --partition=standard
#SBATCH --qos=short
# Created by: mkslurm -S 0 -s 0 -m 2 -C  24 -c  2 -t 00:10:00 -a n01 -j nemo_test

module restore
module load cray-hdf5-parallel
module load cray-netcdf-hdf5parallel
module load xpmem
module load perftools-base

export OMP_NUM_THREADS=1
#
cat > myscript_wrapper2.sh << EOFB
#!/bin/ksh
#
set -A map ./xios_server.exe ./nemo
exec_map=( 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 )
#
exec \${map[\${exec_map[\$SLURM_PROCID]}]} 
##
EOFB
chmod u+x ./myscript_wrapper2.sh
#
srun --mem-bind=local --cpu-bind=v,map_cpu:00,0x2,0x4,0x6,0x8,0xa,0xc,0xe,0x10,0x12,0x14,0x16,0x18,0x1a,0x1c,0x1e,0x20,0x22,0x24,0x26,0x28,0x2a,0x2c,0x2e, ./myscript_wrapper2.sh

Submit with

cd /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/EXP00
# nb: https://docs.archer2.ac.uk/archer2-migration/archer2-differences/
# no short flag required anymore..
sbatch /work/n02/n02/chbull/temp/NEMO404_MO/tests/ISOMIPY/EXP00/runnemo.sh
# keep an eye on it with
squeue -u USERNAME
tail -f ocean.output
# second beer ?
In category: nemo4