exp2.0_run2-expt-3d030595
Helens regional superoz
✅ Latest submission: Run 18 completed successfully (2026-09-29 10:19 AEST). Model date reached: 2014-08-01 (1.6 model years complete).
Notebooks: ocean_stats, ❌ timeseries, ❌ Timeseries_daily_extreme_from_2D_fields, ❌ temp-salt-vs-depth-time
📋 Summary for the GitHub issue
### OM3 Monitor summary: `exp2.0_run2-expt-3d030595`
*Helens regional superoz*
- **Latest submission:** ✅ Run 18 completed successfully (2026-09-29 10:19 AEST).
- **Model date reached:** 2014-08-01 (1.6 model years complete)
- **Average throughput:** 0.58 SYPD
- **Truncations:** none recorded in `ocean.stats`
**References**
1. Archive folder: `/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595`
2. Status page: https://chrisb13.github.io/access-om3-live/live/exp2.0_run2-expt-3d030595/
*Generated automatically by the [OM3 Monitor](https://chrisb13.github.io/access-om3-live/live/) on 2026-10-19 16:38 AEDT; it may contain errors, so check against the files above.*
Archive folder: completed runs
Read from the payu archive folder /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595.
- Output folders: 19 (latest
output018) - Model date reached: 2014-08-01
- Runs recorded by payu: 22, of which failed: 2
Check this on Gadi: these counts
# WHAT THIS DOES: re-counts the 'Archive folder' list: how many output folders there are, which is
# the latest, how many runs payu recorded, how many of those failed, and how far the model got. The
# four lines it prints should match the page.
A=/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595
# Step 1: count the output folders (output000, output001, ...) and name the last one
ls -d "$A"/output[0-9]* | wc -l
basename "$(ls -d "$A"/output[0-9]* | tail -n 1)"
# Step 2: read payu's run records with a few lines of python, and count them
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
# a run failed if payu's exit status for it is not 0 (a run with no status yet is still going)
failed = [r for r in runs if r.get("payu_run_status") not in (0, None)]
print(len(runs), "runs recorded by payu,", len(failed), "failed")
# the model date reached is the finish date of the last run that has one
finished = [r for r in runs if r.get("model_finish_time")]
print("model date reached:", finished[-1]["model_finish_time"][:10] if finished else "unknown")
EOF
Model throughput
Simulated years per day, made by plot_sypd.py from the outputNNN/log/med.log files of the completed submissions in /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595.
Check this on Gadi: the rate in each output's med.log
# WHAT THIS DOES: for each output folder, finds the model's speed in its med.log (the mediator's
# log) and averages it. The speed, called the 'rate', is in simulated years per day (ypd). The
# figure plots these rates. You get one line per output folder: how many rate lines it had, and
# their average.
A=/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595
for o in "$A"/output[0-9]*; do
[ -f "$o/log/med.log" ] || continue # skip folders that have no med.log
# A line looks like: Model Date: 1958-09-04T00:00:00 ... rate = 7.09 ypd. awk reads the number just before
# 'ypd' (the second-last word), adds them up and counts them, then prints the average at the end.
awk -v name="$(basename "$o")" '/Model Date:/ && /ypd/ {count++; total += $(NF-1)} END {if (count) printf "%s: %d rate lines, mean %.2f ypd\n", name, count, total/count}' "$o/log/med.log"
done
Recent runs
From payu's job records in /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/payu_jobs; times are Canberra time.
| Run | PBS job | Result | Model period | Wall time (h) | SYPD | Record last modified |
|---|---|---|---|---|---|---|
| 18 | 180073572.gadi-pbs | ✅ | 2014-07-01 → 2014-08-01 | 2.5 | 0.80 | 2026-09-29 10:19 AEST |
| 17 | 180048183.gadi-pbs | ✅ | 2014-06-01 → 2014-07-01 | 2.5 | 0.80 | 2026-09-28 23:01 AEST |
| 16 | 180039706.gadi-pbs | ✅ | 2014-05-01 → 2014-06-01 | 2.6 | 0.80 | 2026-09-28 20:31 AEST |
| 15 | 180026187.gadi-pbs | ✅ | 2014-04-01 → 2014-05-01 | 2.5 | 0.79 | 2026-09-28 17:56 AEST |
| 14 | 179982470.gadi-pbs | ✅ | 2014-03-01 → 2014-04-01 | 2.6 | 0.78 | 2026-09-28 15:24 AEST |
| 13 | 179828831.gadi-pbs | ✅ | 2014-02-01 → 2014-03-01 | 2.4 | 0.77 | 2026-09-25 13:44 AEST |
| 12 | 179819805.gadi-pbs | ✅ | 2014-01-01 → 2014-02-01 | 2.6 | 0.79 | 2026-09-25 09:21 AEST |
| 11 | 179812149.gadi-pbs | ✅ | 2013-12-01 → 2014-01-01 | 2.6 | 0.79 | 2026-09-25 06:46 AEST |
| 10 | 179803805.gadi-pbs | ✅ | 2013-11-01 → 2013-12-01 | 2.5 | 0.79 | 2026-09-25 04:09 AEST |
| 9 | 179791316.gadi-pbs | ✅ | 2013-10-01 → 2013-11-01 | 2.6 | 0.78 | 2026-09-25 01:38 AEST |
Check this on Gadi: this table
# WHAT THIS DOES: prints the 'Recent runs' table: the 10 newest runs, newest first. 'ok' means
# payu's exit status was 0. SYPD is simulated years per day: the model time the run covered, divided
# by the wall-clock time it took. (If payu did not record a run's start date this shows ?; the page
# may fill it in from the previous run.)
A=/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
import time
from datetime import datetime
print("Run PBS job Result Model period Hours SYPD Record last modified")
for r in reversed(runs[-10:]): # the last 10 runs, newest first
status = r.get("payu_run_status")
result = "ok" if status == 0 else ("failed" if status is not None else str(r.get("stage")))
start, finish = r.get("model_start_time"), r.get("model_finish_time")
seconds = r.get("timings", {}).get("payu_total_duration_seconds") # wall-clock seconds the run took
hours = "%.1f" % (seconds / 3600) if seconds else ""
sypd = ""
if start and finish and seconds:
# the model time the run covered, in whole days (dates look like 1958-01-01T00:00:00)
days = (datetime.strptime(finish[:19], "%Y-%m-%dT%H:%M:%S") - datetime.strptime(start[:19], "%Y-%m-%dT%H:%M:%S")).days
sypd = "%.2f" % (days / 365.25 / (seconds / 86400)) # model years divided by wall-clock days
changed = time.strftime("%Y-%m-%d %H:%M %Z", time.localtime(os.path.getmtime(r["file"])))
print("%-4s %-18s %-10s %-10s -> %-10s %6s %6s %s" % (r.get("payu_current_run"), r.get("scheduler_job_id"), result,
(start or "?")[:10], (finish or "?")[:10], hours, sypd, changed))
EOF
Latest logs in the archive folder
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/output018/access-om3.err (last lines; modified 2026-09-29 10:19 AEST)
WARNING from PE 0: MOM_file_parser : HBBL occurs more times than is permitted. Line: 'HBBL = 10.0' in file MOM_input is being ignored.
WARNING from PE 0: The runtime parameter TIDE_REF_DATE is being set via its soon to be obsolete name of OBC_TIDE_REF_DATE. Please migrate to using TIDE_REF_DATE.
WARNING from PE 0: The runtime parameter TIDE_USE_EQ_PHASE is being set via its soon to be obsolete name of OBC_TIDE_ADD_EQ_PHASE. Please migrate to using TIDE_USE_EQ_PHASE.
WARNING from PE 0: The runtime parameter TIDE_NODAL_REF_DATE is being set via its soon to be obsolete name of OBC_TIDE_REF_DATE. Please migrate to using TIDE_NODAL_REF_DATE.
WARNING from PE 0: MOM_file_parser : HBBL occurs more times than is permitted. Line: 'HBBL = 10.0' in file MOM_input is being ignored.
WARNING from PE 0: Unused line in MOM_input : SURFACE_HEIGHT_IC_FILE = "init_eta.nc"
WARNING from PE 0: Unused line in MOM_input : SURFACE_HEIGHT_IC_VAR = "eta_t"
WARNING from PE 0: Unused line in MOM_input : FLUXCONST = 0.11
WARNING from PE 0: Unused line in MOM_input : SALT_RESTORE_FILE = "salt_sfc_restore.nc"
WARNING from PE 0: Unused line in MOM_input : SRESTORE_AS_SFLUX = True
Memuse(MB) at Memory HiWaterMark= 7.146E+02 1.091E+04 1.786E+02 7.567E+02
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/output018/access-om3.out (last lines; modified 2026-09-29 10:19 AEST)
(Ocean BT global synch) 8928 23.958847 539.299678 241.841006 80.348927 0.027 41 0 4079
(Ocean Coriolis & mom advection) 17856 46.182924 73.306591 62.308458 3.420283 0.007 31 0 4079
(Ocean continuity equation) 26784 300.498787 776.159815 646.268973 71.562083 0.071 31 0 4079
(Ocean pressure force) 8928 291.307738 447.245254 399.362655 27.118379 0.044 31 0 4079
(Ocean vertical viscosity) 26784 44.130651 735.441614 582.228629 119.180074 0.064 31 0 4079
(Ocean horizontal viscosity) 8928 31.928169 46.300480 37.532397 1.704321 0.004 31 0 4079
(Ocean momentum increments) 17856 13.301414 47.289356 36.626979 5.653504 0.004 31 0 4079
(Ocean message passing) 89280 153.781722 1184.135277 380.094686 143.682371 0.042 31 0 4079
(Ocean barotropic mode calc) 17856 3.131464 7.197287 5.295507 0.678593 0.001 31 0 4079
(Ocean barotropic mode stepping) 17856 503.903185 1156.642264 734.913507 100.692054 0.081 31 0 4079
(Ocean barotropic forcing calc) 17856 18.923811 56.078847 36.899287 6.754256 0.004 31 0 4079
(Ocean kappa_shear) 2976 11.406196 610.026406 149.031038 71.321124 0.016 31 0 4079
(Ocean diabatic entrain) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 4079
(Ocean vert remap) 2976 21.108435 434.729606 158.885099 58.576149 0.018 31 0 4079
(Ocean set_diffusivity) 5952 29.643350 639.015161 178.322124 69.676653 0.020 31 0 4079
(Ocean KPP) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 4079
(Ocean tracer_columns) 2976 0.001530 0.003599 0.002107 0.000203 0.000 26 0 4079
(Ocean diabatic tridiag) 2976 4.713292 11.227882 8.577611 0.718545 0.001 41 0 4079
(Ocean diabatic message passing) 5952 6.340487 659.704251 139.773832 76.408749 0.015 41 0 4079
(Ocean find_uv_at_h) 5952 12.518657 32.450787 23.091188 3.459001 0.003 41 0 4079
(Ocean frazil) 0 0.000000 0.000000 0.000000 0.000000 0.000 41 0 4079
(Ocean advect tracer) 2976 203.342150 316.913464 244.914572 17.649457 0.027 31 0 4079
(Ocean tracer halo updates) 17729 57.701173 215.958780 149.525213 32.374907 0.017 41 0 4079
(Ocean tracer global synch) 11777 5.674686 241.548166 78.747854 42.710364 0.009 41 0 4079
(Ocean diffuse tracer) 2976 9.630287 30.893950 23.868101 5.060330 0.003 31 0 4079
(Ocean epipycnal diffuse tracer) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 4079
(Ocean OBC halo updates) 17856 36.984058 171.320726 74.194644 10.966738 0.008 41 0 4079
Ocean surface forcing 5953 21.440385 264.825800 73.992390 31.397561 0.008 11 0 4079
(Ocean forcing diagnostics) 5952 62.370984 64.170713 64.078290 0.034047 0.007 41 0 4079
MPP_STACK high water mark= 0
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/error_logs/access-om3.179722810.gadi-pbs.out (last lines; modified 2026-09-24 13:55 AEST)
X-AXIS = 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15
Y-AXIS = 13 13 13 13 13 13 13 13 13 13 13 13 13 13 13 12 12 12 12 12 12 12 12 12
12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12 12
parse_mask_table: Number of domain regions masked in MOM_in = 780
MOM_in domain decomposition
whalo = 4, ehalo = 4, shalo = 4, nhalo = 4
X-AXIS = 31 31 31 31 31 31 31 31 31 31 31 31 31 31 31 31 31 30 30 30 30 30 30 30
30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 30 31 31 31 31 31
31 31 31 31 31 31 31 31 31 31 31 31
Y-AXIS = 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 21 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20 20
20 20 20 20 20 20 20 20 20 20 20 20 21 21 21 21 21 21 21 21 21 21 21 21
21 21 21 21 21 21 21 21 21
MOM_inc domain decomposition
whalo = 2, ehalo = 2, shalo = 2, nhalo = 2
X-AXIS = 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 16 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15 15
15 15 15 15 15 15 15 15 15 15 15 15
Y-AXIS = 11 11 11 11 11 11 11 11 11 11 11 10 10 10 10 10 10 10 10 10 10 10 10 10
10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 9 10 10 10 10 10 10 10
10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 11 11
11 11 11 11 11 11 11 11 11
NOTE from PE 0: diag_manager_mod::diag_manager_init: prepend_date only supported when diag_manager_init is called with time_init present.
NOTE from PE 0: diag_manager_mod::diag_manager_init: You are using the legacy version of the diag table
FATAL from PE 0: diag_util_mod::init_output_field: file access-om3.mom6.3d.h.1day.mean%4yr.%2mo is NOT found in the diag_table
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/output018/log/med.log (last lines; modified 2026-09-29 10:15 AEST)
Model Date: 2014-07-20T00:00:00 wall clock = 2026-09-29T09:17:11 avg dt = 254.39 s/day, dt = 197.93 s/day, rate = 1.20 ypd
memory_write: model date = 2014-07-20T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-21T00:00:00 wall clock = 2026-09-29T09:20:34 avg dt = 251.86 s/day, dt = 203.72 s/day, rate = 1.16 ypd
memory_write: model date = 2014-07-21T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-22T00:00:00 wall clock = 2026-09-29T09:23:54 avg dt = 249.37 s/day, dt = 199.70 s/day, rate = 1.19 ypd
memory_write: model date = 2014-07-22T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-23T00:00:00 wall clock = 2026-09-29T09:27:10 avg dt = 246.96 s/day, dt = 196.18 s/day, rate = 1.21 ypd
memory_write: model date = 2014-07-23T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-24T00:00:00 wall clock = 2026-09-29T09:30:28 avg dt = 244.81 s/day, dt = 197.58 s/day, rate = 1.20 ypd
memory_write: model date = 2014-07-24T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-25T00:00:00 wall clock = 2026-09-29T09:33:49 avg dt = 242.99 s/day, dt = 201.08 s/day, rate = 1.18 ypd
memory_write: model date = 2014-07-25T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-26T00:00:00 wall clock = 2026-09-29T09:37:13 avg dt = 241.43 s/day, dt = 204.10 s/day, rate = 1.16 ypd
memory_write: model date = 2014-07-26T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-27T00:00:00 wall clock = 2026-09-29T09:40:33 avg dt = 239.84 s/day, dt = 200.11 s/day, rate = 1.18 ypd
memory_write: model date = 2014-07-27T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-28T00:00:00 wall clock = 2026-09-29T09:43:55 avg dt = 238.42 s/day, dt = 201.33 s/day, rate = 1.18 ypd
memory_write: model date = 2014-07-28T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-29T00:00:00 wall clock = 2026-09-29T09:47:17 avg dt = 237.13 s/day, dt = 202.41 s/day, rate = 1.17 ypd
memory_write: model date = 2014-07-29T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-30T00:00:00 wall clock = 2026-09-29T09:50:40 avg dt = 235.96 s/day, dt = 203.24 s/day, rate = 1.16 ypd
memory_write: model date = 2014-07-30T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
Model Date: 2014-07-31T00:00:00 wall clock = 2026-09-29T09:54:02 avg dt = 234.83 s/day, dt = 202.10 s/day, rate = 1.17 ypd
memory_write: model date = 2014-07-31T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
-------->(med_phases_restart_write) mediating for: 2014 7 31 23 45 0 0
(med_io_wopen) creating file access-om3.cpl.r.2014-08-01-00000.nc
Model Date: 2014-08-01T00:00:00 wall clock = 2026-09-29T10:15:15 avg dt = 268.32 s/day, dt = 1273.03 s/day, rate = 0.19 ypd
memory_write: model date = 2014-08-01T00:00:00 memory = -0.00 MB (highwater) 1395.48 MB (usage)
SUCCESSFUL TERMINATION OF CMEPS
: =============== # simulated years / cmp-day = 0.882 ===============
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/pbs_logs/superaus_c.o179848120 (last lines; modified 2026-09-25 13:48 AEST)
laboratory path: /scratch/tm70/hm6113/access-om3
binary path: /scratch/tm70/hm6113/access-om3/bin
input path: /scratch/tm70/hm6113/access-om3/input
work path: /scratch/tm70/hm6113/access-om3/work
archive path: /scratch/tm70/hm6113/access-om3/archive
Found experiment archive: /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595
payu: Found modules in /opt/Modules/v4.3.0
mod access3-share/2026.03.002-m65pkhf
mod access-cice/CICE6.6.3-1-uqsyhj7
mod access-mocsy/2025.07.002-5awgzcl
mod access-generic-tracers/2026.05.000-7jsawsl
mod access-mom6/2026.05.001-impfcxi
mod access-ww3/2026.03.000-kjy7rmt
mod access3/2026.03.002-7t6n2ft
mod access-om3/2026.05.002
mod model-tools/mppnccombine-fast/2025.07.000
{'/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/output013': []}
payu: Found modules in /opt/Modules/v4.3.0
======================================================================================
Resource Usage on 2026-09-25 13:48:22:
Job Id: 179848120.gadi-pbs
Project: tm70
Exit Status: 0
Service Units: 0.07
NCPUs Requested: 4 CPU Time Used: 00:00:06
Memory Requested: 30.0GB Memory Used: 618.62MB
Walltime Requested: 04:00:00 Walltime Used: 00:00:20
JobFS Requested: 20.0GB JobFS Used: 0B
======================================================================================
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/error_logs/env.179722810.gadi-pbs.yaml (last lines; modified 2026-09-24 13:55 AEST)
OMPI_ROOT: /apps/openmpi/4.1.7
VT_MAX_FLUSHES: '0'
LIBRARY_PATH: /apps/openmpi/4.1.7/lib:/apps/openmpi/4.1.7/lib/profilers
OMPI_VERSION: 4.1.7
MODULES_LMCONFLICT:
openmpi/4.1.7&mpi&lam&mpich&openmpi&intel-mpi&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars
OPENMPI_ROOT: /apps/openmpi/4.1.7
UCC_LOG_LEVEL: ERROR
LIBRARY_PATH_modshare:
/apps/openmpi/4.1.7/lib/profilers:1:/apps/openmpi/4.1.7/lib:1
FPATH: /apps/openmpi/4.1.7/include
C_INCLUDE_PATH_modshare: /apps/openmpi/4.1.7/include:1
MODULES_LMCONFLICT_modshare:
openmpi/4.1.7&mpi&lam&mpich&openmpi&intel-mpi&o/wrappers&o/yes-wrappers&o/use-wrappers&o/enable-wrappers&o/with-wrappers&o/no-wrappers&o/not-wrappers&o/disable-wrappers&o/without-wrappers&o/ld_library_path&o/yes-ld_library_path&o/use-ld_library_path&o/enable-ld_library_path&o/with-ld_library_path&o/no-ld_library_path&o/not-ld_library_path&o/disable-ld_library_path&o/without-ld_library_path&o/ld_run_path&o/yes-ld_run_path&o/use-ld_run_path&o/enable-ld_run_path&o/with-ld_run_path&o/no-ld_run_path&o/not-ld_run_path&o/disable-ld_run_path&o/without-ld_run_path&o/show-debug&o/yes-show-debug&o/use-show-debug&o/enable-show-debug&o/with-show-debug&o/no-show-debug&o/not-show-debug&o/disable-show-debug&o/without-show-debug&o/append-paths&o/yes-append-paths&o/use-append-paths&o/enable-append-paths&o/with-append-paths&o/no-append-paths&o/not-append-paths&o/disable-append-paths&o/without-append-paths&o/library_path&o/yes-library_path&o/use-library_path&o/enable-library_path&o/with-library_path&o/no-library_path&o/not-library_path&o/disable-library_path&o/without-library_path&o/packaged-envvars&o/yes-packaged-envvars&o/use-packaged-envvars&o/enable-packaged-envvars&o/with-packaged-envvars&o/no-packaged-envvars&o/not-packaged-envvars&o/disable-packaged-envvars&o/without-packaged-envvars:1
FPATH_modshare: /apps/openmpi/4.1.7/include:1
CPLUS_INCLUDE_PATH: /apps/openmpi/4.1.7/include
OMPI_BASE: /apps/openmpi/4.1.7
OMPI_MCA_orte_tmpdir_base: /jobfs/179722810.gadi-pbs
OPENMPI_BASE: /apps/openmpi/4.1.7
LD_RUN_PATH_modshare:
/apps/openmpi/4.1.7/lib/profilers:1:/apps/openmpi/4.1.7/lib:1
LD_LIBRARY_PATH_modshare:
/apps/openmpi/4.1.7/lib/profilers:1:/apps/openmpi/4.1.7/lib:1
CPATH_modshare: /apps/openmpi/4.1.7/include:1
C_INCLUDE_PATH: /apps/openmpi/4.1.7/include
OPENMPI_VERSION: 4.1.7
UCC_TLS: ^sharp
HCOLL_ENABLE_MCAST: '0'
VT_PFORM_LDIR: /jobfs/179722810.gadi-pbs
PAYU_RUN_ID: 70268f6a4a322a182396192ec5830ded1d519bc5
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/pbs_logs/payu_postscript.o179848123 (last lines; modified 2026-09-25 13:52 AEST)
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.05/lib/python3.11/site-packages/access_nri_intake/source/utils.py:142: UserWarning: Time coordinate does not include bounds information. Guessing start and end times.
warnings.warn(
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.05/lib/python3.11/site-packages/access_nri_intake/source/builders.py:202: UserWarning: Unable to parse 308 assets. A list of these assets can be found in `.invalid_assets` attribute.
self.get_assets().validate_parser().parse().clean_dataframe()
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.05/lib/python3.11/site-packages/intake_esm/cat.py:187: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.11/migration/
data = self.dict().copy()
/g/data/xp65/public/apps/med_conda/envs/analysis3-25.05/lib/python3.11/site-packages/pydantic/deprecated/decorator.py:227: UserWarning: Unable to parse 308 assets/files. A list of these assets can be found in /jobfs/179848123.gadi-pbs/experiment_datastore_invalid_assets.csv.
return self.raw_function(**d, **var_kwargs)
[32m[22mSucessfully built esm-datastore![0m
[34m[22mSaving esm-datastore to [36m[1m/home/189/hm6113/access-om3/superaus/exp2.0_run2/archive[0m
Successfully wrote ESM catalog json file to: file:///home/189/hm6113/access-om3/superaus/exp2.0_run2/archive/experiment_datastore.json
[34m[22mHashing catalog to prevent unnecessary rebuilds.
This may take some time...[0m
[32m[22mCatalog sucessfully hashed![0m
[32m[22mDatastore sucessfully written to [36m[1m/home/189/hm6113/access-om3/superaus/exp2.0_run2/archive/experiment_datastore.json[32m[22m!
[34m[22mPlease note that this has not added the datastore to the access-nri-intake catalog.
To add to catalog, please run '[37m[1mscaffold_catalog_entry[34m[22m' for help on how to do so.
sys:1: DeprecationWarning: Call to deprecated function (or staticmethod) _destroy.
======================================================================================
Resource Usage on 2026-09-25 13:52:31:
Job Id: 179848123.gadi-pbs
Project: tm70
Exit Status: 0
Service Units: 8.32
NCPUs Requested: 104 CPU Time Used: 02:05:07
Memory Requested: 500.0GB Memory Used: 98.62GB
Walltime Requested: 00:30:00 Walltime Used: 00:02:24
JobFS Requested: 100.0GB JobFS Used: 149.57KB
======================================================================================
Check this on Gadi: when these logs were written, and their last lines
# WHAT THIS DOES: for each log file listed above, prints when it was last changed and its last 30
# lines, which are the lines the page shows.
# 'for' repeats the three lines below once for each file; $f is the file it is on.
# 'ls -l' lists a file with details; --time-style=long-iso writes its time as 2026-07-01 00:02.
for f in /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/output018/access-om3.err /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/output018/access-om3.out /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/error_logs/access-om3.179722810.gadi-pbs.out /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/output018/log/med.log /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/pbs_logs/superaus_c.o179848120 /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/error_logs/env.179722810.gadi-pbs.yaml /scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/pbs_logs/payu_postscript.o179848123; do
echo "== $f" # which file this is
ls -l --time-style=long-iso "$f" # when it was last changed (the date and time are in the middle)
tail -n 30 "$f" # its last 30 lines
done
/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595/metadata.yaml
# ---- Automatically generated by Payu. Do not edit. ----
experiment_uuid: 3d030595-452a-4fe0-9a55-92aa5b16e7eb
# ---- Automatically generated by Payu. User may edit if necessary. ----
name: exp2.0_run2-expt-3d030595
contact: Helen Macdonald
email: 179985228+helenmacdonald@users.noreply.github.com
created: '2026-09-22'
url: https://github.com/ACCESS-NRI/access-om3-configs
model: ACCESS-OM3
schema_version: 1-0-4
# ---- User feel free to update. ----
description: |-
25km ACCESS-OM3 global configuration under interannual forcing.
Initial conditions of temperature and salinity are from WOA23.
Run with JRA55-do v1.6.0 interannually-varying forcing, 1 Jan 1958 - 31 Dec 2023.
notes: |-
ACCESS-NRI and COSIMA request that users of this model acknowledge those organisations
following the examples at https://www.access-nri.org.au/resources/acknowledging-us/
and https://cosima.org.au/index.php/get-involved/
keywords:
- JRA55
- access-om3
- interannual
- global
realm:
- ocean
- seaIce
nominal_resolution: 25 km
license: CC-BY-4.0
version: 2.0-beta
long_description: __REPLACE_ME__ # Long description of the experiment (string)
# frequency: The frequency(/ies) included in the experiment (array of strings)
# variable: The variable(s) included in the experiment (array of strings)
# reference: Citation or reference information (string)
# parent_experiment: experiment_uuid for parent experiment if appropriate (string)
# related_experiments: experiment_uuids for any related experiment(s) (array of strings)
# parent_experiment_branch_time: The model time corresponding to the restart path used to create this branch, e.g., 1951-02-01T00:00:00 (string)
# parent_experiment_branch_commit: Git commit hash of the source branch when this new branch was created (string)
Truncation locations for all completed run segments
✅ No velocity truncations: the Truncs column of ocean.stats is zero in all 19 output folders.
Check this on Gadi: the truncation counts, locations and settings
# WHAT THIS DOES: re-checks the 'Truncation locations' section from the files MOM6 wrote. A
# truncation is when MOM6 had to cut a velocity that was too fast (its CFL number was above 0.5): a
# sign of numerical trouble. The steps below are separate, so you can paste just the one you need.
# Each prints something you can compare with the page.
A=/scratch/tm70/hm6113/access-om3/archive/exp2.0_run2-expt-3d030595
# Step 1: for each output folder: how many truncations MOM6 reported, the largest CFL number, and how many location records were written
for o in "$A"/output[0-9]*; do
# ocean.stats is a table with one line per time step. Its 3rd column ('Truncs') is the truncations at that
# step. awk adds that column up. (-F, means the columns are separated by commas. The header line is skipped
# because its first column is not a number.)
total=$(awk -F, '$1+0 == $1 {sum += $3} END {print sum+0}' "$o/ocean.stats" 2>/dev/null)
# The 5th column holds two numbers; the second one is the CFL number. awk keeps the largest.
cfl=$(awk -F, '$1+0 == $1 {split($5, a, " "); if (a[2] > max) max = a[2]} END {if (max != "") printf "%.2f", max}' "$o/ocean.stats" 2>/dev/null)
# Every truncation also gets a 'location record' in the U and V files. grep -c counts the matching lines.
u=$(grep -c 'velocity violation' "$o/U_velocity_truncations" 2>/dev/null)
v=$(grep -c 'velocity violation' "$o/V_velocity_truncations" 2>/dev/null)
# ${x:--} means: use x, or a dash if x is empty (no such file)
echo "$(basename "$o"): truncations ${total:--}, max CFL ${cfl:--}, location records U/V ${u:--}/${v:--}"
done
# Step 2: where it happens. Each record contains the place, like '( 73.27 E 67.89 N)'. This counts the records in
# boxes one degree wide (negative longitude means west), busiest first. The pipeline does this:
# grep: keep the lines that start a record sed: keep only the longitude and latitude
# awk: round them to whole degrees sort | uniq -c: count the records in each box
# sort -rn | head: busiest boxes first
grep -h 'velocity violation' "$A"/output[0-9]*/[UV]_velocity_truncations 2>/dev/null \
| sed -E 's/.*\( *(-?[0-9.]+) E +(-?[0-9.]+) N\).*/\1 \2/' \
| awk '{printf "%4.0fE %4.0fN\n", $1, $2}' \
| sort | uniq -c | sort -rn | head -n 15
# Step 3: MOM6 writes at most 50 records per PE (one of the parallel processes) in one run, then stops. A PE with
# 50 records may have truncated more often than the records show. This lists the PEs that reached 50.
for f in "$A"/output[0-9]*/[UV]_velocity_truncations; do
[ -f "$f" ] || continue # skip if there is no such file
# each record says which PE it came from ('violation at 6073:'); count the records per PE, keep PEs with 50
busy=$(grep -h 'velocity violation' "$f" | sed -E 's/.*violation at ([0-9]+):.*/PE\1/' | sort | uniq -c | awk '$1 >= 50 {printf "%s (%d records) ", $2, $1}')
echo "$(basename "$(dirname "$f")")/$(basename "$f"): $busy"
done
# Step 4: the model period of each output folder, from payu's run records (output NNN is run NNN)
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
for r in runs:
run, finish = r.get("payu_current_run"), r.get("model_finish_time")
if str(run).isdigit() and finish: # only runs that finished have a model period
start = r.get("model_start_time") or "?" # payu did not record the start of the first run
print("output%03d %s -> %s" % (int(run), start[:7], finish[:7]))
EOF
Settings changed during this run
This comes from the git history of the payu control directory, and this experiment has no run_dir in experiments.yml. Add one to see here every edit made to the model's settings and build during the run, with the old and new values.