MC_25km_jra_iaf+wombatlite-test4-d28e0359
25km ACCESS-OM3 global configuration under interannual forcing. (from the run's metadata.yaml)
✅ Latest submission: Run 65 completed successfully (2026-07-01 00:02 AEST). Model date reached: 2024-01-01 (66.0 model years complete).
Notebooks: ocean_stats, timeseries, Timeseries_daily_extreme_from_2D_fields, temp-salt-vs-depth-time
📋 Summary for the GitHub issue
### OM3 Monitor summary: `MC_25km_jra_iaf+wombatlite-test4-d28e0359`
*25km ACCESS-OM3 global configuration under interannual forcing. *(from the run's metadata.yaml)**
- **Latest submission:** ✅ Run 65 completed successfully (2026-07-01 00:02 AEST).
- **Model date reached:** 2024-01-01 (66.0 model years complete)
- **Average throughput:** 6.19 SYPD
- **Truncations:** 10 in 1 of 66 output folders
| Where | Records | Output folders | Max speed (m/s) | Max CFL |
|---|---:|---|---:|---:|
| [65.2°W, 61.3°N](https://www.google.com/maps/place/61.34,-65.25/@61.34,-65.25,5z) | 9 | 000 | 32.8 | 2.23 |
| [65.6°W, 61.4°N](https://www.google.com/maps/place/61.43,-65.62/@61.43,-65.62,5z) | 1 | 000 | 8.2 | 0.55 |
**References**
1. Archive folder: `/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359`
2. Status page: https://chrisb13.github.io/access-om3-live/live/MC_25km_jra_iaf%2Bwombatlite-test4-d28e0359/
*Generated automatically by the [OM3 Monitor](https://chrisb13.github.io/access-om3-live/live/) on 2026-10-19 16:38 AEDT; it may contain errors, so check against the files above.*
Archive folder: completed runs
Read from the payu archive folder /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359.
- Output folders: 66 (latest
output065) - Model date reached: 2024-01-01
- Runs recorded by payu: 73, of which failed: 5
Check this on Gadi: these counts
# WHAT THIS DOES: re-counts the 'Archive folder' list: how many output folders there are, which is
# the latest, how many runs payu recorded, how many of those failed, and how far the model got. The
# four lines it prints should match the page.
A=/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359
# Step 1: count the output folders (output000, output001, ...) and name the last one
ls -d "$A"/output[0-9]* | wc -l
basename "$(ls -d "$A"/output[0-9]* | tail -n 1)"
# Step 2: read payu's run records with a few lines of python, and count them
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
# a run failed if payu's exit status for it is not 0 (a run with no status yet is still going)
failed = [r for r in runs if r.get("payu_run_status") not in (0, None)]
print(len(runs), "runs recorded by payu,", len(failed), "failed")
# the model date reached is the finish date of the last run that has one
finished = [r for r in runs if r.get("model_finish_time")]
print("model date reached:", finished[-1]["model_finish_time"][:10] if finished else "unknown")
EOF
Model throughput
Simulated years per day, made by plot_sypd.py from the outputNNN/log/med.log files of the completed submissions in /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359.
Check this on Gadi: the rate in each output's med.log
# WHAT THIS DOES: for each output folder, finds the model's speed in its med.log (the mediator's
# log) and averages it. The speed, called the 'rate', is in simulated years per day (ypd). The
# figure plots these rates. You get one line per output folder: how many rate lines it had, and
# their average.
A=/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359
for o in "$A"/output[0-9]*; do
[ -f "$o/log/med.log" ] || continue # skip folders that have no med.log
# A line looks like: Model Date: 1958-09-04T00:00:00 ... rate = 7.09 ypd. awk reads the number just before
# 'ypd' (the second-last word), adds them up and counts them, then prints the average at the end.
awk -v name="$(basename "$o")" '/Model Date:/ && /ypd/ {count++; total += $(NF-1)} END {if (count) printf "%s: %d rate lines, mean %.2f ypd\n", name, count, total/count}' "$o/log/med.log"
done
Recent runs
From payu's job records in /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/payu_jobs; times are Canberra time.
| Run | PBS job | Result | Model period | Wall time (h) | SYPD | Record last modified |
|---|---|---|---|---|---|---|
| 65 | 172719523.gadi-pbs | ✅ | 2023-01-01 → 2024-01-01 | 3.8 | 6.23 | 2026-07-01 00:02 AEST |
| 64 | 172690409.gadi-pbs | ✅ | 2022-01-01 → 2023-01-01 | 3.9 | 6.15 | 2026-06-30 19:50 AEST |
| 63 | 172664064.gadi-pbs | ✅ | 2021-01-01 → 2022-01-01 | 7.9 | 3.03 | 2026-06-30 12:28 AEST |
| 62 | 172653972.gadi-pbs | ✅ | 2020-01-01 → 2021-01-01 | 3.8 | 6.32 | 2026-06-30 04:31 AEST |
| 61 | 172641065.gadi-pbs | ✅ | 2019-01-01 → 2020-01-01 | 3.8 | 6.38 | 2026-06-30 00:42 AEST |
| 60 | 172614421.gadi-pbs | ✅ | 2018-01-01 → 2019-01-01 | 3.8 | 6.31 | 2026-06-29 20:56 AEST |
| 59 | 172591679.gadi-pbs | ✅ | 2017-01-01 → 2018-01-01 | 3.8 | 6.30 | 2026-06-29 15:09 AEST |
| 58 | 172572698.gadi-pbs | ✅ | 2016-01-01 → 2017-01-01 | 3.8 | 6.37 | 2026-06-29 11:19 AEST |
| 57 | 172562659.gadi-pbs | ✅ | 2015-01-01 → 2016-01-01 | 3.8 | 6.30 | 2026-06-29 07:30 AEST |
| 56 | 172550695.gadi-pbs | ✅ | 2014-01-01 → 2015-01-01 | 3.8 | 6.33 | 2026-06-29 03:41 AEST |
Check this on Gadi: this table
# WHAT THIS DOES: prints the 'Recent runs' table: the 10 newest runs, newest first. 'ok' means
# payu's exit status was 0. SYPD is simulated years per day: the model time the run covered, divided
# by the wall-clock time it took. (If payu did not record a run's start date this shows ?; the page
# may fill it in from the previous run.)
A=/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
import time
from datetime import datetime
print("Run PBS job Result Model period Hours SYPD Record last modified")
for r in reversed(runs[-10:]): # the last 10 runs, newest first
status = r.get("payu_run_status")
result = "ok" if status == 0 else ("failed" if status is not None else str(r.get("stage")))
start, finish = r.get("model_start_time"), r.get("model_finish_time")
seconds = r.get("timings", {}).get("payu_total_duration_seconds") # wall-clock seconds the run took
hours = "%.1f" % (seconds / 3600) if seconds else ""
sypd = ""
if start and finish and seconds:
# the model time the run covered, in whole days (dates look like 1958-01-01T00:00:00)
days = (datetime.strptime(finish[:19], "%Y-%m-%dT%H:%M:%S") - datetime.strptime(start[:19], "%Y-%m-%dT%H:%M:%S")).days
sypd = "%.2f" % (days / 365.25 / (seconds / 86400)) # model years divided by wall-clock days
changed = time.strftime("%Y-%m-%d %H:%M %Z", time.localtime(os.path.getmtime(r["file"])))
print("%-4s %-18s %-10s %-10s -> %-10s %6s %6s %s" % (r.get("payu_current_run"), r.get("scheduler_job_id"), result,
(start or "?")[:10], (finish or "?")[:10], hours, sypd, changed))
EOF
Latest logs in the archive folder
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/output065/access-om3.err (last lines; modified 2026-07-01 00:00 AEST)
(t_initf) Read in prof_inparm namelist from: drv_in
(t_initf) Using profile_disable= F
(t_initf) profile_timer= 4
(t_initf) profile_depth_limit= 4
(t_initf) profile_detail_limit= 2
(t_initf) profile_barrier= F
(t_initf) profile_outpe_num= 1
(t_initf) profile_outpe_stride= 0
(t_initf) profile_single_file= F
(t_initf) profile_global_stats= T
(t_initf) profile_ovhd_measurement= F
(t_initf) profile_add_detail= F
(t_initf) profile_papi_enable= F
WARNING from PE 0: set_diffusivity_init: SIMPLE_TKE_TO_KD can not be used reliably with USE_REGRIDDING.
Memuse(MB) at Memory HiWaterMark= 8.662E+02 5.727E+03 6.977E+02 9.917E+02
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/error_logs/access-om3.172341945.gadi-pbs.err (last lines; modified 2026-06-26 12:28 AEST)
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
access-om3-MOM6-C 000000000067920E Unknown Unknown Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
Unknown
access-om3-MOM6-C 000000000067920E Unknown Unknown Unknown
Unknown
access-om3-MOM6-C 000000000067920E Unknown Unknown Unknown
Unknown
Unknown
--------------------------------------------------------------------------
mpirun noticed that process rank 117 with PID 154772 on node gadi-cpu-spr-0580 exited on signal 7 (Bus error).
--------------------------------------------------------------------------
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/output065/access-om3.out (last lines; modified 2026-07-01 00:01 AEST)
(Ocean continuity equation) 105120 307.942733 835.367698 638.145938 90.810840 0.047 31 0 2428
(Ocean pressure force) 35040 1109.411049 1288.866854 1193.285884 22.007895 0.087 31 0 2428
(Ocean vertical viscosity) 105120 39.394804 785.811064 564.154956 184.830583 0.041 31 0 2428
(Ocean horizontal viscosity) 35040 47.929453 122.683616 56.998952 3.963222 0.004 31 0 2428
(Ocean momentum increments) 70080 16.128591 39.845111 28.515315 4.104740 0.002 31 0 2428
(Ocean message passing) 315360 235.444815 1953.627175 675.768264 360.274297 0.049 31 0 2428
(Ocean barotropic mode calc) 70080 1.226363 3.957012 2.692039 0.327667 0.000 31 0 2428
(Ocean barotropic mode stepping) 70080 1328.094657 2302.367281 1644.912914 114.745023 0.120 31 0 2428
(Ocean barotropic forcing calc) 70080 21.904626 51.273771 35.651818 6.371618 0.003 31 0 2428
(Ocean kappa_shear) 4380 15.032538 744.474829 225.673641 109.420472 0.016 31 0 2428
(Ocean diabatic entrain) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 2428
(Ocean vert remap) 4380 20.099946 152.679357 92.691376 26.448914 0.007 31 0 2428
(Ocean set_diffusivity) 8760 40.894801 786.714469 266.584919 112.366682 0.019 31 0 2428
(Ocean KPP) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 2428
(Ocean tracer_columns) 4380 1839.083161 2694.124873 2299.337366 120.637145 0.168 26 0 2428
(Ocean diabatic tridiag) 4380 1.947179 2.973378 2.262624 0.125564 0.000 41 0 2428
(Ocean diabatic message passing) 8760 16.253802 1023.571080 233.518608 187.323840 0.017 41 0 2428
(Ocean differential diffusion) 0 0.000000 0.000000 0.000000 0.000000 0.000 41 0 2428
(Ocean find_uv_at_h) 8760 3.716233 7.766232 6.552979 0.970653 0.000 41 0 2428
(Ocean frazil) 8760 0.790721 7.008680 2.932985 1.355066 0.000 41 0 2428
(Ocean advect tracer) 4380 428.466034 550.536173 471.806915 21.579936 0.034 31 0 2428
(Ocean tracer halo updates) 26910 87.705623 473.193891 322.161842 87.307347 0.024 41 0 2428
(Ocean tracer global synch) 18150 21.244053 427.290080 154.492684 89.192401 0.011 41 0 2428
(Ocean diffuse tracer) 4380 49.982483 691.933050 538.508243 177.900126 0.039 31 0 2428
(Ocean epipycnal diffuse tracer) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 2428
Ocean surface forcing 70081 83.654581 489.996911 179.524161 64.560955 0.013 11 0 2428
(Ocean forcing diagnostics) 39420 39.192669 44.223285 42.099522 0.759722 0.003 41 0 2428
MPP_STACK high water mark= 0
max rss=1022414848.0 MB
med_finalize max rss=1022414848.0 MB
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/error_logs/access-om3.172341945.gadi-pbs.out (last lines; modified 2026-06-26 12:28 AEST)
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
max rss=1017049088.0 MB
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/output065/log/med.log (last lines; modified 2026-07-01 00:01 AEST)
Model Date: 2023-12-20T00:00:00 wall clock = 2026-06-30T23:52:51 avg dt = 37.14 s/day, dt = 36.23 s/day, rate = 6.53 ypd
memory_write: model date = 2023-12-20T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-21T00:00:00 wall clock = 2026-06-30T23:53:28 avg dt = 37.14 s/day, dt = 36.34 s/day, rate = 6.51 ypd
memory_write: model date = 2023-12-21T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-22T00:00:00 wall clock = 2026-06-30T23:54:04 avg dt = 37.14 s/day, dt = 36.54 s/day, rate = 6.48 ypd
memory_write: model date = 2023-12-22T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-23T00:00:00 wall clock = 2026-06-30T23:54:40 avg dt = 37.13 s/day, dt = 36.32 s/day, rate = 6.52 ypd
memory_write: model date = 2023-12-23T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-24T00:00:00 wall clock = 2026-06-30T23:55:17 avg dt = 37.13 s/day, dt = 36.40 s/day, rate = 6.50 ypd
memory_write: model date = 2023-12-24T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-25T00:00:00 wall clock = 2026-06-30T23:55:53 avg dt = 37.13 s/day, dt = 36.31 s/day, rate = 6.52 ypd
memory_write: model date = 2023-12-25T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-26T00:00:00 wall clock = 2026-06-30T23:56:29 avg dt = 37.13 s/day, dt = 36.11 s/day, rate = 6.55 ypd
memory_write: model date = 2023-12-26T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-27T00:00:00 wall clock = 2026-06-30T23:57:06 avg dt = 37.12 s/day, dt = 36.37 s/day, rate = 6.51 ypd
memory_write: model date = 2023-12-27T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-28T00:00:00 wall clock = 2026-06-30T23:57:42 avg dt = 37.12 s/day, dt = 36.02 s/day, rate = 6.57 ypd
memory_write: model date = 2023-12-28T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-29T00:00:00 wall clock = 2026-06-30T23:58:18 avg dt = 37.12 s/day, dt = 36.03 s/day, rate = 6.57 ypd
memory_write: model date = 2023-12-29T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-30T00:00:00 wall clock = 2026-06-30T23:58:54 avg dt = 37.12 s/day, dt = 36.12 s/day, rate = 6.55 ypd
memory_write: model date = 2023-12-30T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
Model Date: 2023-12-31T00:00:00 wall clock = 2026-06-30T23:59:30 avg dt = 37.11 s/day, dt = 35.98 s/day, rate = 6.58 ypd
memory_write: model date = 2023-12-31T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
-------->(med_phases_restart_write) mediating for: 2023 12 31 23 45 0 0
(med_io_wopen) creating file access-om3.cpl.r.2024-01-01-00000.nc
Model Date: 2024-01-01T00:00:00 wall clock = 2026-07-01T00:01:22 avg dt = 37.32 s/day, dt = 112.42 s/day, rate = 2.11 ypd
memory_write: model date = 2024-01-01T00:00:00 memory = -0.00 MB (highwater) 975.05 MB (usage)
SUCCESSFUL TERMINATION OF CMEPS
: =============== # simulated years / cmp-day = 6.343 ===============
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/pbs_logs/25km_jra_iaf__c.o172728426 (last lines; modified 2026-07-01 00:30 AEST)
1-00000_7.nc.0038 access-om3.mom6.r.2023-01-01-00000_7.nc.0039 access-om3.mom6.r.2023-01-01-00000_7.nc.0040 access-om3.mom6.r.2023-01-01-00000_7.nc.0041 access-om3.mom6.r.2023-01-01-00000_7.nc.0042 access-om3.mom6.r.2023-01-01-00000_7.nc.0043 access-om3.mom6.r.2023-01-01-00000_7.nc.0044 access-om3.mom6.r.2023-01-01-00000_7.nc.0045 access-om3.mom6.r.2023-01-01-00000_7.nc.0046 access-om3.mom6.r.2023-01-01-00000_7.nc.0047 access-om3.mom6.r.2023-01-01-00000_7.nc.0048 access-om3.mom6.r.2023-01-01-00000_7.nc.0050 access-om3.mom6.r.2023-01-01-00000_7.nc.0051 access-om3.mom6.r.2023-01-01-00000_7.nc.0052 access-om3.mom6.r.2023-01-01-00000_7.nc.0053
mpirun -n 4 /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast -r -o access-om3.mom6.r.2023-01-01-00000_1.nc access-om3.mom6.r.2023-01-01-00000_1.nc.0000 access-om3.mom6.r.2023-01-01-00000_1.nc.0001 access-om3.mom6.r.2023-01-01-00000_1.nc.0002 access-om3.mom6.r.2023-01-01-00000_1.nc.0003 access-om3.mom6.r.2023-01-01-00000_1.nc.0004 access-om3.mom6.r.2023-01-01-00000_1.nc.0005 access-om3.mom6.r.2023-01-01-00000_1.nc.0006 access-om3.mom6.r.2023-01-01-00000_1.nc.0007 access-om3.mom6.r.2023-01-01-00000_1.nc.0008 access-om3.mom6.r.2023-01-01-00000_1.nc.0009 access-om3.mom6.r.2023-01-01-00000_1.nc.0010 access-om3.mom6.r.2023-01-01-00000_1.nc.0011 access-om3.mom6.r.2023-01-01-00000_1.nc.0012 access-om3.mom6.r.2023-01-01-00000_1.nc.0013 access-om3.mom6.r.2023-01-01-00000_1.nc.0014 access-om3.mom6.r.2023-01-01-00000_1.nc.0015 access-om3.mom6.r.2023-01-01-00000_1.nc.0016 access-om3.mom6.r.2023-01-01-00000_1.nc.0017 access-om3.mom6.r.2023-01-01-00000_1.nc.0018 access-om3.mom6.r.2023-01-01-00000_1.nc.0019 access-om3.mom6.r.2023-01-01-00000_1.nc.0020 access-om3.mom6.r.2023-01-01-00000_1.nc.0021 access-om3.mom6.r.2023-01-01-00000_1.nc.0022 access-om3.mom6.r.2023-01-01-00000_1.nc.0023 access-om3.mom6.r.2023-01-01-00000_1.nc.0024 access-om3.mom6.r.2023-01-01-00000_1.nc.0025 access-om3.mom6.r.2023-01-01-00000_1.nc.0026 access-om3.mom6.r.2023-01-01-00000_1.nc.0027 access-om3.mom6.r.2023-01-01-00000_1.nc.0028 access-om3.mom6.r.2023-01-01-00000_1.nc.0029 access-om3.mom6.r.2023-01-01-00000_1.nc.0030 access-om3.mom6.r.2023-01-01-00000_1.nc.0031 access-om3.mom6.r.2023-01-01-00000_1.nc.0032 access-om3.mom6.r.2023-01-01-00000_1.nc.0033 access-om3.mom6.r.2023-01-01-00000_1.nc.0034 access-om3.mom6.r.2023-01-01-00000_1.nc.0035 access-om3.mom6.r.2023-01-01-00000_1.nc.0037 access-om3.mom6.r.2023-01-01-00000_1.nc.0038 access-om3.mom6.r.2023-01-01-00000_1.nc.0039 access-om3.mom6.r.2023-01-01-00000_1.nc.0040 access-om3.mom6.r.2023-01-01-00000_1.nc.0041 access-om3.mom6.r.2023-01-01-00000_1.nc.0042 access-om3.mom6.r.2023-01-01-00000_1.nc.0043 access-om3.mom6.r.2023-01-01-00000_1.nc.0044 access-om3.mom6.r.2023-01-01-00000_1.nc.0045 access-om3.mom6.r.2023-01-01-00000_1.nc.0046 access-om3.mom6.r.2023-01-01-00000_1.nc.0047 access-om3.mom6.r.2023-01-01-00000_1.nc.0048 access-om3.mom6.r.2023-01-01-00000_1.nc.0050 access-om3.mom6.r.2023-01-01-00000_1.nc.0051 access-om3.mom6.r.2023-01-01-00000_1.nc.0052 access-om3.mom6.r.2023-01-01-00000_1.nc.0053
mpirun -n 4 /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast -r -o access-om3.mom6.r.2023-01-01-00000_5.nc access-om3.mom6.r.2023-01-01-00000_5.nc.0000 access-om3.mom6.r.2023-01-01-00000_5.nc.0001 access-om3.mom6.r.2023-01-01-00000_5.nc.0002 access-om3.mom6.r.2023-01-01-00000_5.nc.0003 access-om3.mom6.r.2023-01-01-00000_5.nc.0004 access-om3.mom6.r.2023-01-01-00000_5.nc.0005 access-om3.mom6.r.2023-01-01-00000_5.nc.0006 access-om3.mom6.r.2023-01-01-00000_5.nc.0007 access-om3.mom6.r.2023-01-01-00000_5.nc.0008 access-om3.mom6.r.2023-01-01-00000_5.nc.0009 access-om3.mom6.r.2023-01-01-00000_5.nc.0010 access-om3.mom6.r.2023-01-01-00000_5.nc.0011 access-om3.mom6.r.2023-01-01-00000_5.nc.0012 access-om3.mom6.r.2023-01-01-00000_5.nc.0013 access-om3.mom6.r.2023-01-01-00000_5.nc.0014 access-om3.mom6.r.2023-01-01-00000_5.nc.0015 access-om3.mom6.r.2023-01-01-00000_5.nc.0016 access-om3.mom6.r.2023-01-01-00000_5.nc.0017 access-om3.mom6.r.2023-01-01-00000_5.nc.0018 access-om3.mom6.r.2023-01-01-00000_5.nc.0019 access-om3.mom6.r.2023-01-01-00000_5.nc.0020 access-om3.mom6.r.2023-01-01-00000_5.nc.0021 access-om3.mom6.r.2023-01-01-00000_5.nc.0022 access-om3.mom6.r.2023-01-01-00000_5.nc.0023 access-om3.mom6.r.2023-01-01-00000_5.nc.0024 access-om3.mom6.r.2023-01-01-00000_5.nc.0025 access-om3.mom6.r.2023-01-01-00000_5.nc.0026 access-om3.mom6.r.2023-01-01-00000_5.nc.0027 access-om3.mom6.r.2023-01-01-00000_5.nc.0028 access-om3.mom6.r.2023-01-01-00000_5.nc.0029 access-om3.mom6.r.2023-01-01-00000_5.nc.0030 access-om3.mom6.r.2023-01-01-00000_5.nc.0031 access-om3.mom6.r.2023-01-01-00000_5.nc.0032 access-om3.mom6.r.2023-01-01-00000_5.nc.0033 access-om3.mom6.r.2023-01-01-00000_5.nc.0034 access-om3.mom6.r.2023-01-01-00000_5.nc.0035 access-om3.mom6.r.2023-01-01-00000_5.nc.0037 access-om3.mom6.r.2023-01-01-00000_5.nc.0038 access-om3.mom6.r.2023-01-01-00000_5.nc.0039 access-om3.mom6.r.2023-01-01-00000_5.nc.0040 access-om3.mom6.r.2023-01-01-00000_5.nc.0041 access-om3.mom6.r.2023-01-01-00000_5.nc.0042 access-om3.mom6.r.2023-01-01-00000_5.nc.0043 access-om3.mom6.r.2023-01-01-00000_5.nc.0044 access-om3.mom6.r.2023-01-01-00000_5.nc.0045 access-om3.mom6.r.2023-01-01-00000_5.nc.0046 access-om3.mom6.r.2023-01-01-00000_5.nc.0047 access-om3.mom6.r.2023-01-01-00000_5.nc.0048 access-om3.mom6.r.2023-01-01-00000_5.nc.0050 access-om3.mom6.r.2023-01-01-00000_5.nc.0051 access-om3.mom6.r.2023-01-01-00000_5.nc.0052 access-om3.mom6.r.2023-01-01-00000_5.nc.0053
mpirun -n 4 /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast -r -o access-om3.mom6.r.2023-01-01-00000_4.nc access-om3.mom6.r.2023-01-01-00000_4.nc.0000 access-om3.mom6.r.2023-01-01-00000_4.nc.0001 access-om3.mom6.r.2023-01-01-00000_4.nc.0002 access-om3.mom6.r.2023-01-01-00000_4.nc.0003 access-om3.mom6.r.2023-01-01-00000_4.nc.0004 access-om3.mom6.r.2023-01-01-00000_4.nc.0005 access-om3.mom6.r.2023-01-01-00000_4.nc.0006 access-om3.mom6.r.2023-01-01-00000_4.nc.0007 access-om3.mom6.r.2023-01-01-00000_4.nc.0008 access-om3.mom6.r.2023-01-01-00000_4.nc.0009 access-om3.mom6.r.2023-01-01-00000_4.nc.0010 access-om3.mom6.r.2023-01-01-00000_4.nc.0011 access-om3.mom6.r.2023-01-01-00000_4.nc.0012 access-om3.mom6.r.2023-01-01-00000_4.nc.0013 access-om3.mom6.r.2023-01-01-00000_4.nc.0014 access-om3.mom6.r.2023-01-01-00000_4.nc.0015 access-om3.mom6.r.2023-01-01-00000_4.nc.0016 access-om3.mom6.r.2023-01-01-00000_4.nc.0017 access-om3.mom6.r.2023-01-01-00000_4.nc.0018 access-om3.mom6.r.2023-01-01-00000_4.nc.0019 access-om3.mom6.r.2023-01-01-00000_4.nc.0020 access-om3.mom6.r.2023-01-01-00000_4.nc.0021 access-om3.mom6.r.2023-01-01-00000_4.nc.0022 access-om3.mom6.r.2023-01-01-00000_4.nc.0023 access-om3.mom6.r.2023-01-01-00000_4.nc.0024 access-om3.mom6.r.2023-01-01-00000_4.nc.0025 access-om3.mom6.r.2023-01-01-00000_4.nc.0026 access-om3.mom6.r.2023-01-01-00000_4.nc.0027 access-om3.mom6.r.2023-01-01-00000_4.nc.0028 access-om3.mom6.r.2023-01-01-00000_4.nc.0029 access-om3.mom6.r.2023-01-01-00000_4.nc.0030 access-om3.mom6.r.2023-01-01-00000_4.nc.0031 access-om3.mom6.r.2023-01-01-00000_4.nc.0032 access-om3.mom6.r.2023-01-01-00000_4.nc.0033 access-om3.mom6.r.2023-01-01-00000_4.nc.0034 access-om3.mom6.r.2023-01-01-00000_4.nc.0035 access-om3.mom6.r.2023-01-01-00000_4.nc.0037 access-om3.mom6.r.2023-01-01-00000_4.nc.0038 access-om3.mom6.r.2023-01-01-00000_4.nc.0039 access-om3.mom6.r.2023-01-01-00000_4.nc.0040 access-om3.mom6.r.2023-01-01-00000_4.nc.0041 access-om3.mom6.r.2023-01-01-00000_4.nc.0042 access-om3.mom6.r.2023-01-01-00000_4.nc.0043 access-om3.mom6.r.2023-01-01-00000_4.nc.0044 access-om3.mom6.r.2023-01-01-00000_4.nc.0045 access-om3.mom6.r.2023-01-01-00000_4.nc.0046 access-om3.mom6.r.2023-01-01-00000_4.nc.0047 access-om3.mom6.r.2023-01-01-00000_4.nc.0048 access-om3.mom6.r.2023-01-01-00000_4.nc.0050 access-om3.mom6.r.2023-01-01-00000_4.nc.0051 access-om3.mom6.r.2023-01-01-00000_4.nc.0052 access-om3.mom6.r.2023-01-01-00000_4.nc.0053
mpirun -n 4 /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast -r -o ocean_wombatlite_airsea_flux.res.nc ocean_wombatlite_airsea_flux.res.nc.0000 ocean_wombatlite_airsea_flux.res.nc.0001 ocean_wombatlite_airsea_flux.res.nc.0002 ocean_wombatlite_airsea_flux.res.nc.0003 ocean_wombatlite_airsea_flux.res.nc.0004 ocean_wombatlite_airsea_flux.res.nc.0005 ocean_wombatlite_airsea_flux.res.nc.0006 ocean_wombatlite_airsea_flux.res.nc.0007 ocean_wombatlite_airsea_flux.res.nc.0008 ocean_wombatlite_airsea_flux.res.nc.0009 ocean_wombatlite_airsea_flux.res.nc.0010 ocean_wombatlite_airsea_flux.res.nc.0011 ocean_wombatlite_airsea_flux.res.nc.0012 ocean_wombatlite_airsea_flux.res.nc.0013 ocean_wombatlite_airsea_flux.res.nc.0014 ocean_wombatlite_airsea_flux.res.nc.0015 ocean_wombatlite_airsea_flux.res.nc.0016 ocean_wombatlite_airsea_flux.res.nc.0017 ocean_wombatlite_airsea_flux.res.nc.0018 ocean_wombatlite_airsea_flux.res.nc.0019 ocean_wombatlite_airsea_flux.res.nc.0020 ocean_wombatlite_airsea_flux.res.nc.0021 ocean_wombatlite_airsea_flux.res.nc.0022 ocean_wombatlite_airsea_flux.res.nc.0023 ocean_wombatlite_airsea_flux.res.nc.0024 ocean_wombatlite_airsea_flux.res.nc.0025 ocean_wombatlite_airsea_flux.res.nc.0026 ocean_wombatlite_airsea_flux.res.nc.0027 ocean_wombatlite_airsea_flux.res.nc.0028 ocean_wombatlite_airsea_flux.res.nc.0029 ocean_wombatlite_airsea_flux.res.nc.0030 ocean_wombatlite_airsea_flux.res.nc.0031 ocean_wombatlite_airsea_flux.res.nc.0032 ocean_wombatlite_airsea_flux.res.nc.0033 ocean_wombatlite_airsea_flux.res.nc.0034 ocean_wombatlite_airsea_flux.res.nc.0035 ocean_wombatlite_airsea_flux.res.nc.0037 ocean_wombatlite_airsea_flux.res.nc.0038 ocean_wombatlite_airsea_flux.res.nc.0039 ocean_wombatlite_airsea_flux.res.nc.0040 ocean_wombatlite_airsea_flux.res.nc.0041 ocean_wombatlite_airsea_flux.res.nc.0042 ocean_wombatlite_airsea_flux.res.nc.0043 ocean_wombatlite_airsea_flux.res.nc.0044 ocean_wombatlite_airsea_flux.res.nc.0045 ocean_wombatlite_airsea_flux.res.nc.0046 ocean_wombatlite_airsea_flux.res.nc.0047 ocean_wombatlite_airsea_flux.res.nc.0048 ocean_wombatlite_airsea_flux.res.nc.0050 ocean_wombatlite_airsea_flux.res.nc.0051 ocean_wombatlite_airsea_flux.res.nc.0052 ocean_wombatlite_airsea_flux.res.nc.0053
mpirun -n 4 /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast -r -o access-om3.mom6.r.2023-01-01-00000_6.nc access-om3.mom6.r.2023-01-01-00000_6.nc.0000 access-om3.mom6.r.2023-01-01-00000_6.nc.0001 access-om3.mom6.r.2023-01-01-00000_6.nc.0002 access-om3.mom6.r.2023-01-01-00000_6.nc.0003 access-om3.mom6.r.2023-01-01-00000_6.nc.0004 access-om3.mom6.r.2023-01-01-00000_6.nc.0005 access-om3.mom6.r.2023-01-01-00000_6.nc.0006 access-om3.mom6.r.2023-01-01-00000_6.nc.0007 access-om3.mom6.r.2023-01-01-00000_6.nc.0008 access-om3.mom6.r.2023-01-01-00000_6.nc.0009 access-om3.mom6.r.2023-01-01-00000_6.nc.0010 access-om3.mom6.r.2023-01-01-00000_6.nc.0011 access-om3.mom6.r.2023-01-01-00000_6.nc.0012 access-om3.mom6.r.2023-01-01-00000_6.nc.0013 access-om3.mom6.r.2023-01-01-00000_6.nc.0014 access-om3.mom6.r.2023-01-01-00000_6.nc.0015 access-om3.mom6.r.2023-01-01-00000_6.nc.0016 access-om3.mom6.r.2023-01-01-00000_6.nc.0017 access-om3.mom6.r.2023-01-01-00000_6.nc.0018 access-om3.mom6.r.2023-01-01-00000_6.nc.0019 access-om3.mom6.r.2023-01-01-00000_6.nc.0020 access-om3.mom6.r.2023-01-01-00000_6.nc.0021 access-om3.mom6.r.2023-01-01-00000_6.nc.0022 access-om3.mom6.r.2023-01-01-00000_6.nc.0023 access-om3.mom6.r.2023-01-01-00000_6.nc.0024 access-om3.mom6.r.2023-01-01-00000_6.nc.0025 access-om3.mom6.r.2023-01-01-00000_6.nc.0026 access-om3.mom6.r.2023-01-01-00000_6.nc.0027 access-om3.mom6.r.2023-01-01-00000_6.nc.0028 access-om3.mom6.r.2023-01-01-00000_6.nc.0029 access-om3.mom6.r.2023-01-01-00000_6.nc.0030 access-om3.mom6.r.2023-01-01-00000_6.nc.0031 access-om3.mom6.r.2023-01-01-00000_6.nc.0032 access-om3.mom6.r.2023-01-01-00000_6.nc.0033 access-om3.mom6.r.2023-01-01-00000_6.nc.0034 access-om3.mom6.r.2023-01-01-00000_6.nc.0035 access-om3.mom6.r.2023-01-01-00000_6.nc.0037 access-om3.mom6.r.2023-01-01-00000_6.nc.0038 access-om3.mom6.r.2023-01-01-00000_6.nc.0039 access-om3.mom6.r.2023-01-01-00000_6.nc.0040 access-om3.mom6.r.2023-01-01-00000_6.nc.0041 access-om3.mom6.r.2023-01-01-00000_6.nc.0042 access-om3.mom6.r.2023-01-01-00000_6.nc.0043 access-om3.mom6.r.2023-01-01-00000_6.nc.0044 access-om3.mom6.r.2023-01-01-00000_6.nc.0045 access-om3.mom6.r.2023-01-01-00000_6.nc.0046 access-om3.mom6.r.2023-01-01-00000_6.nc.0047 access-om3.mom6.r.2023-01-01-00000_6.nc.0048 access-om3.mom6.r.2023-01-01-00000_6.nc.0050 access-om3.mom6.r.2023-01-01-00000_6.nc.0051 access-om3.mom6.r.2023-01-01-00000_6.nc.0052 access-om3.mom6.r.2023-01-01-00000_6.nc.0053
mpirun -n 4 /g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast -r -o access-om3.mom6.r.2023-01-01-00000.nc access-om3.mom6.r.2023-01-01-00000.nc.0000 access-om3.mom6.r.2023-01-01-00000.nc.0001 access-om3.mom6.r.2023-01-01-00000.nc.0002 access-om3.mom6.r.2023-01-01-00000.nc.0003 access-om3.mom6.r.2023-01-01-00000.nc.0004 access-om3.mom6.r.2023-01-01-00000.nc.0005 access-om3.mom6.r.2023-01-01-00000.nc.0006 access-om3.mom6.r.2023-01-01-00000.nc.0007 access-om3.mom6.r.2023-01-01-00000.nc.0008 access-om3.mom6.r.2023-01-01-00000.nc.0009 access-om3.mom6.r.2023-01-01-00000.nc.0010 access-om3.mom6.r.2023-01-01-00000.nc.0011 access-om3.mom6.r.2023-01-01-00000.nc.0012 access-om3.mom6.r.2023-01-01-00000.nc.0013 access-om3.mom6.r.2023-01-01-00000.nc.0014 access-om3.mom6.r.2023-01-01-00000.nc.0015 access-om3.mom6.r.2023-01-01-00000.nc.0016 access-om3.mom6.r.2023-01-01-00000.nc.0017 access-om3.mom6.r.2023-01-01-00000.nc.0018 access-om3.mom6.r.2023-01-01-00000.nc.0019 access-om3.mom6.r.2023-01-01-00000.nc.0020 access-om3.mom6.r.2023-01-01-00000.nc.0021 access-om3.mom6.r.2023-01-01-00000.nc.0022 access-om3.mom6.r.2023-01-01-00000.nc.0023 access-om3.mom6.r.2023-01-01-00000.nc.0024 access-om3.mom6.r.2023-01-01-00000.nc.0025 access-om3.mom6.r.2023-01-01-00000.nc.0026 access-om3.mom6.r.2023-01-01-00000.nc.0027 access-om3.mom6.r.2023-01-01-00000.nc.0028 access-om3.mom6.r.2023-01-01-00000.nc.0029 access-om3.mom6.r.2023-01-01-00000.nc.0030 access-om3.mom6.r.2023-01-01-00000.nc.0031 access-om3.mom6.r.2023-01-01-00000.nc.0032 access-om3.mom6.r.2023-01-01-00000.nc.0033 access-om3.mom6.r.2023-01-01-00000.nc.0034 access-om3.mom6.r.2023-01-01-00000.nc.0035 access-om3.mom6.r.2023-01-01-00000.nc.0037 access-om3.mom6.r.2023-01-01-00000.nc.0038 access-om3.mom6.r.2023-01-01-00000.nc.0039 access-om3.mom6.r.2023-01-01-00000.nc.0040 access-om3.mom6.r.2023-01-01-00000.nc.0041 access-om3.mom6.r.2023-01-01-00000.nc.0042 access-om3.mom6.r.2023-01-01-00000.nc.0043 access-om3.mom6.r.2023-01-01-00000.nc.0044 access-om3.mom6.r.2023-01-01-00000.nc.0045 access-om3.mom6.r.2023-01-01-00000.nc.0046 access-om3.mom6.r.2023-01-01-00000.nc.0047 access-om3.mom6.r.2023-01-01-00000.nc.0048 access-om3.mom6.r.2023-01-01-00000.nc.0050 access-om3.mom6.r.2023-01-01-00000.nc.0051 access-om3.mom6.r.2023-01-01-00000.nc.0052 access-om3.mom6.r.2023-01-01-00000.nc.0053
payu: Found modules in /opt/Modules/v4.3.0
======================================================================================
Resource Usage on 2026-07-01 00:30:26:
Job Id: 172728426.gadi-pbs
Project: ol01
Exit Status: 255
Service Units: 5.46
NCPUs Requested: 4 CPU Time Used: 01:23:47
Memory Requested: 30.0GB Memory Used: 30.0GB
Walltime Requested: 06:00:00 Walltime Used: 00:27:17
JobFS Requested: 10.0GB JobFS Used: 204.22KB
======================================================================================
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/pbs_logs/25km_jra_iaf__c.e172728426 (last lines; modified 2026-07-01 00:30 AEST)
Error message:
Copying non-collated variables
[rank 000] ERROR in NetCDF /scratch/tm70/tm70_ci/tmp/spack-stage/spack-stage-mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/spack-src/mppnccombine-fast.c:206 -46 NetCDF: Invalid dimension ID or name
/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast[0x40a8bf]
/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast[0x40567a]
/half-root/usr/lib64/libc.so.6(__libc_start_main+0xe5)[0x14b587b6e865]
/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast[0x404c0e]
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
Proc: [[46433,1],0]
Errorcode: -46
NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.
--------------------------------------------------------------------------
[gadi-cpu-spr-0070:4027410] [0] func:/apps/openmpi/5.0.5/lib/libopen-pal.so.80(opal_backtrace_buffer+0x20) [0x14b5841b4ce0]
[gadi-cpu-spr-0070:4027410] [1] func:/apps/openmpi/5.0.5/lib/libmpi.so.40(ompi_mpi_abort+0x122) [0x14b5883ac442]
[gadi-cpu-spr-0070:4027410] [2] func:/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast() [0x40a8db]
[gadi-cpu-spr-0070:4027410] [3] func:/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast() [0x40567a]
[gadi-cpu-spr-0070:4027410] [4] func:/half-root/usr/lib64/libc.so.6(__libc_start_main+0xe5) [0x14b587b6e865]
[gadi-cpu-spr-0070:4027410] [5] func:/g/data/vk83/apps/spack/0.22/release/linux-rocky8-x86_64_v4/intel-2021.10.0/mppnccombine-fast-2025.07.000-6kyw6dwnfpnz3k74v3dkjmzkglpdxvq5/bin/mppnccombine-fast() [0x404c0e]
--------------------------------------------------------------------------
prterun has exited due to process rank 0 with PID 0 on node gadi-cpu-spr-0070 calling
"abort". This may have caused other processes in the application to be
terminated by signals sent by prterun (as reported here).
--------------------------------------------------------------------------
Check this on Gadi: when these logs were written, and their last lines
# WHAT THIS DOES: for each log file listed above, prints when it was last changed and its last 30
# lines, which are the lines the page shows.
# 'for' repeats the three lines below once for each file; $f is the file it is on.
# 'ls -l' lists a file with details; --time-style=long-iso writes its time as 2026-07-01 00:02.
for f in /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/output065/access-om3.err /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/error_logs/access-om3.172341945.gadi-pbs.err /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/output065/access-om3.out /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/error_logs/access-om3.172341945.gadi-pbs.out /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/output065/log/med.log /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/pbs_logs/25km_jra_iaf__c.o172728426 /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/pbs_logs/25km_jra_iaf__c.e172728426; do
echo "== $f" # which file this is
ls -l --time-style=long-iso "$f" # when it was last changed (the date and time are in the middle)
tail -n 30 "$f" # its last 30 lines
done
/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359/metadata.yaml
contact: Andrew Kiss
email: 31054815+aekiss@users.noreply.github.com
created: '2026-06-16'
description: |-
25km ACCESS-OM3 global configuration under interannual forcing.
Initial conditions of temperature and salinity are from WOA23.
Run with JRA55-do v1.6.0 interannually-varying forcing, 1 Jan 1958 - 31 Dec 2023.
This configuration includes WOMBATlite biogeochemistry with time-varying, globally-
averaged atmospheric CO2 forcing. Initial WOMBATlite prognostic tracer
concentrations are:
- no3, o2, dic, alk: from GLODAPv2 2016b
- fe: from Tagliabue et al. 2016 (doi:10.1002/2015GB005289)
- remaining tracers: set to constant values
notes: |-
ACCESS-NRI and COSIMA request that users of this model acknowledge those organisations
following the examples at https://www.access-nri.org.au/resources/acknowledging-us/
and https://cosima.org.au/index.php/get-involved/
keywords:
- JRA55
- access-om3
- interannual
- global
- bgc
realm:
- ocean
- seaIce
- ocnBgchem
nominal_resolution: 25 km
license: CC-BY-4.0
model: ACCESS-OM3
url: https://github.com/ACCESS-NRI/access-om3-configs
version: 0.0
experiment_uuid: d28e0359-7b3b-4d29-81a8-bd03deb2b0d9
name: MC_25km_jra_iaf+wombatlite-test4-d28e0359
schema_version: 1-0-3
long_description: 4th IAF test run # Long description of the experiment (string)
# frequency: The frequency(/ies) included in the experiment (array of strings)
# variable: The variable(s) included in the experiment (array of strings)
# reference: Citation or reference information (string)
# parent_experiment: experiment_uuid for parent experiment if appropriate (string)
# related_experiments: experiment_uuids for any related experiment(s) (array of strings)
Truncation locations for all completed run segments
⚠️ MOM6 truncated velocities 10 times in 1 of 66 output folders (the Truncs column of each outputNNN/ocean.stats). MOM6 truncates a velocity when its CFL number exceeds CFL_TRUNCATE (0.5 by default), and stops the run if more than MAXTRUNC truncations happen between energy saves (MOM6's default is 0, so any truncation stops it; the ACCESS-OM3 configurations set 10,000). See MOM6 velocity truncations for how to read the records, and Examine_truncation_data.ipynb for mapping them.
Location records are the detailed records MOM6 also writes, one per truncation, to U_velocity_truncations (zonal velocity) and V_velocity_truncations (meridional velocity); the location table below is built from them. MOM6 writes at most 50 per PE per run, so there are usually fewer of them than truncations; – means no file of that kind.
| Output folder | Model period | Truncations | Max CFL | Location records (U / V) | PEs at the record cap |
|---|---|---|---|---|---|
| output000 | ? → 1959-01 | 10 | 0.33 | 1 / 9 | – |
The other 65 output folders had no truncations.
Where MOM6 truncates (1° boxes, most records first; from the U_velocity_truncations and V_velocity_truncations files):
| Location | Records | Grid cells | Seen in output | Velocity | Layers | Max speed (m/s) | Max CFL | Surface height (m) | Depth (m) |
|---|---|---|---|---|---|---|---|---|---|
| 65.2°W, 61.3°N | 9 | 5 | 000 | U, V | 1–1 | 32.8 | 2.23 | 0.04 | 153 |
| 65.6°W, 61.4°N | 1 | 1 | 000 | V | 1–1 | 8.2 | 0.55 | 0.02 | 243 |
* MOM6 writes at most 50 records per PE per run (MAX_TRUNC_FILE_SIZE_PER_PE), and this box's PE hit that cap in at least one run, so it truncated more often than the record count shows. Surface height is the top-interface height e, shown only for records that include layer 1; depth is the shallower of the two cells (D).
Check this on Gadi: the truncation counts, locations and settings
# WHAT THIS DOES: re-checks the 'Truncation locations' section from the files MOM6 wrote. A
# truncation is when MOM6 had to cut a velocity that was too fast (its CFL number was above 0.5): a
# sign of numerical trouble. The steps below are separate, so you can paste just the one you need.
# Each prints something you can compare with the page.
A=/g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359
# Step 1: for each output folder: how many truncations MOM6 reported, the largest CFL number, and how many location records were written
for o in "$A"/output[0-9]*; do
# ocean.stats is a table with one line per time step. Its 3rd column ('Truncs') is the truncations at that
# step. awk adds that column up. (-F, means the columns are separated by commas. The header line is skipped
# because its first column is not a number.)
total=$(awk -F, '$1+0 == $1 {sum += $3} END {print sum+0}' "$o/ocean.stats" 2>/dev/null)
# The 5th column holds two numbers; the second one is the CFL number. awk keeps the largest.
cfl=$(awk -F, '$1+0 == $1 {split($5, a, " "); if (a[2] > max) max = a[2]} END {if (max != "") printf "%.2f", max}' "$o/ocean.stats" 2>/dev/null)
# Every truncation also gets a 'location record' in the U and V files. grep -c counts the matching lines.
u=$(grep -c 'velocity violation' "$o/U_velocity_truncations" 2>/dev/null)
v=$(grep -c 'velocity violation' "$o/V_velocity_truncations" 2>/dev/null)
# ${x:--} means: use x, or a dash if x is empty (no such file)
echo "$(basename "$o"): truncations ${total:--}, max CFL ${cfl:--}, location records U/V ${u:--}/${v:--}"
done
# Step 2: where it happens. Each record contains the place, like '( 73.27 E 67.89 N)'. This counts the records in
# boxes one degree wide (negative longitude means west), busiest first. The pipeline does this:
# grep: keep the lines that start a record sed: keep only the longitude and latitude
# awk: round them to whole degrees sort | uniq -c: count the records in each box
# sort -rn | head: busiest boxes first
grep -h 'velocity violation' "$A"/output[0-9]*/[UV]_velocity_truncations 2>/dev/null \
| sed -E 's/.*\( *(-?[0-9.]+) E +(-?[0-9.]+) N\).*/\1 \2/' \
| awk '{printf "%4.0fE %4.0fN\n", $1, $2}' \
| sort | uniq -c | sort -rn | head -n 15
# Step 3: MOM6 writes at most 50 records per PE (one of the parallel processes) in one run, then stops. A PE with
# 50 records may have truncated more often than the records show. This lists the PEs that reached 50.
for f in "$A"/output[0-9]*/[UV]_velocity_truncations; do
[ -f "$f" ] || continue # skip if there is no such file
# each record says which PE it came from ('violation at 6073:'); count the records per PE, keep PEs with 50
busy=$(grep -h 'velocity violation' "$f" | sed -E 's/.*violation at ([0-9]+):.*/PE\1/' | sort | uniq -c | awk '$1 >= 50 {printf "%s (%d records) ", $2, $1}')
echo "$(basename "$(dirname "$f")")/$(basename "$f"): $busy"
done
# Step 4: one record in full. The page reads the speed (u, v), the CFL number, the surface height (e) and the
# depth (D) from lines like these.
head -n 30 "$A"/output000/U_velocity_truncations
# Step 5: the model period of each output folder, from payu's run records (output NNN is run NNN)
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
for r in runs:
run, finish = r.get("payu_current_run"), r.get("model_finish_time")
if str(run).isdigit() and finish: # only runs that finished have a model period
start = r.get("model_start_time") or "?" # payu did not record the start of the first run
print("output%03d %s -> %s" % (int(run), start[:7], finish[:7]))
EOF
Settings changed during this run
This comes from the git history of the payu control directory, and this experiment has no run_dir in experiments.yml. Add one to see here every edit made to the model's settings and build during the run, with the old and new values.