MC_25km_jra_iaf+wombatlite-test5-38867fcd

25km JRA IAF test 5 (stopped)

❌ Latest submission: Run 16 failed (payu exit status 1): probably a model crash. See the logs below. Model date reached: 1974-01-01 (16.0 model years complete).

Notebooks: ocean_stats, timeseries, Timeseries_daily_extreme_from_2D_fields, temp-salt-vs-depth-time

📋 Summary for the GitHub issue

### OM3 Monitor summary: `MC_25km_jra_iaf+wombatlite-test5-38867fcd`
*25km JRA IAF test 5 (stopped)*

- **Latest submission:** ❌ Run 16 failed (payu exit status 1): probably a model crash. (run 16, PBS job 179499766.gadi-pbs, last recorded 2026-09-21 19:25 AEST, model date 1974-01-01)
- **Model date reached:** 1974-01-01 (16.0 model years complete)
- **Average throughput:** 5.40 SYPD
- **Truncations:** 10 in 1 of 16 output folders

<details><summary>Last lines of the model logs</summary>

`access-om3.err` [1]
```
 (t_initf)       profile_global_stats=     T
 (t_initf)       profile_ovhd_measurement= F
 (t_initf)       profile_add_detail=       F
 (t_initf)       profile_papi_enable=      F
WARNING from PE     0: set_diffusivity_init: SIMPLE_TKE_TO_KD can not be used reliably with USE_REGRIDDING.
WARNING from PE     0: diag_manager_mod::closing_file: module/output_field ocean_model/total_hfevapds, skip one time level, maybe send_data never called
WARNING from PE     0: diag_manager_mod::closing_file: module/output_field ocean_model/total_heat_content_massin, skip one time level, maybe send_data never called
                               Memuse(MB) at Memory HiWaterMark=  8.617E+02  6.634E+03  8.294E+02  1.006E+03
```
`access-om3.out` [2]
```
(Ocean tracer global synch)          18105     14.281316    461.657668    147.898966    102.536214  0.009    41     0  2429
(Ocean diffuse tracer)                4380     51.033526    770.800106    597.495499    197.687638  0.037    31     0  2429
(Ocean epipycnal diffuse tracer)         0      0.000000      0.000000      0.000000      0.000000  0.000    31     0  2429
Ocean surface forcing                70081     78.797491    436.930813    177.380369     65.133821  0.011    11     0  2429
(Ocean forcing diagnostics)          39420     36.372946     39.472944     37.666157      0.465821  0.002    41     0  2429
 MPP_STACK high water mark=           0
  max rss=994824192.0 MB
med_finalize max rss=994824192.0 MB
```
</details>

| Where | Records | Output folders | Max speed (m/s) | Max CFL |
|---|---:|---|---:|---:|
| [65.2°W, 61.3°N](https://www.google.com/maps/place/61.34,-65.25/@61.34,-65.25,5z) | 9 | 000 | 32.6 | 2.22 |
| [65.6°W, 61.4°N](https://www.google.com/maps/place/61.43,-65.62/@61.43,-65.62,5z) | 1 | 000 | 8.1 | 0.55 |

**References**
1. `/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/access-om3.err`
2. `/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/access-om3.out`
3. Archive folder: `/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd`
4. Status page: https://chrisb13.github.io/access-om3-live/live/MC_25km_jra_iaf%2Bwombatlite-test5-38867fcd/

*Generated automatically by the [OM3 Monitor](https://chrisb13.github.io/access-om3-live/live/) on 2026-10-19 16:38 AEDT; it may contain errors, so check against the files above.*

Archive folder: completed runs

Read from the payu archive folder /scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd.

Output folders: 16 (latest output015)
Model date reached: 1974-01-01
Runs recorded by payu: 17, of which failed: 1

Check this on Gadi: these counts

# WHAT THIS DOES: re-counts the 'Archive folder' list: how many output folders there are, which is
# the latest, how many runs payu recorded, how many of those failed, and how far the model got. The
# four lines it prints should match the page.
A=/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd

# Step 1: count the output folders (output000, output001, ...) and name the last one
ls -d "$A"/output[0-9]* | wc -l
basename "$(ls -d "$A"/output[0-9]* | tail -n 1)"

# Step 2: read payu's run records with a few lines of python, and count them
python3 - "$A" <<'EOF'
import glob, json, os, sys

# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1]                  # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
    try:
        with open(path) as f:
            record = json.load(f)
        record["file"] = path          # remember which file it came from
        runs.append(record)
    except (OSError, ValueError):
        pass                           # skip a file we may not read, or that is not valid JSON

# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))

# a run failed if payu's exit status for it is not 0 (a run with no status yet is still going)
failed = [r for r in runs if r.get("payu_run_status") not in (0, None)]
print(len(runs), "runs recorded by payu,", len(failed), "failed")
# the model date reached is the finish date of the last run that has one
finished = [r for r in runs if r.get("model_finish_time")]
print("model date reached:", finished[-1]["model_finish_time"][:10] if finished else "unknown")
EOF

Model throughput

Simulated years per day, made by plot_sypd.py from the outputNNN/log/med.log files of the completed submissions in /scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd.

Simulated years per day

Check this on Gadi: the rate in each output's med.log

# WHAT THIS DOES: for each output folder, finds the model's speed in its med.log (the mediator's
# log) and averages it. The speed, called the 'rate', is in simulated years per day (ypd). The
# figure plots these rates. You get one line per output folder: how many rate lines it had, and
# their average.
A=/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd

for o in "$A"/output[0-9]*; do
    [ -f "$o/log/med.log" ] || continue            # skip folders that have no med.log
    # A line looks like: Model Date: 1958-09-04T00:00:00 ... rate = 7.09 ypd. awk reads the number just before
    # 'ypd' (the second-last word), adds them up and counts them, then prints the average at the end.
    awk -v name="$(basename "$o")" '/Model Date:/ && /ypd/ {count++; total += $(NF-1)} END {if (count) printf "%s: %d rate lines, mean %.2f ypd\n", name, count, total/count}' "$o/log/med.log"
done

Recent runs

From payu's job records in /scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/payu_jobs; times are Canberra time.

Run	PBS job	Result	Model period	Wall time (h)	SYPD	Record last modified
16	179499766.gadi-pbs	❌	1974-01-01 → ?	4.4		2026-09-21 19:25 AEST
15	179470130.gadi-pbs	✅	1973-01-01 → 1974-01-01	4.5	5.36	2026-09-21 15:01 AEST
14	179457069.gadi-pbs	✅	1972-01-01 → 1973-01-01	4.4	5.41	2026-09-21 10:32 AEST
13	179451713.gadi-pbs	✅	1971-01-01 → 1972-01-01	4.4	5.45	2026-09-21 06:02 AEST
12	179445135.gadi-pbs	✅	1970-01-01 → 1971-01-01	4.5	5.39	2026-09-21 01:35 AEST
11	179437528.gadi-pbs	✅	1969-01-01 → 1970-01-01	4.4	5.40	2026-09-20 21:06 AEST
10	179428622.gadi-pbs	✅	1968-01-01 → 1969-01-01	4.5	5.39	2026-09-20 16:37 AEST
9	179418051.gadi-pbs	✅	1967-01-01 → 1968-01-01	4.5	5.39	2026-09-20 12:07 AEST
8	179410390.gadi-pbs	✅	1966-01-01 → 1967-01-01	4.4	5.39	2026-09-20 07:38 AEST
7	179398831.gadi-pbs	✅	1965-01-01 → 1966-01-01	4.4	5.44	2026-09-20 03:10 AEST

Check this on Gadi: this table

# WHAT THIS DOES: prints the 'Recent runs' table: the 10 newest runs, newest first. 'ok' means
# payu's exit status was 0. SYPD is simulated years per day: the model time the run covered, divided
# by the wall-clock time it took. (If payu did not record a run's start date this shows ?; the page
# may fill it in from the previous run.)
A=/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd

python3 - "$A" <<'EOF'
import glob, json, os, sys

# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1]                  # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
    try:
        with open(path) as f:
            record = json.load(f)
        record["file"] = path          # remember which file it came from
        runs.append(record)
    except (OSError, ValueError):
        pass                           # skip a file we may not read, or that is not valid JSON

# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))

import time
from datetime import datetime

print("Run  PBS job             Result     Model period              Hours  SYPD   Record last modified")
for r in reversed(runs[-10:]):                     # the last 10 runs, newest first
    status = r.get("payu_run_status")
    result = "ok" if status == 0 else ("failed" if status is not None else str(r.get("stage")))
    start, finish = r.get("model_start_time"), r.get("model_finish_time")
    seconds = r.get("timings", {}).get("payu_total_duration_seconds")     # wall-clock seconds the run took
    hours = "%.1f" % (seconds / 3600) if seconds else ""
    sypd = ""
    if start and finish and seconds:
        # the model time the run covered, in whole days (dates look like 1958-01-01T00:00:00)
        days = (datetime.strptime(finish[:19], "%Y-%m-%dT%H:%M:%S") - datetime.strptime(start[:19], "%Y-%m-%dT%H:%M:%S")).days
        sypd = "%.2f" % (days / 365.25 / (seconds / 86400))     # model years divided by wall-clock days
    changed = time.strftime("%Y-%m-%d %H:%M %Z", time.localtime(os.path.getmtime(r["file"])))
    print("%-4s %-18s %-10s %-10s -> %-10s %6s %6s  %s" % (r.get("payu_current_run"), r.get("scheduler_job_id"), result,
          (start or "?")[:10], (finish or "?")[:10], hours, sypd, changed))
EOF

Latest logs in the archive folder

/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/access-om3.err (last lines; modified 2026-09-21 15:00 AEST)

 (t_initf) Read in prof_inparm namelist from: drv_in
 (t_initf) Using profile_disable=          F
 (t_initf)       profile_timer=                      4
 (t_initf)       profile_depth_limit=                4
 (t_initf)       profile_detail_limit=               2
 (t_initf)       profile_barrier=          F
 (t_initf)       profile_outpe_num=                  1
 (t_initf)       profile_outpe_stride=               0
 (t_initf)       profile_single_file=      F
 (t_initf)       profile_global_stats=     T
 (t_initf)       profile_ovhd_measurement= F
 (t_initf)       profile_add_detail=       F
 (t_initf)       profile_papi_enable=      F

WARNING from PE     0: set_diffusivity_init: SIMPLE_TKE_TO_KD can not be used reliably with USE_REGRIDDING.


WARNING from PE     0: diag_manager_mod::closing_file: module/output_field ocean_model/total_hfevapds, skip one time level, maybe send_data never called


WARNING from PE     0: diag_manager_mod::closing_file: module/output_field ocean_model/total_heat_content_massin, skip one time level, maybe send_data never called

                               Memuse(MB) at Memory HiWaterMark=  8.617E+02  6.634E+03  8.294E+02  1.006E+03

/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/access-om3.out (last lines; modified 2026-09-21 15:01 AEST)

(Ocean continuity equation)         105120    316.278032    815.474389    640.666294     93.623695  0.040    31     0  2429
(Ocean pressure force)               35040   1121.286055   1241.851387   1200.271244     20.732916  0.075    31     0  2429
(Ocean vertical viscosity)          175200     93.154705   1154.139869    876.146120    273.658007  0.055    31     0  2429
(Ocean horizontal viscosity)         35040     76.909790    338.213882    229.564631     47.924312  0.014    31     0  2429
(Ocean momentum increments)          70080     16.247861     44.465946     31.287701      4.557157  0.002    31     0  2429
(Ocean message passing)             490560    269.691718   2251.366592    708.810550    415.000420  0.044    31     0  2429
(Ocean barotropic mode calc)         70080      1.464687      5.743879      2.191464      0.499198  0.000    31     0  2429
(Ocean barotropic mode stepping)     70080   1188.730190   2513.055898   1553.674221    195.167821  0.097    31     0  2429
(Ocean barotropic forcing calc)      70080     20.744652     55.466532     35.689471      8.297128  0.002    31     0  2429
(Ocean kappa_shear)                   4380     17.097308    949.126995    235.598296    116.105315  0.015    31     0  2429
(Ocean diabatic entrain)                 0      0.000000      0.000000      0.000000      0.000000  0.000    31     0  2429
(Ocean vert remap)                    4380     20.390865    156.488919     91.460146     25.727569  0.006    31     0  2429
(Ocean set_diffusivity)               8760     50.453790   1004.702299    287.922761    119.704878  0.018    31     0  2429
(Ocean KPP)                              0      0.000000      0.000000      0.000000      0.000000  0.000    31     0  2429
(Ocean tracer_columns)                4380   1806.778606   3135.060078   2472.765297    108.480654  0.155    26     0  2429
(Ocean diabatic tridiag)              4380      1.914145      3.179507      2.300865      0.137853  0.000    41     0  2429
(Ocean diabatic message passing)      8760      3.951239    947.473819    193.872800    155.354471  0.012    41     0  2429
(Ocean differential diffusion)           0      0.000000      0.000000      0.000000      0.000000  0.000    41     0  2429
(Ocean find_uv_at_h)                  8760      3.672232      7.684366      6.434889      0.956820  0.000    41     0  2429
(Ocean frazil)                        8760      0.778075      7.008531      2.959415      1.362275  0.000    41     0  2429
(Ocean advect tracer)                 4380    486.496114    589.521326    521.895336     21.485445  0.033    31     0  2429
(Ocean tracer halo updates)          26865     95.994359    515.843406    370.496124    101.176720  0.023    41     0  2429
(Ocean tracer global synch)          18105     14.281316    461.657668    147.898966    102.536214  0.009    41     0  2429
(Ocean diffuse tracer)                4380     51.033526    770.800106    597.495499    197.687638  0.037    31     0  2429
(Ocean epipycnal diffuse tracer)         0      0.000000      0.000000      0.000000      0.000000  0.000    31     0  2429
Ocean surface forcing                70081     78.797491    436.930813    177.380369     65.133821  0.011    11     0  2429
(Ocean forcing diagnostics)          39420     36.372946     39.472944     37.666157      0.465821  0.002    41     0  2429
 MPP_STACK high water mark=           0
  max rss=994824192.0 MB
med_finalize max rss=994824192.0 MB

/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/log/med.log (last lines; modified 2026-09-21 15:01 AEST)

Model Date: 1973-12-20T00:00:00 wall clock = 2026-09-21T14:51:02 avg dt =    43.45 s/day, dt =    43.92 s/day, rate =     5.39 ypd
 memory_write: model date = 1973-12-20T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-21T00:00:00 wall clock = 2026-09-21T14:51:46 avg dt =    43.46 s/day, dt =    43.79 s/day, rate =     5.41 ypd
 memory_write: model date = 1973-12-21T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-22T00:00:00 wall clock = 2026-09-21T14:52:29 avg dt =    43.46 s/day, dt =    43.56 s/day, rate =     5.43 ypd
 memory_write: model date = 1973-12-22T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-23T00:00:00 wall clock = 2026-09-21T14:53:13 avg dt =    43.46 s/day, dt =    43.61 s/day, rate =     5.43 ypd
 memory_write: model date = 1973-12-23T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-24T00:00:00 wall clock = 2026-09-21T14:53:57 avg dt =    43.46 s/day, dt =    43.76 s/day, rate =     5.41 ypd
 memory_write: model date = 1973-12-24T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-25T00:00:00 wall clock = 2026-09-21T14:54:40 avg dt =    43.46 s/day, dt =    43.52 s/day, rate =     5.44 ypd
 memory_write: model date = 1973-12-25T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-26T00:00:00 wall clock = 2026-09-21T14:55:25 avg dt =    43.46 s/day, dt =    44.51 s/day, rate =     5.32 ypd
 memory_write: model date = 1973-12-26T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-27T00:00:00 wall clock = 2026-09-21T14:56:09 avg dt =    43.46 s/day, dt =    43.83 s/day, rate =     5.40 ypd
 memory_write: model date = 1973-12-27T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-28T00:00:00 wall clock = 2026-09-21T14:56:53 avg dt =    43.46 s/day, dt =    44.70 s/day, rate =     5.30 ypd
 memory_write: model date = 1973-12-28T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-29T00:00:00 wall clock = 2026-09-21T14:57:38 avg dt =    43.47 s/day, dt =    44.27 s/day, rate =     5.35 ypd
 memory_write: model date = 1973-12-29T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-30T00:00:00 wall clock = 2026-09-21T14:58:22 avg dt =    43.47 s/day, dt =    44.76 s/day, rate =     5.29 ypd
 memory_write: model date = 1973-12-30T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
Model Date: 1973-12-31T00:00:00 wall clock = 2026-09-21T14:59:07 avg dt =    43.47 s/day, dt =    44.51 s/day, rate =     5.32 ypd
 memory_write: model date = 1973-12-31T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
 -------->(med_phases_restart_write) mediating for: 1973 12 31 23 45  0   0
(med_io_wopen) creating file access-om3.cpl.r.1974-01-01-00000.nc
Model Date: 1974-01-01T00:00:00 wall clock = 2026-09-21T15:01:03 avg dt =    43.67 s/day, dt =   115.81 s/day, rate =     2.04 ypd
 memory_write: model date = 1974-01-01T00:00:00 memory =      -0.00 MB (highwater)        948.74 MB (usage)
  SUCCESSFUL TERMINATION OF CMEPS
: ===============  # simulated years / cmp-day =        5.420  ===============

Check this on Gadi: when these logs were written, and their last lines

# WHAT THIS DOES: for each log file listed above, prints when it was last changed and its last 30
# lines, which are the lines the page shows.

# 'for' repeats the three lines below once for each file; $f is the file it is on.
# 'ls -l' lists a file with details; --time-style=long-iso writes its time as 2026-07-01 00:02.
for f in /scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/access-om3.err /scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/access-om3.out /scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/output015/log/med.log; do
    echo "== $f"                          # which file this is
    ls -l --time-style=long-iso "$f"      # when it was last changed (the date and time are in the middle)
    tail -n 30 "$f"                       # its last 30 lines
done

/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd/metadata.yaml

# ---- Automatically generated by Payu. Do not edit. ----
experiment_uuid: 38867fcd-509d-4559-b313-299de40bbff4

# ---- Automatically generated by Payu. User may edit if necessary. ----
name: MC_25km_jra_iaf+wombatlite-test5-38867fcd
contact: Andrew Kiss
email: 31054815+aekiss@users.noreply.github.com
created: '2026-09-18'
url: https://github.com/ACCESS-NRI/access-om3-configs
model: ACCESS-OM3
schema_version: 1-0-4

# ---- User feel free to update. ----
description: |-
  25km ACCESS-OM3 global configuration under interannual forcing.
  Initial conditions of temperature and salinity are from WOA23.
  Run with JRA55-do v1.6.0 interannually-varying forcing, 1 Jan 1958 - 31 Dec 2023.
  This configuration includes WOMBATlite biogeochemistry with time-varying, globally-
  averaged atmospheric CO2 forcing. Initial WOMBATlite prognostic tracer
  concentrations are:
  - no3, o2: from WOA23
  - dic, alk: from GLODAPv2 2016b
  - fe: from Huang et al. 2022 (DOI: 10.5281/zenodo.6994318)
  - remaining tracers: set to constant values
notes: |-
  ACCESS-NRI and COSIMA request that users of this model acknowledge those organisations 
  following the examples at https://www.access-nri.org.au/resources/acknowledging-us/ 
  and https://cosima.org.au/index.php/get-involved/
keywords:
- JRA55
- access-om3
- interannual
- global
- bgc
realm:
- ocean
- seaIce
- ocnBgchem
nominal_resolution: 25 km
license: CC-BY-4.0
version: 0.0
long_description: 5th test - see 
  https://github.com/ACCESS-NRI/access-om3-configs/issues/1534
# frequency: The frequency(/ies) included in the experiment (array of strings)
# variable: The variable(s) included in the experiment (array of strings)
# reference: Citation or reference information (string)
# parent_experiment: experiment_uuid for parent experiment if appropriate (string)
# related_experiments: experiment_uuids for any related experiment(s) (array of strings)
# parent_experiment_branch_time: The model time corresponding to the restart path used to create this branch, e.g., 1951-02-01T00:00:00 (string)
# parent_experiment_branch_commit: Git commit hash of the source branch when this new branch was created (string)

Truncation locations for all completed run segments

⚠️ MOM6 truncated velocities 10 times in 1 of 16 output folders (the Truncs column of each outputNNN/ocean.stats). MOM6 truncates a velocity when its CFL number exceeds CFL_TRUNCATE (0.5 by default), and stops the run if more than MAXTRUNC truncations happen between energy saves (MOM6's default is 0, so any truncation stops it; the ACCESS-OM3 configurations set 10,000). See MOM6 velocity truncations for how to read the records, and Examine_truncation_data.ipynb for mapping them.

Location records are the detailed records MOM6 also writes, one per truncation, to U_velocity_truncations (zonal velocity) and V_velocity_truncations (meridional velocity); the location table below is built from them. MOM6 writes at most 50 per PE per run, so there are usually fewer of them than truncations; – means no file of that kind.

Output folder	Model period	Truncations	Max CFL	Location records (U / V)	PEs at the record cap
output000	? → 1959-01	10	0.33	1 / 9	–

The other 15 output folders had no truncations.

Where MOM6 truncates (1° boxes, most records first; from the U_velocity_truncations and V_velocity_truncations files):

Location	Records	Grid cells	Seen in output	Velocity	Layers	Max speed (m/s)	Max CFL	Surface height (m)	Depth (m)
65.2°W, 61.3°N	9	5	000	U, V	1–1	32.6	2.22	0.04	153
65.6°W, 61.4°N	1	1	000	V	1–1	8.1	0.55	0.02	243

* MOM6 writes at most 50 records per PE per run (MAX_TRUNC_FILE_SIZE_PER_PE), and this box's PE hit that cap in at least one run, so it truncated more often than the record count shows. Surface height is the top-interface height e, shown only for records that include layer 1; depth is the shallower of the two cells (D).

Check this on Gadi: the truncation counts, locations and settings

# WHAT THIS DOES: re-checks the 'Truncation locations' section from the files MOM6 wrote. A
# truncation is when MOM6 had to cut a velocity that was too fast (its CFL number was above 0.5): a
# sign of numerical trouble. The steps below are separate, so you can paste just the one you need.
# Each prints something you can compare with the page.
A=/scratch/x77/aek156/access-om3/archive/MC_25km_jra_iaf+wombatlite-test5-38867fcd

# Step 1: for each output folder: how many truncations MOM6 reported, the largest CFL number, and how many location records were written
for o in "$A"/output[0-9]*; do
    # ocean.stats is a table with one line per time step. Its 3rd column ('Truncs') is the truncations at that
    # step. awk adds that column up. (-F, means the columns are separated by commas. The header line is skipped
    # because its first column is not a number.)
    total=$(awk -F, '$1+0 == $1 {sum += $3} END {print sum+0}' "$o/ocean.stats" 2>/dev/null)
    # The 5th column holds two numbers; the second one is the CFL number. awk keeps the largest.
    cfl=$(awk -F, '$1+0 == $1 {split($5, a, " "); if (a[2] > max) max = a[2]} END {if (max != "") printf "%.2f", max}' "$o/ocean.stats" 2>/dev/null)
    # Every truncation also gets a 'location record' in the U and V files. grep -c counts the matching lines.
    u=$(grep -c 'velocity violation' "$o/U_velocity_truncations" 2>/dev/null)
    v=$(grep -c 'velocity violation' "$o/V_velocity_truncations" 2>/dev/null)
    # ${x:--} means: use x, or a dash if x is empty (no such file)
    echo "$(basename "$o"): truncations ${total:--}, max CFL ${cfl:--}, location records U/V ${u:--}/${v:--}"
done

# Step 2: where it happens. Each record contains the place, like '( 73.27 E 67.89 N)'. This counts the records in
# boxes one degree wide (negative longitude means west), busiest first. The pipeline does this:
#   grep: keep the lines that start a record    sed: keep only the longitude and latitude
#   awk: round them to whole degrees            sort | uniq -c: count the records in each box
#   sort -rn | head: busiest boxes first
grep -h 'velocity violation' "$A"/output[0-9]*/[UV]_velocity_truncations 2>/dev/null \
  | sed -E 's/.*\( *(-?[0-9.]+) E +(-?[0-9.]+) N\).*/\1 \2/' \
  | awk '{printf "%4.0fE %4.0fN\n", $1, $2}' \
  | sort | uniq -c | sort -rn | head -n 15

# Step 3: MOM6 writes at most 50 records per PE (one of the parallel processes) in one run, then stops. A PE with
# 50 records may have truncated more often than the records show. This lists the PEs that reached 50.
for f in "$A"/output[0-9]*/[UV]_velocity_truncations; do
    [ -f "$f" ] || continue                 # skip if there is no such file
    # each record says which PE it came from ('violation at 6073:'); count the records per PE, keep PEs with 50
    busy=$(grep -h 'velocity violation' "$f" | sed -E 's/.*violation at ([0-9]+):.*/PE\1/' | sort | uniq -c | awk '$1 >= 50 {printf "%s (%d records) ", $2, $1}')
    echo "$(basename "$(dirname "$f")")/$(basename "$f"): $busy"
done

# Step 4: one record in full. The page reads the speed (u, v), the CFL number, the surface height (e) and the
# depth (D) from lines like these.
head -n 30 "$A"/output000/U_velocity_truncations

# Step 5: the model period of each output folder, from payu's run records (output NNN is run NNN)
python3 - "$A" <<'EOF'
import glob, json, os, sys

# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1]                  # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
    try:
        with open(path) as f:
            record = json.load(f)
        record["file"] = path          # remember which file it came from
        runs.append(record)
    except (OSError, ValueError):
        pass                           # skip a file we may not read, or that is not valid JSON

# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))

for r in runs:
    run, finish = r.get("payu_current_run"), r.get("model_finish_time")
    if str(run).isdigit() and finish:      # only runs that finished have a model period
        start = r.get("model_start_time") or "?"      # payu did not record the start of the first run
        print("output%03d  %s -> %s" % (int(run), start[:7], finish[:7]))
EOF

Settings changed during this run

This comes from the git history of the payu control directory, and this experiment has no run_dir in experiments.yml. Add one to see here every edit made to the model's settings and build during the run, with the old and new values.