MC_100km_jra_iaf+wombatlite-test1-e1de2a94
✅ Latest submission: Run 65 completed successfully (2026-10-03 21:06 AEST). Model date reached: 2024-01-01 (66.0 model years complete).
Notebooks: ocean_stats, timeseries, Timeseries_daily_extreme_from_2D_fields, temp-salt-vs-depth-time
GitHub issue: ACCESS-NRI/access-om3-configs#1669
📋 Summary for the GitHub issue
### OM3 Monitor summary: `MC_100km_jra_iaf+wombatlite-test1-e1de2a94`
- **Latest submission:** ✅ Run 65 completed successfully (2026-10-03 21:06 AEST).
- **Model date reached:** 2024-01-01 (66.0 model years complete)
- **Average throughput:** 21.92 SYPD
- **Truncations:** 1,216 in 2 of 66 output folders
| Where | Records | Output folders | Max speed (m/s) | Max CFL |
|---|---:|---|---:|---:|
| [108.9°W, 72.9°N](https://www.google.com/maps/place/72.91,-108.87/@72.91,-108.87,5z) | 24 | 042 | 13.8 | 1.42 |
| [108.1°W, 72.4°N](https://www.google.com/maps/place/72.38,-108.12/@72.38,-108.12,5z) | 24 | 042 | 13.5 | 1.38 |
| [96.9°W, 75.9°N](https://www.google.com/maps/place/75.88,-96.86/@75.88,-96.86,5z) | 17 | 036, 042 | 15.9 | 1.81 |
| [62.4°W, 82.0°N](https://www.google.com/maps/place/82.01,-62.41/@82.01,-62.41,5z) | 10 | 042 | 6.6 | 0.70 |
| [97.1°W, 75.3°N](https://www.google.com/maps/place/75.31,-97.10/@75.31,-97.10,5z) | 9 | 036, 042 | 13.4 | 1.50 |
**References**
1. Archive folder: `/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94`
2. Status page: https://chrisb13.github.io/access-om3-live/live/MC_100km_jra_iaf%2Bwombatlite-test1-e1de2a94/
*Generated automatically by the [OM3 Monitor](https://chrisb13.github.io/access-om3-live/live/) on 2026-10-19 16:38 AEDT; it may contain errors, so check against the files above.*
Archive folder: completed runs
Read from the payu archive folder /g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94.
- Output folders: 66 (latest
output065) - Model date reached: 2024-01-01
- Runs recorded by payu: 67, of which failed: 0
Check this on Gadi: these counts
# WHAT THIS DOES: re-counts the 'Archive folder' list: how many output folders there are, which is
# the latest, how many runs payu recorded, how many of those failed, and how far the model got. The
# four lines it prints should match the page.
A=/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94
# Step 1: count the output folders (output000, output001, ...) and name the last one
ls -d "$A"/output[0-9]* | wc -l
basename "$(ls -d "$A"/output[0-9]* | tail -n 1)"
# Step 2: read payu's run records with a few lines of python, and count them
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
# a run failed if payu's exit status for it is not 0 (a run with no status yet is still going)
failed = [r for r in runs if r.get("payu_run_status") not in (0, None)]
print(len(runs), "runs recorded by payu,", len(failed), "failed")
# the model date reached is the finish date of the last run that has one
finished = [r for r in runs if r.get("model_finish_time")]
print("model date reached:", finished[-1]["model_finish_time"][:10] if finished else "unknown")
EOF
Model throughput
Simulated years per day, made by plot_sypd.py from the outputNNN/log/med.log files of the completed submissions in /g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94.
Check this on Gadi: the rate in each output's med.log
# WHAT THIS DOES: for each output folder, finds the model's speed in its med.log (the mediator's
# log) and averages it. The speed, called the 'rate', is in simulated years per day (ypd). The
# figure plots these rates. You get one line per output folder: how many rate lines it had, and
# their average.
A=/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94
for o in "$A"/output[0-9]*; do
[ -f "$o/log/med.log" ] || continue # skip folders that have no med.log
# A line looks like: Model Date: 1958-09-04T00:00:00 ... rate = 7.09 ypd. awk reads the number just before
# 'ypd' (the second-last word), adds them up and counts them, then prints the average at the end.
awk -v name="$(basename "$o")" '/Model Date:/ && /ypd/ {count++; total += $(NF-1)} END {if (count) printf "%s: %d rate lines, mean %.2f ypd\n", name, count, total/count}' "$o/log/med.log"
done
Recent runs
From payu's job records in /g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/payu_jobs; times are Canberra time.
| Run | PBS job | Result | Model period | Wall time (h) | SYPD | Record last modified |
|---|---|---|---|---|---|---|
| 65 | 180395779.gadi-pbs | ✅ | 2023-01-01 → 2024-01-01 | 1.1 | 22.00 | 2026-10-03 21:06 AEST |
| 64 | 180364870.gadi-pbs | ✅ | 2022-01-01 → 2023-01-01 | 1.1 | 21.85 | 2026-10-03 09:49 AEST |
| 63 | 180362780.gadi-pbs | ✅ | 2021-01-01 → 2022-01-01 | 1.1 | 21.53 | 2026-10-03 08:33 AEST |
| 62 | 180361626.gadi-pbs | ✅ | 2020-01-01 → 2021-01-01 | 1.1 | 22.00 | 2026-10-03 07:18 AEST |
| 61 | 180360310.gadi-pbs | ✅ | 2019-01-01 → 2020-01-01 | 1.1 | 21.74 | 2026-10-03 06:05 AEST |
| 60 | 180358665.gadi-pbs | ✅ | 2018-01-01 → 2019-01-01 | 1.1 | 2026-10-03 04:47 AEST | |
| 59 | 180356325.gadi-pbs | ✅ | 2017-01-01 → 2018-01-01 | 1.1 | 22.10 | 2026-10-03 03:33 AEST |
| 58 | 180353427.gadi-pbs | ✅ | 2016-01-01 → 2017-01-01 | 1.1 | 21.65 | 2026-10-03 02:16 AEST |
| 57 | 180351342.gadi-pbs | ✅ | 2015-01-01 → 2016-01-01 | 1.1 | 21.75 | 2026-10-03 00:58 AEST |
| 56 | 180348038.gadi-pbs | ✅ | 2014-01-01 → 2015-01-01 | 1.1 | 21.92 | 2026-10-02 23:44 AEST |
Check this on Gadi: this table
# WHAT THIS DOES: prints the 'Recent runs' table: the 10 newest runs, newest first. 'ok' means
# payu's exit status was 0. SYPD is simulated years per day: the model time the run covered, divided
# by the wall-clock time it took. (If payu did not record a run's start date this shows ?; the page
# may fill it in from the previous run.)
A=/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
import time
from datetime import datetime
print("Run PBS job Result Model period Hours SYPD Record last modified")
for r in reversed(runs[-10:]): # the last 10 runs, newest first
status = r.get("payu_run_status")
result = "ok" if status == 0 else ("failed" if status is not None else str(r.get("stage")))
start, finish = r.get("model_start_time"), r.get("model_finish_time")
seconds = r.get("timings", {}).get("payu_total_duration_seconds") # wall-clock seconds the run took
hours = "%.1f" % (seconds / 3600) if seconds else ""
sypd = ""
if start and finish and seconds:
# the model time the run covered, in whole days (dates look like 1958-01-01T00:00:00)
days = (datetime.strptime(finish[:19], "%Y-%m-%dT%H:%M:%S") - datetime.strptime(start[:19], "%Y-%m-%dT%H:%M:%S")).days
sypd = "%.2f" % (days / 365.25 / (seconds / 86400)) # model years divided by wall-clock days
changed = time.strftime("%Y-%m-%d %H:%M %Z", time.localtime(os.path.getmtime(r["file"])))
print("%-4s %-18s %-10s %-10s -> %-10s %6s %6s %s" % (r.get("payu_current_run"), r.get("scheduler_job_id"), result,
(start or "?")[:10], (finish or "?")[:10], hours, sypd, changed))
EOF
Latest logs in the archive folder
/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/output065/access-om3.err (last lines; modified 2026-10-03 21:05 AEST)
(t_initf) Read in prof_inparm namelist from: drv_in
(t_initf) Using profile_disable= F
(t_initf) profile_timer= 4
(t_initf) profile_depth_limit= 4
(t_initf) profile_detail_limit= 2
(t_initf) profile_barrier= F
(t_initf) profile_outpe_num= 1
(t_initf) profile_outpe_stride= 0
(t_initf) profile_single_file= F
(t_initf) profile_global_stats= T
(t_initf) profile_ovhd_measurement= F
(t_initf) profile_add_detail= F
(t_initf) profile_papi_enable= F
WARNING from PE 0: set_diffusivity_init: SIMPLE_TKE_TO_KD can not be used reliably with USE_REGRIDDING.
WARNING from PE 0: diag_manager_mod::closing_file: module/output_field ocean_model/total_hfevapds, skip one time level, maybe send_data never called
WARNING from PE 0: diag_manager_mod::closing_file: module/output_field ocean_model/total_heat_content_massin, skip one time level, maybe send_data never called
Memuse(MB) at Memory HiWaterMark= 4.564E+02 8.491E+03 3.597E+02 4.906E+02
/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/output065/access-om3.out (last lines; modified 2026-10-03 21:05 AEST)
(Ocean BT global synch) 8760 10.053530 177.997733 59.898857 30.737459 0.016 41 0 495
(Ocean Coriolis & mom advection) 17520 4.950229 8.866551 6.367064 0.726888 0.002 31 0 495
(Ocean continuity equation) 26280 38.369356 96.699875 68.947827 15.376553 0.018 31 0 495
(Ocean pressure force) 8760 103.328302 147.027789 128.814422 10.329897 0.034 31 0 495
(Ocean vertical viscosity) 43800 21.983983 120.536488 83.103422 29.514689 0.022 31 0 495
(Ocean horizontal viscosity) 8760 14.751858 37.314638 28.377657 5.676531 0.007 31 0 495
(Ocean momentum increments) 17520 2.635587 6.518128 4.548819 0.765894 0.001 31 0 495
(Ocean message passing) 122640 25.057144 267.962686 117.735520 51.894187 0.031 31 0 495
(Ocean barotropic mode calc) 17520 0.145385 0.301328 0.216602 0.027264 0.000 31 0 495
(Ocean barotropic mode stepping) 17520 131.730720 366.240797 197.562237 42.661338 0.052 31 0 495
(Ocean barotropic forcing calc) 17520 2.587689 6.590680 3.932876 0.690386 0.001 31 0 495
(Ocean kappa_shear) 4380 8.540190 303.352370 98.481027 56.277480 0.026 31 0 495
(Ocean diabatic entrain) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 495
(Ocean vert remap) 4380 14.537574 61.017440 40.989398 11.968789 0.011 31 0 495
(Ocean set_diffusivity) 8760 41.068896 340.083229 136.784461 57.834671 0.036 31 0 495
(Ocean KPP) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 495
(Ocean tracer_columns) 4380 898.525692 1328.048393 1158.203138 70.595923 0.305 26 0 495
(Ocean diabatic tridiag) 4380 0.828275 1.281302 0.968505 0.074917 0.000 41 0 495
(Ocean diabatic message passing) 8760 1.686879 443.777457 142.624226 95.651966 0.038 41 0 495
(Ocean differential diffusion) 0 0.000000 0.000000 0.000000 0.000000 0.000 41 0 495
(Ocean find_uv_at_h) 8760 1.585654 4.679681 2.891291 0.836683 0.001 41 0 495
(Ocean frazil) 8760 0.409560 2.302511 1.021465 0.382243 0.000 41 0 495
(Ocean advect tracer) 4380 151.651224 181.179280 159.293084 4.909674 0.042 31 0 495
(Ocean tracer halo updates) 21909 56.569940 151.524822 114.130663 18.127296 0.030 41 0 495
(Ocean tracer global synch) 13149 2.860594 118.985279 42.355412 21.983045 0.011 41 0 495
(Ocean diffuse tracer) 4380 29.489048 263.341050 176.845642 71.854799 0.047 31 0 495
(Ocean epipycnal diffuse tracer) 0 0.000000 0.000000 0.000000 0.000000 0.000 31 0 495
Ocean surface forcing 17521 33.115755 89.929574 65.266536 11.168953 0.017 11 0 495
(Ocean forcing diagnostics) 13140 32.422083 34.429360 34.324256 0.087758 0.009 41 0 495
MPP_STACK high water mark= 0
/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/output065/log/med.log (last lines; modified 2026-10-03 21:05 AEST)
Model Date: 2023-12-20T00:00:00 wall clock = 2026-10-03T21:02:38 avg dt = 10.17 s/day, dt = 9.05 s/day, rate = 26.16 ypd
memory_write: model date = 2023-12-20T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-21T00:00:00 wall clock = 2026-10-03T21:02:47 avg dt = 10.17 s/day, dt = 9.13 s/day, rate = 25.94 ypd
memory_write: model date = 2023-12-21T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-22T00:00:00 wall clock = 2026-10-03T21:02:56 avg dt = 10.16 s/day, dt = 9.44 s/day, rate = 25.07 ypd
memory_write: model date = 2023-12-22T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-23T00:00:00 wall clock = 2026-10-03T21:03:05 avg dt = 10.16 s/day, dt = 8.90 s/day, rate = 26.58 ypd
memory_write: model date = 2023-12-23T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-24T00:00:00 wall clock = 2026-10-03T21:03:14 avg dt = 10.16 s/day, dt = 9.00 s/day, rate = 26.29 ypd
memory_write: model date = 2023-12-24T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-25T00:00:00 wall clock = 2026-10-03T21:03:23 avg dt = 10.15 s/day, dt = 8.91 s/day, rate = 26.57 ypd
memory_write: model date = 2023-12-25T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-26T00:00:00 wall clock = 2026-10-03T21:03:32 avg dt = 10.15 s/day, dt = 8.99 s/day, rate = 26.34 ypd
memory_write: model date = 2023-12-26T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-27T00:00:00 wall clock = 2026-10-03T21:03:41 avg dt = 10.15 s/day, dt = 9.03 s/day, rate = 26.22 ypd
memory_write: model date = 2023-12-27T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-28T00:00:00 wall clock = 2026-10-03T21:03:50 avg dt = 10.14 s/day, dt = 8.81 s/day, rate = 26.88 ypd
memory_write: model date = 2023-12-28T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-29T00:00:00 wall clock = 2026-10-03T21:03:59 avg dt = 10.14 s/day, dt = 8.88 s/day, rate = 26.66 ypd
memory_write: model date = 2023-12-29T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-30T00:00:00 wall clock = 2026-10-03T21:04:08 avg dt = 10.14 s/day, dt = 8.98 s/day, rate = 26.36 ypd
memory_write: model date = 2023-12-30T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
Model Date: 2023-12-31T00:00:00 wall clock = 2026-10-03T21:04:17 avg dt = 10.13 s/day, dt = 9.02 s/day, rate = 26.23 ypd
memory_write: model date = 2023-12-31T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
-------->(med_phases_restart_write) mediating for: 2023 12 31 23 0 0 0
(med_io_wopen) creating file access-om3.cpl.r.2024-01-01-00000.nc
Model Date: 2024-01-01T00:00:00 wall clock = 2026-10-03T21:05:02 avg dt = 10.23 s/day, dt = 45.28 s/day, rate = 5.23 ypd
memory_write: model date = 2024-01-01T00:00:00 memory = -0.00 MB (highwater) 776.81 MB (usage)
SUCCESSFUL TERMINATION OF CMEPS
: =============== # simulated years / cmp-day = 23.140 ===============
Check this on Gadi: when these logs were written, and their last lines
# WHAT THIS DOES: for each log file listed above, prints when it was last changed and its last 30
# lines, which are the lines the page shows.
# 'for' repeats the three lines below once for each file; $f is the file it is on.
# 'ls -l' lists a file with details; --time-style=long-iso writes its time as 2026-07-01 00:02.
for f in /g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/output065/access-om3.err /g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/output065/access-om3.out /g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/output065/log/med.log; do
echo "== $f" # which file this is
ls -l --time-style=long-iso "$f" # when it was last changed (the date and time are in the middle)
tail -n 30 "$f" # its last 30 lines
done
/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94/metadata.yaml
# ---- Automatically generated by Payu. Do not edit. ----
experiment_uuid: e1de2a94-f745-4192-b7c3-5c7606daf4ff
# ---- Automatically generated by Payu. User may edit if necessary. ----
name: MC_100km_jra_iaf+wombatlite-test1-e1de2a94
contact: minghangli-uni
email: 24727729+minghangli-uni@users.noreply.github.com
created: '2026-09-28'
url: https://github.com/ACCESS-NRI/access-om3-configs.git
model: ACCESS-OM3
Truncation locations for all completed run segments
⚠️ MOM6 truncated velocities 1,216 times in 2 of 66 output folders (the Truncs column of each outputNNN/ocean.stats). MOM6 truncates a velocity when its CFL number exceeds CFL_TRUNCATE (0.5 by default), and stops the run if more than MAXTRUNC truncations happen between energy saves (MOM6's default is 0, so any truncation stops it; the ACCESS-OM3 configurations set 10,000). See MOM6 velocity truncations for how to read the records, and Examine_truncation_data.ipynb for mapping them.
Location records are the detailed records MOM6 also writes, one per truncation, to U_velocity_truncations (zonal velocity) and V_velocity_truncations (meridional velocity); the location table below is built from them. MOM6 writes at most 50 per PE per run, so there are usually fewer of them than truncations; – means no file of that kind.
| Output folder | Model period | Truncations | Max CFL | Location records (U / V) | PEs at the record cap |
|---|---|---|---|---|---|
| output036 | 1994-01 → 1995-01 | 6 | 0.18 | 6 / – | – |
| output042 | 2000-01 → 2001-01 | 1,210 | 0.49 | 159 / 17 | – |
The other 64 output folders had no truncations.
Where MOM6 truncates (1° boxes, most records first; from the U_velocity_truncations and V_velocity_truncations files):
| Location | Records | Grid cells | Seen in output | Velocity | Layers | Max speed (m/s) | Max CFL | Surface height (m) | Depth (m) |
|---|---|---|---|---|---|---|---|---|---|
| 108.9°W, 72.9°N | 24 | 2 | 042 | U | 1–12 | 13.8 | 1.42 | 5.90 | 118 |
| 108.1°W, 72.4°N | 24 | 2 | 042 | U | 1–9 | 13.5 | 1.38 | 12.62 | 47 |
| 96.9°W, 75.9°N | 17 | 3 | 036, 042 | U | 1–11 | 15.9 | 1.81 | -1.07 | 55 |
| 62.4°W, 82.0°N | 10 | 1 | 042 | U | 1–2 | 6.6 | 0.70 | -1.59 | 499 |
| 97.1°W, 75.3°N | 9 | 2 | 036, 042 | U | 1–11 | 13.4 | 1.50 | -0.93 | 103 |
| 61.2°W, 82.2°N | 9 | 1 | 042 | U | 1–2 | 5.1 | 0.68 | 2.56 | 272 |
| 63.8°W, 81.2°N | 8 | 1 | 042 | U | 1–3 | 5.8 | 0.76 | -1.46 | 329 |
| 101.0°W, 69.5°N | 7 | 2 | 042 | V | 1–3 | 2.9 | 0.64 | -1.35 | 17 |
| 65.9°W, 80.8°N | 6 | 1 | 042 | U | 1–2 | 5.9 | 0.77 | -1.04 | 272 |
| 64.8°W, 81.0°N | 5 | 1 | 042 | U | 1–2 | 5.8 | 0.76 | -1.36 | 300 |
| 66.9°W, 80.6°N | 5 | 1 | 042 | U | 1–2 | 5.5 | 0.71 | -0.73 | 272 |
| 22.2°W, 82.0°N | 4 | 1 | 042 | U | 1–5 | 6.3 | 1.63 | -1.05 | 48 |
| 67.9°W, 80.4°N | 4 | 1 | 042 | U | 1–1 | 4.7 | 0.61 | -0.96 | 271 |
| 104.4°W, 75.8°N | 4 | 2 | 042 | U | 1–6 | 9.2 | 1.04 | -1.70 | 106 |
| 23.9°W, 81.9°N | 3 | 1 | 042 | U | 1–4 | 6.1 | 0.81 | -2.05 | 48 |
| 96.6°W, 76.5°N | 3 | 1 | 042 | U | 1–3 | 25.1 | 2.88 | -0.48 | 137 |
| 97.8°W, 76.9°N | 3 | 2 | 042 | U | 1–2 | 5.5 | 0.65 | -0.51 | 161 |
| 104.7°W, 76.3°N | 3 | 2 | 042 | U | 1–7 | 8.6 | 0.99 | -1.14 | 121 |
| 89.6°W, 76.0°N | 3 | 1 | 042 | V | 1–8 | 9.6 | 0.99 | 9.98 | 23 |
| 38.6°W, 83.7°N | 2 | 1 | 042 | U | 1–1 | 4.5 | 0.62 | -0.58 | 122 |
* MOM6 writes at most 50 records per PE per run (MAX_TRUNC_FILE_SIZE_PER_PE), and this box's PE hit that cap in at least one run, so it truncated more often than the record count shows. Surface height is the top-interface height e, shown only for records that include layer 1; depth is the shallower of the two cells (D).
Check this on Gadi: the truncation counts, locations and settings
# WHAT THIS DOES: re-checks the 'Truncation locations' section from the files MOM6 wrote. A
# truncation is when MOM6 had to cut a velocity that was too fast (its CFL number was above 0.5): a
# sign of numerical trouble. The steps below are separate, so you can paste just the one you need.
# Each prints something you can compare with the page.
A=/g/data/ol01/outputs/access-om3-100km/MC_100km_jra_iaf+wombatlite-test1-e1de2a94
# Step 1: for each output folder: how many truncations MOM6 reported, the largest CFL number, and how many location records were written
for o in "$A"/output[0-9]*; do
# ocean.stats is a table with one line per time step. Its 3rd column ('Truncs') is the truncations at that
# step. awk adds that column up. (-F, means the columns are separated by commas. The header line is skipped
# because its first column is not a number.)
total=$(awk -F, '$1+0 == $1 {sum += $3} END {print sum+0}' "$o/ocean.stats" 2>/dev/null)
# The 5th column holds two numbers; the second one is the CFL number. awk keeps the largest.
cfl=$(awk -F, '$1+0 == $1 {split($5, a, " "); if (a[2] > max) max = a[2]} END {if (max != "") printf "%.2f", max}' "$o/ocean.stats" 2>/dev/null)
# Every truncation also gets a 'location record' in the U and V files. grep -c counts the matching lines.
u=$(grep -c 'velocity violation' "$o/U_velocity_truncations" 2>/dev/null)
v=$(grep -c 'velocity violation' "$o/V_velocity_truncations" 2>/dev/null)
# ${x:--} means: use x, or a dash if x is empty (no such file)
echo "$(basename "$o"): truncations ${total:--}, max CFL ${cfl:--}, location records U/V ${u:--}/${v:--}"
done
# Step 2: where it happens. Each record contains the place, like '( 73.27 E 67.89 N)'. This counts the records in
# boxes one degree wide (negative longitude means west), busiest first. The pipeline does this:
# grep: keep the lines that start a record sed: keep only the longitude and latitude
# awk: round them to whole degrees sort | uniq -c: count the records in each box
# sort -rn | head: busiest boxes first
grep -h 'velocity violation' "$A"/output[0-9]*/[UV]_velocity_truncations 2>/dev/null \
| sed -E 's/.*\( *(-?[0-9.]+) E +(-?[0-9.]+) N\).*/\1 \2/' \
| awk '{printf "%4.0fE %4.0fN\n", $1, $2}' \
| sort | uniq -c | sort -rn | head -n 15
# Step 3: MOM6 writes at most 50 records per PE (one of the parallel processes) in one run, then stops. A PE with
# 50 records may have truncated more often than the records show. This lists the PEs that reached 50.
for f in "$A"/output[0-9]*/[UV]_velocity_truncations; do
[ -f "$f" ] || continue # skip if there is no such file
# each record says which PE it came from ('violation at 6073:'); count the records per PE, keep PEs with 50
busy=$(grep -h 'velocity violation' "$f" | sed -E 's/.*violation at ([0-9]+):.*/PE\1/' | sort | uniq -c | awk '$1 >= 50 {printf "%s (%d records) ", $2, $1}')
echo "$(basename "$(dirname "$f")")/$(basename "$f"): $busy"
done
# Step 4: one record in full. The page reads the speed (u, v), the CFL number, the surface height (e) and the
# depth (D) from lines like these.
head -n 30 "$A"/output036/U_velocity_truncations
# Step 5: the model period of each output folder, from payu's run records (output NNN is run NNN)
python3 - "$A" <<'EOF'
import glob, json, os, sys
# payu keeps one small file for every run: payu_jobs/<number>/run/<job>.json. Read them all.
archive = sys.argv[1] # the archive folder, handed over by the shell line above
runs = []
for path in glob.glob(archive + "/payu_jobs/*/run/*.json"):
try:
with open(path) as f:
record = json.load(f)
record["file"] = path # remember which file it came from
runs.append(record)
except (OSError, ValueError):
pass # skip a file we may not read, or that is not valid JSON
# Put the runs in order: by run number, and by start time if a run was tried more than once.
runs.sort(key=lambda r: (int(r.get("payu_current_run") or 0), r.get("timings", {}).get("payu_start_time") or ""))
for r in runs:
run, finish = r.get("payu_current_run"), r.get("model_finish_time")
if str(run).isdigit() and finish: # only runs that finished have a model period
start = r.get("model_start_time") or "?" # payu did not record the start of the first run
print("output%03d %s -> %s" % (int(run), start[:7], finish[:7]))
EOF
Settings changed during this run
This comes from the git history of the payu control directory, and this experiment has no run_dir in experiments.yml. Add one to see here every edit made to the model's settings and build during the run, with the old and new values.