OM3 Monitor

Follows ACCESS-OM3 runs while they are running: each night it re-runs a few evaluation notebooks on each run's latest output and builds a status page from its payu records and logs. Pages are replaced nightly; nothing is kept.

Experiments are listed by their latest activity with newest first. An experiment's notebooks are only re-run when it has new output. Click an experiment for its run status. Times are Canberra time.

Written with Claude: please check

OM3 Monitor pages and the code that builds them were written mostly by Claude with feedback from @chrisb13. Each status page has copyable checks that one can run on Gadi. Please report anything wrong or unclear: see Contributing and feedback.

Experiment	Description	Latest submission (model date reached) last payu record ▾	Notebooks last run	Notebooks shown	Failed
MC_25km_jra_iaf+wombatlite-test5.1-8f50b769	25km JRA IAF test "5.1" – GitHub issue	🏃 1973-01-01 (15 years complete, avg 6.7 SYPD) 2026-10-19 13:29 AEDT	2026-10-19 16:19 AEDT	ocean_stats timeseries Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	–
MC_8km_jra_ryf-test2-expt-c349f800	8km jra_ryf-test2 – GitHub issue	🏃 1904-05-01 (4.4 years complete, avg 1.7 SYPD) 2026-10-19 11:58 AEDT	2026-10-19 10:39 AEDT ⏱ last job didn't finish	ocean_stats Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	timeseries
test1-MCW_100km_era_iaf-908e8682	MCW beta long-run	❌ 1980-01-01 (22 years complete, avg 10.3 SYPD) 2026-10-16 01:50 AEDT	2026-10-19 16:12 AEDT	timeseries Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	ocean_stats
MC_100km_jra_iaf+wombatlite-test1-e1de2a94	GitHub issue	✅ 2024-01-01 (66 years complete, avg 21.9 SYPD) 2026-10-03 21:06 AEST	2026-10-19 16:12 AEDT	ocean_stats timeseries Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	–
MC_8km_jra_ryf-test1-expt-4f2841fe	8km jra_ryf-test1	❌ 1902-06-01 (2.3 years complete, avg 0.7 SYPD) 2026-10-02 19:27 AEST	2026-10-19 16:29 AEDT	Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	ocean_stats timeseries
exp2.0_run2-expt-3d030595	Helens regional superoz	✅ 2014-08-01 (1.6 years complete, avg 0.6 SYPD) 2026-09-29 10:19 AEST	2026-10-19 16:02 AEDT	ocean_stats	timeseries Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time
MC_25km_jra_iaf+wombatlite-test5-38867fcd	25km JRA IAF test 5 (stopped)	❌ 1974-01-01 (16 years complete, avg 5.4 SYPD) 2026-09-21 19:25 AEST	2026-10-19 16:19 AEDT	ocean_stats timeseries Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	–
MC_25km_era_iaf+wombatlite-test5-era-8a51ed1d	25km ERA IAF test 5 (stopped)	⚠️ 1970-01-01 (12 years complete, avg 5.0 SYPD) 2026-09-21 18:50 AEST	2026-10-19 16:18 AEDT	ocean_stats timeseries Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	–
cm3-run-27-07-2026-PI-control	CM3 pre-industrial control run. This run uses the GC5-central UM configuration, CMIP6 inputs, and includes several polar and tripolar bugfixes.	– (datastore given directly; 50 model years, 1981–2030) 2026-08-19 11:49 AEST	2026-10-19 15:55 AEDT	Timeseries_daily_extreme_from_2D_fields	ocean_stats timeseries temp-salt-vs-depth-time
cm3-run-27-07-2026-PD-control	CM3 present day control run. This run uses the GC5-central UM configuration, CMIP6 inputs, and includes several polar and tripolar bugfixes.	– (datastore given directly; 50 model years, 1981–2030) 2026-08-17 16:25 AEST	2026-10-19 15:55 AEDT	Timeseries_daily_extreme_from_2D_fields	ocean_stats timeseries temp-salt-vs-depth-time
MC_25km_jra_iaf+wombatlite-test4-d28e0359	25km ACCESS-OM3 global configuration under interannual forcing. (from the run's metadata.yaml)	✅ 2024-01-01 (66 years complete, avg 6.2 SYPD) 2026-07-01 00:02 AEST	2026-10-19 16:26 AEDT	ocean_stats timeseries Timeseries_daily_extreme_from_2D_fields temp-salt-vs-depth-time	–

Throughput of all runs

Simulated years per day (SYPD) of every completed run, from payu's job records, against the model year it reached: one line per experiment in the table above.

How it works

Each night, around midnight Canberra time, a GitHub Actions workflow in chrisb13/access-om3-live logs in to NCI's Gadi and checks every monitored experiment for new output. For each experiment that has changed, it submits a Gadi job that re-runs the notebooks from access-om3-paper-1; the others keep the figures from their last run. The jobs run in parallel and the workflow waits for them, then updates the site. A job still running after about five and a half hours is shown as re-running, and its results appear the next night. The table above shows when each experiment was last run, whether a job is running, and which notebooks failed.

Monitoring a new experiment

Add an entry under experiments: in experiments.yml and open a pull request. Only the archive folder is needed; the other keys are optional:

experiments:
  # minimal: just the archive folder
  - /g/data/ol01/outputs/access-om3-25km/MC_25km_jra_iaf+wombatlite-test4-d28e0359
  # with optional keys
  - archive: /g/data/ol01/outputs/access-om3-8km/MC_8km_jra_ryf-test2-expt-c349f800
    run_dir: /g/data/tm70/cyb561/access-om3-runs/8k/MC_8km_jra_ryf-test2
    notebooks: [MLD, SST]
    description: One line on what this run is testing

The terms match the sections of each experiment's status page:

• Archive folder (archive, required): the folder holding output000, output001, …: where a payu control directory's archive link points (readlink -f archive), or its synced copy under /g/data/ol01/outputs/. Its name becomes the experiment's name here, so it must be unique.
• Run directory (run_dir, optional): the payu control directory, for the current or crashed run's PBS and model logs, and the git history of its settings (shown as "Settings changed during this run", the last section of the status page).
• Notebooks (notebooks, optional): the access-om3-paper-1 notebooks to run for this experiment, instead of the standard ones (ocean_stats, timeseries, Timeseries_daily_extreme_from_2D_fields, temp-salt-vs-depth-time).
• Description (description, optional): one line shown in the OM3 Monitor table and at the top of the status page. Without it, the first line of the description in the run's metadata.yaml is used.
• GitHub issue (issue, optional): where the run is discussed, linked at the top of its status page.

A run that isn't a payu run, such as ACCESS-CM3, can be given as its intake-esm datastore (a .json path under archive). The datastore is used as it is, and only its notebooks are run.

The folders must be readable by the Gadi account that runs the nightly job. To stop monitoring an experiment, remove its entry.