Changelog#
Versions follow Semantic Versioning (<major>.<minor>.<patch>).
Backward incompatible (breaking) changes will only be introduced in major versions with advance notice in the Deprecations section of releases.
climate-ref 0.20.5 (2026-11-12)#
Bug Fixes#
- Registered the model climatologies computed by the PMP annual cycle diagnostic as outputs: the NetCDF files for the annual cycle and each season, and their summary plots. (#1053)
Trivial/Internal Changes#
climate-ref 0.20.4 (2026-11-02)#
Features#
- Added
--lockand--lock-fileoptions toref solve, also settable withREF_LOCKandREF_LOCK_FILE. A solve that cannot take the lock exits with code 75 without solving, so overlapping scheduled solves no longer clash.--lockusesref.lockin the REF configuration directory. (#1042)
Bug Fixes#
- Fixed executions recording the wrong input datasets when obs4MIPs data was filled in from obs4REF, which could list unrelated CMIP6 datasets on an execution. Releases v0.18.0 to v0.20.3 are affected when obs4REF data has been ingested, and upgrading does not correct executions that already exist.
To repair them, back up the database and run scripts/repair-execution-datasets.py with the climate-ref version that produced the executions, either before or after upgrading.
It is a dry run unless given --apply. (#1039)
climate-ref 0.20.3 (2026-10-29)#
Bug Fixes#
- Requires intake-esgf 2026.11.28, so downloaded files follow the umask instead of landing as 0600
and STAC searches by
file_startorfile_endno longer fail on fx variables. (#1041)
Improved Documentation#
- Fixed the "Using pre-computed results" notebook, which broke after the API replaced
latest_executionwith a list ofexecutions. (#1038)
climate-ref 0.20.2 (2026-10-27)#
Improvements#
- Sped up reading pages of series values, and of scalar values without outlier detection, when few values match. They now read only the values of the matching executions rather than every value of that type. (#1036)
Bug Fixes#
- Fixed the
regional-historical-trenddiagnostic reporting reference dataset trends in place of the model's trends when both were in the same output file. (#1019) - Required cartopy 0.26 in the ESMValTool environment, so the climate drivers for fire diagnostic no longer fails while plotting its maps. (#1035)
Trivial/Internal Changes#
climate-ref 0.20.1 (2026-10-24)#
Bug Fixes#
- Fixed the synthetic test database in
climate_ref.testing.syntheticrecording no source types for its execution groups, so filtering it bysource_typesmatched nothing. Each group now draws on one model era. (#1023)
climate-ref 0.20.0 (2026-10-23)#
Highlights#
- This release reworks how results are read.
Every filter and query builder now lives in a new
climate_ref.querypackage, and a field shared by several filters has the same meaning and default on each. Results can be filtered by the data that went into them, such as a model, a MIP era or specific datasets. Common reads are much faster on large databases, and therefCLI starts faster. It also adds downloads of published CMIP7 data from ESGF. The ozone, sea ice and ENSO diagnostics now follow ESMValTool v2.15, which changes their results. The provider and results APIs have breaking changes, listed below. (#999)
Breaking Changes#
- Providers now declare the fetched data they need ingested by returning
DataIngestRequests fromDiagnosticProvider.data_to_ingest(config), andref providers setupingests it. This replacesDiagnosticProvider.ingest_data(config, db)and removes thedbargument ofDiagnosticProvider.setup(). Third-party providers that overrodeingest_dataneed to implementdata_to_ingestinstead. (#972) - An empty
execution_ids,execution_group_idsorisolate_idsonMetricValueFilternow matches no values. Previously an empty list was ignored and matched every value. (#985) total_counton the results collections is now typedint | None. It isNoneonly whenwith_count=Falseis passed. (#986)- Renamed the
CMIP7Requestthat converts CMIP6 data to CMIP7 toConvertedCMIP7Requestand moved it toclimate_ref_core.esgf.cmip7_converted, along withLatestFile. The newCMIP7Requesthas nofacet_mappingorcmip7_only_facets, because it searches with CMIP7 facet names directly.CMIP7Requestnow downloads published CMIP7 data from ESGF. Removedenable_ceda_solr_indexfromclimate_ref_core.esgf.Obs4MIPsRequestnow searches the CEDA Solr index only for the duration of its own fetch. (#995) -
Reworked the
climate_ref.resultsfilters so a field means the same thing, with the same default, on every reader. -
Every filter now keeps only the promoted diagnostic version and drops groups whose newest execution is retracted. Pass
include_superseded=Trueorinclude_retracted=Trueto keep them. TheDatasetFilterslug fields follow the same rule, withinclude_retracted_executionsin place ofinclude_retracted. - Bad input now raises
ValueErrorinstead of matching nothing or being ignored. This covers unknownselectors,dimensions,OutlierPolicy.group_byand resourcesgroup_bykeys, unknownsource_types, a negativeoffsetorlimit,reference_onlyon a scalar read and a base column inDatasetFilter.facets. A string id such as"12"raisesTypeError. - An empty list now matches nothing in every list field, except
exclude_ids=[], which drops nothing. - List fields such as
diagnostic_slug,provider_slug,isolate_idsandexclude_idsare now tuples. They still take a single value. latest_successful=Truenow drops groups with no successful execution.execution_group_idsno longer drops groups that have never run.dimensions={"kind": "model"}now also matches values with nokind.- Removed
ExecutionsReader.statistics,DiagnosticsReader.stats,ExecutionStatsandselect_execution_statistics. Usereader.diagnostics.list. - Removed
latest_execution_for_groupand theget_execution_group_and_latest*helpers. Useselect_latest_executionorExecutionsReader.group_rows. - Removed
AnnotatedScalaranddetect_scalar_outliers. Passoutliers=OutlierPolicy(...)toreader.values.scalar_values. - Removed
MetricValueFilter(promoted_only=...)andExecutionGroupFilter(facets=...). Useinclude_supersededandselectors. - Renamed fields to match across views:
DiagnosticView.slugisdiagnostic_slugandtotalisexecution_count. Group counts moved toDiagnosticView.groups.ResourceProfileuses theResourceSummarynames, such assample_countandpeak_memory_bytes_p95, in Python, inref executions resourcesand in its JSON.ResourceFilter.sinceiscreated_after, andResourcesReader.profilestakes aResourceFilter.- The value builders take
filtersinstead off.
ref executions inspectnames its argumentGROUP_IDand warns rather than errors for a group that has not run.
(#999)
- Moved every filter and query builder into a new climate_ref.query package.
climate_ref.models now holds only the tables, and climate_ref.results holds the readers built on climate_ref.query.
- Removed
climate_ref.models.dataset_query. ImportDatasetFilterandselect_datasetsfromclimate_ref.query. DatasetFilterandselect_datasetsare no longer re-exported fromclimate_ref.results.datasets.DatasetFilter.execution_idis nowexecution_ids, which takes one id or a list.DatasetFilter.diagnostic_slugnow takes one slug or a list, andDatasetFiltergainsprovider_slug,diagnostic_containsandprovider_contains, like the other filters.DatasetFilter.source_typenow also takes a string such as"cmip6".
(#1011)
- Scalar outlier detection now tests each region, experiment and other label on its own,
rather than pooling everything except statistic and metric.
This stops a metric reported on different scales from being flagged and dropped as a whole.
Pass OutlierPolicy(group_by=...) to pool labels as before.
Values with kind="reference" are never flagged, and detection now runs in the database. (#1018)
Features#
- Added
select_latest_executionsand alatest_onlyoption onMetricValueFilter, so values can be limited to the latest execution in each group. (#973) - Added
InputDatasetFilterto the results reader. Execution groups, metric values and diagnostics can now be filtered by the datasets their executions ran against, such as asource_id, a MIP era or a list ofdataset_ids. Group-level reads check each group's latest execution. (#974) - Added
slug_containstoDatasetFilter, a case-insensitive substring match on the dataset slug. (#975) - Added
ResourcesReader.summaryandResourcesReader.summaries, which aggregate resource usage in SQL. (#976) - Added
latest_onlytoResourceFilter, matchingMetricValueFilter. Retracted executions are now excluded by default from resource summaries, measurements and profiles, as they already are from values.ResourcesReader.measurementstill finds a retracted execution by id. (#977) - Added
SourceDatasetType.is_model_output, which is true for source types that hold model simulation output rather than observations. (#985) - Added a
with_countoption to the paged results readers, so callers can skip the count query, and a sharedPagebase for their collections. (#986) - Added
CMIP7Requestfor downloading CMIP7 data published on ESGF. It searches the ESGF STAC indices and accepts the same facet names as data requirements. (#995) -
Every reader filter now lives in
climate_ref.query.filters, and a field shared by several filters has the same meaning and default on each: -
diagnostic_slugandprovider_slugfor exact matches, besidediagnostic_containsandprovider_contains. include_supersededanddiagnostic_version, to read one diagnostic version, promoted or not.include_retracted,created_after,created_before,execution_idsandexecution_group_ids.source_typesandor_no_model_input, to scope results by the kind of input data, such as a MIP era.
ref executions stats now also lists diagnostics with no execution groups, and no longer counts every execution.
Filter fields that pick slugs, ids, source types or failure reasons take one value or a list, so diagnostic_slug="enso_tel", source_types="cmip6" and execution_ids=42 all work.
ref executions list-groups, stats and requeue take --since, like resources, to only count executions newer than an age (30d) or a date (2026-10-01).
(#999)
- Added a SelectSingleGrid constraint that keeps one grid per variable when a model publishes it on several.
The grid is chosen by a pluggable GridPreference, which defaults to PreferNativeGrid: the native gn grid, then the finest nominal_resolution. (#1001)
- Added ExecutionGroupSourceType, a table recording which source dataset types each execution group draws on. This lets groups be filtered by their inputs in plain SQL. Existing groups are backfilled from their selectors. (#1003)
- Added climate_ref.testing.synthetic, which builds a production-shaped synthetic database for performance testing reads.
It also provides large_table_scans, which reports the large tables a set of queries reads in full. (#1009)
- Added --include-superseded and --dry-run to ref executions delete-groups.
It still deletes only groups for each diagnostic's promoted version unless --include-superseded is given, and its preview now shows each group's version and whether its latest execution is retracted. (#1015)
Improvements#
- Reduced the size of the ESMValTool conda environment. (#885)
- Added
DiagnosticsReader.with_valuesand an index on metric value type per execution, so checking which diagnostics have scalar or series values no longer reads every series value. (#971) - Added
ExecutionsReader.group_rows, which returns live execution group rows for callers that mutate them.delete-groups,requeueand reingest now use it, so they share one filter mapping withExecutionsReader.groups. (#979) - Models and results now pick the latest execution with the same SQL,
climate_ref.query.select_latest_executions. Picking each group's latest successful execution no longer rescans every execution per group on SQLite, solatest_successful=Truedropped from 35 s to under 0.1 s on a 22k-group database. (#980) - Sped up the slowest
Readerreads on large databases, including the diagnostics list, diagnostic-scoped value pages and outlier detection. Results are unchanged. (#989) - The regression mint workflow now accepts several diagnostics in one dispatch, and
ref test-cases mintbumpstest_case_versionautomatically when the committed bundle changes. (#1002) - Refreshes SQLite query planner statistics with
PRAGMA optimizeafterref solveandref datasets ingest. Without statistics SQLite can pick a poor index and read far more rows than needed. (#1007) - The
refCLI now starts faster because each subcommand is only imported when it is used, soref --helpno longer loads additional libraries. (#1013) - Sped up finding each execution group's newest execution, which the per-diagnostic group counts in the results reader do for every group. A new index on
execution (execution_group_id, created_at, id)replaces a sort of each group's executions. (#1017)
Bug Fixes#
- Updated the ozone diagnostics to use the unified ozone recipe from ESMValTool v2.15. The northern hemisphere polar cap now spans 60N to 85N, matching the upstream boundary fix. The zonal mean ozone profiles are now averaged over 1990 to 2000 rather than 2005 to 2014. CMIP7 executions now evaluate the historical experiment through 2021. (#894)
- Updated the sea ice sensitivity diagnostic for the recipe changes in ESMValTool v2.15. (#899)
- Updated the ENSO diagnostics for the renamed recipes in ESMValTool v2.15. They now compare against TROPFLUX v2 and GPCP v3.3, as the upstream recipes do. (#984)
- Fixed
Reader.artifactsresolving paths that escape the results root through a symlink. Such paths now raiseValueError, so a symlink in the results tree can no longer expose files outside it. A symlink loop also raisesValueErrorrather than an unexpected error. (#993) -
Minor correctness fixes:
-
Nonein adimensionslist now also matches rows where that dimension is empty. - An aware
created_afterorcreated_beforeis converted to UTC before comparing. - Outlier detection tests each diagnostic separately, so one diagnostic's values no longer change another's verdicts.
- The provider roll-up in
reader.resources.profileskeeps a diagnostic measured by a different memory source, so the largest diagnostic still sizes the worker. DatasetFilter(finalised=True)no longer returns an older finalised version when the newest version is not finalised.
(#999)
- Fixed the ESMValTool ENSO basic climatology and sea ice sensitivity diagnostics so they solve against published CMIP7 data, which puts each realm on its own grid.
They no longer group CMIP6 data by grid_label either, so models with their atmosphere and ocean published with different grid labels (such as IPSL, CNRM, EC-Earth and GFDL-ESM4) are now included.
The ENSO, sea ice and ozone CMIP7 test cases now use published UKESM1-3-LL data.
Fixed ref test-cases fetch pinning every request of a test case to all of its recorded datasets, which broke test cases that split one source type across requests.
Fixed reading ESMValTool series when a multi-dataset file names its variable after a reference dataset, such as sic from OSI-450 instead of siconc. (#1001)
- Fixed PMP runs that could hang forever on a Dask deadlock in xarray (pydata/xarray#11645).
The PMP provider now sets DASK_SCHEDULER=synchronous unless the environment already sets one. (#1012)
- Fixed a blank --diagnostic or --provider matching every execution group, which let delete-groups, requeue and reingest act on every group. A blank value is now rejected. (#1015)
- Fixed small bugs in the results readers and CLI.
total_counton an already pagedSelectnow counts every row.- An empty value read keeps its columns in
to_pandas(). reader.datasets.listreturns one row per dataset even when versions do not parse.ref executions flag-dirtytakes an execution group id, as it always did, and now says so.ref executions list-groupsvalidates--columnon an empty result and handles--limit 0.ref datasets list --include-files --column slugnow works.
(#1018)
Improved Documentation#
- Corrected stale class names, CLI flags and the default CMIP6 parser in the documentation, fixed note boxes that did not render, and added the example provider to the API reference. (#1005)
Trivial/Internal Changes#
climate-ref 0.19.2 (2026-10-20)#
Features#
- Added a
--skip-existingoption toref datasets ingest. Files that are already in the database are skipped without being parsed, which speeds up repeated ingests of a large archive. Files are assumed to be immutable, so run a full ingest periodically to pick up changed files. Combine it with--chunk-sizeto keep memory bounded on archives with millions of files. (#968) - Added
--failure-reasontoref executions list-groupsand a newref executions requeuecommand.requeueflags failed execution groups as dirty so the nextref solvereruns them, optionally narrowed to one kind of failure, eg.ref executions requeue --failure-reason system_error. (#969)
Improvements#
ref datasets ingest --chunk-sizenow streams obs4MIPs, obs4REF, PMP climatology and ESMValTool reference datasets instead of falling back to loading the whole catalog. (#968)
climate-ref 0.19.1 (2026-10-19)#
Bug Fixes#
- Fixed
ILAMBStandard.executepassing a glob string to ILAMB where it would have expected a regular expression. (#952) - The ECS, TCR, TCRE and ZEC diagnostics now decode
branch_time_in_parentwith the child'sparent_time_units, as the CMIP6 conventions specify, instead of the parent's own time units. (#964) - Fixed fetching CMIP6 test data with intake-esgf 2026.9 and later, which searches CMIP7 unless a project is given. ESGF requests now always name their project. (#966)
climate-ref 0.19.0 (2026-10-16)#
Breaking Changes#
ref solveandref datasets ingest --solvenow exit non-zero when any execution submitted by the solve fails while waiting for results. Scripts that ignored failed executions will now see a failing exit code. (#961)
Features#
ref solvenow health checks each provider before solving when diagnostics run on the solving host. The diagnostics of an unhealthy provider are skipped and the command exits non-zero.ref providers check-healthruns the same check on its own. (#955)- Long-running solves and ingests now log their progress every five minutes, including when stuck.
Set
REF_LOG_PROGRESS_INTERVALto change the interval in seconds, or 0 to turn it off. (#959)
Improvements#
- Failed executions now record a
failure_reason, shown byref executions list-groupsandref executions inspect. This tells an execution killed mid-run or never started apart from a diagnostic error. The partial log of an abandoned execution is now kept, so the API can serve it. (#955) - Execution warnings and errors now reach the console while a diagnostic runs, instead of only its
out.log, prefixed with the execution they came from. SetREF_LOG_JSON=trueto write console logs as JSON, one record per line, tagged with the execution. Console logs are only coloured on a terminal, and a failed conda command now logs its exit code and the end of its output. (#958) - A failed execution's log now ends with a line saying why it failed, such as a timeout or an output that could not be copied.
ref solvenow logs why execution groups were skipped and how many of its submitted executions succeeded or failed. (#961)
Bug Fixes#
- Database backups taken before migrations and by
ref db backupare now consistent while another process writes to the database. Previously a plain file copy could produce a corrupt backup. (#953) - An execution is now retried when micromamba fails to start, rather than being recorded as a diagnostic error. (#958)
- Halved the peak memory of the ILAMB
lai-avh15c1diagnostic, from around 14 GB to around 7 GB per execution. The AVHRR LAI reference is now coarsened to a true 0.5 degree grid with correct cell bounds, where before it was coarsened to ~0.35 degrees with invalid bounds. Its scores change slightly as a result. (#960) - Executions interrupted by Ctrl-C during an HPC solve, rejected on submit, or failing while their failure was recorded are now marked failed and retried, instead of being left in progress. A diagnostic that raises while being solved is now skipped rather than aborting the whole solve. (#961)
climate-ref 0.18.3 (2026-10-12)#
Improvements#
- Fetching test data for a test case that already has a catalog now resolves the datasets that catalog records, rather than re-running the facet search each request declares.
obs4MIPs and obs4REF instance_ids are now the identifiers ESGF publishes the datasets under. (#930)
Bug Fixes#
- CMIP6
instance_ids now only name the primary activity of a dataset that belongs to several, to match the identifier ESGF publishes it under. Theactivity_idmetadata still reports every activity the dataset belongs to.
Fixed CMIP7 conversion of datasets extended past the end of their CMIP6 source (extend_historical_to).
A converted CMIP7 file's tracking_id is now derived from the file being written rather
than minted at random.
ref test-cases fetch now rewrites a test case's catalog whenever its content would change. (#930)
- Existing databases are migrated to the new ESGF-style instance_ids,
so the identifier format change no longer re-registers every dataset under a new id on the next ingestion.
Execution hashes are not rewritten, so affected execution groups re-run on the next solve. (#933)
- Fixed ref solve failing with Invalid value ... for dtype float64 on CMIP6 or CMIP7 data ingested with the drs parser, when a file stores branch_time_in_parent or branch_time_in_child as a string. (#949)
- The ESMValTool cloud-radiative-effects and sea-ice-area-basic diagnostics no longer swap the model
and reference series for models whose name sorts after the observation. (#950)
climate-ref 0.18.2 (2026-09-09)#
Features#
- Added the monthly annual cycle values as series in the PMP annual cycle diagnostic.
Each region and statistic pair now produces a 12 point series indexed by month number,
with a
leveldimension for the 3D pressure-level variables. (#901)
Bug Fixes#
- Updated the PMP conda environment to
pcmdi_metrics4.2.1,enso_metrics2.0.1 and Python 3.12. The extratropical modes of variability diagnostics now run roughly 40x faster because the variability modes analysis chunks its SVD (singular value decomposition).enso_metrics2.0.1 fixes the ENSO feedback metrics on partial calendar years (CLIVAR-PRP/ENSO_metrics#98). (#901) - Fixed integration tests silently passing when test-case data was missing or stale.
ref test-cases fetch --strictnow fails if any catalog cannot be resolved, and cached CMIP7 conversions are validated and rebuilt when broken. Added a CMIP7 test case for the ozone diagnostic. (#925) - Pinned the PMP annual cycle and modes of variability CMIP6 requirements to the
Amontable. Previously monthly variables published under other tables, such asEmonorAERmonZ, were included in the same execution and the diagnostic failed to find its input files. (#926)
climate-ref 0.18.1 (2026-09-04)#
Improvements#
- Disabled micromamba's file locking for commands run inside a provider environment causing performance issues on NFS filesystems. Environment creation still takes the lock. (#922)
- Baseline diff reports now shade a NetCDF variable amber and tag it
noisewhen every difference is at most1e-9times the base ref's own magnitude, instead of shading it as a real change. This keeps the last-bit wobble of a run reproduced on different hardware from burying the rows that actually moved. (#923)
Bug Fixes#
- Fixed the noise determination for integers or an infinite value. (#924)
climate-ref 0.18.0 (2026-09-04)#
Features#
- Accepts esm-hist alongside historical in diagnostic data requirements. (#893)
- Ingests the obs4REF collection under its own
obs4refsource type. Theinstance_idforobs4refdatasets now starts withobs4REF. The diagnostics keep asking for obs4MIPs data. Where a dataset is ingested from both, the newest version is used, and the obs4MIPs copy wins a tie. Publishing a dataset to obs4MIPs therefore takes over from the registry copy without any change to the REF. A newer registry version is still preferred over a stale published one, so nothing reruns for no reason.
ref doctor gains three checks: misfiled-obs4ref, superseded-obs4ref and unsolvable-diagnostics.
Data ingested with --source-type obs4mips in earlier releases still solves,
but re-ingesting it with --source-type obs4ref is recommended.
A data requirement can now declare the collections that may supply it,
through the new fallback_source_types field on DataRequirement.
A dataset present under source_type still wins,
and the data is delivered under source_type whichever collection it came from.
The solver, ref doctor and the generated reference data documentation all read the declaration.
Every observational requirement declares obs4ref as a fallback. (#898)
- ref test-cases diff renders the regression baselines changed on a branch as a local HTML report.
Images appear old and new side by side, with a button that overlays them.
Text outputs get a coloured diff,
and NetCDF and other binaries are listed with a size delta and a link to each blob.
--html-dir chooses where the report is written,
and --no-fetch skips blob downloads and reports sizes only. (#910)
- ref test-cases diff --upload <prefix> publishes the report to a public object store,
and --comment-output <path> writes the pull request comment markdown that links into it.
Reports are stored in their own bucket, separate from the baselines. (#912)
- Summarise the changes to .nc files in the ref test-cases diff report.
Each file gets a collapsible listing of its ncdump-style header and a table with one row per
data variable carrying min, max, mean, NaN count, the largest absolute and relative difference,
and the number of cells that differ. (#913)
- Regression baseline diffs are now published as an HTML report linked from a short pull request comment, with images shown side by side and NetCDF outputs summarised per variable. (#915)
- Diff the committed regression artefacts in the ref test-cases diff report, alongside the native outputs.
Each one carries an "in PR" pill, because those bytes also appear in the pull request's own file list.
Every case page also lists the folder structure of its captured baseline, the committed bundle included,
showing the files that did not move as well as the ones that did. (#919)
Improvements#
- The ESMValTool diagnostics that run on a single historical experiment now group by
experiment_id, so their metric values carry the experiment as a dimension. Each affected diagnostic has its version bumped, so existing executions are recomputed on the next solve. (#914)
Bug Fixes#
- Celery workers now reuse a single database connection for the lifetime of a worker process
instead of opening a new one for every result they handle.
An execution that is still in flight when
ref solvefails is now recorded as failed-retryable. (#879) - Fabricated CMIP7 test data now reaches years without real CMIP6 source data by repeating the final year, rather than relabelling the whole series onto the requested end date. (#887)
-
- Fabricated CMIP7 test data now bumps its dataset version,
so a series padded with repeated years no longer shares an
instance_idwith the real data.
- Fabricated CMIP7 test data now bumps its dataset version,
so a series padded with repeated years no longer shares an
- The synthetic CMIP7 catalog generator now extends full-length historical runs on every calendar.
(#895) - - The synthetic CMIP7 catalog now covers the five models using a 360_day calendar, which previously ended in 2014 and were dropped by any diagnostic needing coverage past that year. - CMIP7 tracking ids are now derived from the dataset instance id, so regenerating the catalog leaves unchanged rows alone.
(#896)
- Dataset discovery now walks the whole directory tree instead of stopping 10 directories below the root.
The CMIP7 DRS nests files deeper than that, so ingesting a MIP-DRS7 root found no files at all. (#897)
- Removed the immutable flag from read-only SQLite connections.
The flag told SQLite to skip locking, which did not hold when a background run was writing
to the same database. The connection is still opened read-only. (#908)
- PMP executions are now capped at two threads each.
Without a limit the scientific stack sized its thread pools from the machine's core count. (#911)
Trivial/Internal Changes#
climate-ref 0.17.2 (2026-08-14)#
Features#
-
Added
ref doctor, which checks a deployment for the problems that a solve hides rather than reports: reference data that is missing (so its diagnostics silently never run), data ingested under a source type no diagnostic requires (so nothing selects it), datasets whose files cover the same period twice, and reference collections that overlap. -
Findings are grouped by check, under the remedy they share, so a deployment missing twenty reference datasets reads as one instruction and twenty names rather than twenty copies of the instruction.
- Added a generated Reference data page listing every observational dataset the diagnostics require and which collection supplies it.
The page and
ref doctorare built from the same function, so they cannot disagree.
(#865)
- Added Database.session_scope, a context manager that hands out a short-lived database session.
The session is rolled back if the block raises and always closed,
while the engine and its connection pool stay shared.
Reader and its sub-readers now accept an optional session,
so a concurrent caller such as a web request can read through its own scope
instead of the shared Database.session, whose behaviour is unchanged. (#876)
Bug Fixes#
- Removed the password from the database URL that is logged on connection. A deployment using Postgres wrote its credentials into the log at every startup, and into anything that log was pasted into. (#865)
- Fixed
ref doctorproducing a different report on each run. Diagnostics that share a name were ordered by set iteration order, so two identical runs disagreed. (#870)
Improved Documentation#
- Fixed the development installation instructions, which wrote
ref.tomlby redirectingref config listinto a hand-made directory instead of usingref config init. The development documentation now links to the installation and getting started guides rather than repeating them. (#869)
Trivial/Internal Changes#
climate-ref 0.17.1 (2026-08-13)#
Improvements#
- Peak memory is now measured over the execution's own process tree unless the executor declares that the execution has its cgroup to itself.
proc_treeis now preferred tocgroupreadings, which cover every process in the container. (#868)
Bug Fixes#
- Fixed the
resources_exclusiveflag recorded against each execution. Executors now declare whether an execution has the container to itself, which meansref executions resourcesno longer treats contaminated readings as clean samples. (#868)
Improved Documentation#
- Documented the obs4MIPs reference datasets that must be fetched from ESGF because the obs4REF registry does not carry them
(
20CR-V2,C3S-GTO-ECV-9-0,CERES-EBAF-4-2-1,ERA-5andNOAA-NCEI-LAI-AVHRR-5-0), including which diagnostics require each one and thescripts/fetch-esgf.pycommands that fetch them. (#864)
climate-ref 0.17.0 (2026-08-13)#
Breaking Changes#
- Removed the
ref test-cases migrate-manifestsandref test-cases check-storecommands, along with the unusedvalidate_cmec_bundles,validate_series_regressionandcapture_executionhelpers and the slot.source.jsonstamp. The CI coupling gate no longer emits anexecuteaction, and adiagnostic_versiondecrease now always fails the gate, so reverting a diagnostic requires re-minting its baseline. (#854) - Removes the
ref test-cases buildandref test-cases synccommands.buildis nowref test-cases run --from-slot, which rebuilds the committed bundle from the native already in the output slot without executing.syncis no longer needed becausereplaywarms the same store cache when it materialises. Also replaces theNativeStoreprotocol and its three implementations (LocalFilesystemStore,PoochReadStoreandR2WriteStore) with a singleNativeStoreclass built from the store URL. (#858)
Features#
- Added opt-in solve-time selection of ESMValTool reference data. Diagnostics can now request ESMValTool observational and reanalysis datasets the same way they request other reference data so each request resolves against the tracked catalog. (#835)
- Added a
key_parsertoRegistryRequest, so the package that owns a registry owns how its keys are read. The ESMValTool provider uses it to fetch the reference data an opt-in diagnostic selects. The fetch and ingest sides parse through the same function, so a request cannot name a facet value that ingest would spell differently. (#843) - Ingests the ESMValTool reference data as part of
ref providers setup, so the catalog a diagnostic selects from is populated by the same command that downloads the data.sea-ice-area-basicis the first diagnostic to declare the requirement. It now selects the OSI-450 observations it compares against rather than being handed the whole registry. An existing installation already has the files on disk but no catalog entries for them, soref providers setup --provider esmvaltoolneeds running again before sea ice will solve. (#846) - Records what each execution cost in wall time, CPU time and peak memory.
ref executions resourcesaggregates the results per diagnostic or per provider to help right-size the executors. The capture can be turned off withexecutor.measure_resources = false, or withREF_EXECUTOR_MEASURE_RESOURCES=false. (#859) - Added queue routing for the Celery executor.
A TOML routing table named by the
REF_CELERY_ROUTESenvironment variable maps diagnostics to queue names, so each execution is submitted to a queue such asesmvaltool-largeand differently sized worker pools can consume them independently. Without the variable set, behaviour is unchanged. (#860)
Improvements#
-
Reworks the finalisation of datasets that were ingested with a
drsparser. -
Adds an
n_jobsconfiguration value that controls how many worker processes parse dataset files. Solve-time finalisation used to be permanently serial becausen_jobswas only plumbed throughref datasets ingest. - Finalisation now runs in chunks of whole datasets, committing each chunk before parsing the next. Progress is logged as it goes, and a cancelled solve keeps the work it has already done.
ref datasets ingestreports how many datasets are still missing metadata that can only be read from the files themselves. This makes the cost that thedrsparser defers to the first solve visible up front.- Fixes per-file metadata (
start_time,end_time) being broadcast across every file of a dataset during finalisation. A catalog loaded from the database repeats its dataset id once per file, and the per-row update did not account for that.
(#837) - Reworks how the controlled vocabulary and the grey list are distributed. Both now resolve the same way. Both now work on read-only filesystems and in air-gapped deployments.
- Adds
climate_ref_core.datawithPackagedResourceandLayeredResource. These resolve a data file across an operator override, a local cache and the copy shipped in the package. - The packaged copy is now the floor for both files, so neither the network nor a writable cache is needed to run.
paths.dimensions_cvandignore_datasets_fileare now optional. Leaving them unset uses the packaged copy.- A failed grey list refresh is no longer fatal. An unreachable network, an HTTP error and an unwritable cache all fall back to the cached or packaged copy with a warning.
- A cached grey list that has not been refreshed for 30 days is ignored in favour of the packaged copy.
- The grey list cache moved under
REF_DATASET_CACHE_DIR, so it shares one cache root with the dataset registries. The copy in the old location is left in place and is no longer read. paths.dimensions_cvis now resolved to an absolute path like every other path field, rather than being left relative.- Reading the configuration no longer imports the diagnostic bundle parsing machinery,
so
climate_ref.configno longer pulls in numpy and pydantic. - Removes
IgnoreDatasetsRefreshError, which nothing raises any more.
(#840)
- Improved reference dataset download times by fetching up to four registry files concurrently.
Set REF_DATASET_FETCH_WORKERS to tune the concurrency or to 1 for sequential fetching. (#848)
- Encoded Celery tasks and results as JSON instead of pickle, and compressed the message bodies with gzip.
pickle is no longer in accept_content, and bodies shrink by roughly 80%,
which cuts broker memory on a full solve.
Compression is tuned with CELERY_TASK_COMPRESSION and CELERY_RESULT_COMPRESSION,
or disabled by setting either to an empty string.
Messages queued by an earlier release cannot be decoded by this one,
so purge the queues as part of the upgrade and re-solve to requeue the outstanding work.
A deployment that overrides flower.celeryConfig needs to accept the new format there too. (#850)
- Extracts the duplicated per-verb driver loop in the ref test-cases CLI
into a shared VerbDriver helper,
covering selector validation, case enumeration, skip policy and the summary epilogue. (#856)
Bug Fixes#
- Fixed
esmvaltool/sea-ice-sensitivityfailing over the full CMIP6 archive. The diagnostic script expects one ensemble member per model, but every ingested member was fed into its single execution, trippingNone or too many matching files found for siconc. A newSelectFirstMemberconstraint now keeps one member per model (the lowest CMIP index, e.g.r1i1p1f1) for both the CMIP6 and CMIP7 requirements. TheCAS-ESM2-0grey-list entry was also broadened from a single member to model-wide, since its curvilinear siconc grid fails the CMOR check for every member. (#827) - Stops
climate_ref_pmpdownloading micromamba while the provider registry is built.
PMPDiagnosticProvider.configure resolved its conda executable eagerly,
which downloaded micromamba when it was missing or more than a week old.
configure runs whenever the provider registry is built, so that download sat on the solve path
and a solve on a host with no outbound network access failed there.
It now records where the executable will be, using the new CondaDiagnosticProvider.conda_exe_path.
The download stays in ref providers setup, and in the execute path that already installs on demand. (#840)
- Stored non-finite floats as null in JSON columns, so a metric series containing NaN can be ingested on PostgreSQL.
json.dumps writes the bare tokens NaN, Infinity and -Infinity by default,
which Python reads back and SQLite stores without complaint,
but PostgreSQL validates the value on insert and rejects the whole row. (#850)
- Fixed ilamb/mrsos-wangmao, ilamb/snc-esacci and ilamb/amoc-rapid,
none of which could complete over the full CMIP6 archive.
ilamb3 is now required at >=2026.6.15.
This also retires the xarray<2025.11 and scipy<1.16 pins, which existed only for ilamb3 issues the new release resolves.
amoc-rapid additionally needed a climate_ref_msftmz_to_rapid transform.
ILAMB's msftmz_to_rapid selects the Atlantic with a hardcoded isel(basin=0),
but the RAPID-2023-1a reference has no basin dimension. (#861)
- Fixed pmp/enso_proc and pmp/enso_tel errors found when running.
The REF ENSO driver required exactly one file per variable and raised NotImplementedError: Multiple paths found ... otherwise.
The driver now joins the files into one before handing them over. (#862)
climate-ref 0.16.2 (2026-07-15)#
Breaking Changes#
- Split the
ilambdataset registry: it now holds only the 3 ILAMB reference products without an obs4REF equivalent (mrsol/WangMao,tas/CRU4.02,evspsbl/GLEAMv3.3a), and the two region masks moved to a newilamb-regionsregistry. Theiombregistry has been removed entirely as all diagnostics have been migrated to obs4REF. The ILAMB and IOMB diagnostic configuration files (configure/ilamb.yaml,configure/iomb.yaml) now declarerealm: land/realm: oceanand, where relevant,region_masks: ilamb-regionsin place of the oldregistry:key. Thegpp-WECANNdiagnostic'sprrelationship now reads GPCP from obs4REF (NOAA-NCEI/GPCP-2-3/mon) instead of the retiredilamb/pr/GPCPv2.3/pr.nc(sha1e1b942863ec76a75aa972b6d75e2e08646741259). This is a different processing of the same underlying product, so if it moves an ILAMB regression baseline the baseline will be re-minted separately. (#815)
Features#
- Added dataset retraction.
ref datasets retract <slug>marks a dataset ineligible for future solve-time selection without deleting its row or its execution provenance links, andref datasets unretract <slug>reverses it. Retracting an already-retracted dataset (or unretracting an already-active one) is a no-op.ref datasets listnow surfaces retraction state via aretracted_atcolumn. Retracted datasets are excluded by default, matching what the solver selects, and can be shown with the new--include-retractedflag, which also lists every version of a dataset instead of only the latest. "Retract" follows the term ESGF uses for pulling a published dataset from recommended use. (#802) - Added two new source dataset types,
obs4REFandESMValToolReference, together with their database tables.
obs4REF is REF-curated observational data that follows the obs4MIPs metadata conventions that has not yet been published to the obs4MIPs ESGF archive.
ESMValToolReference is ESMValTool's own reference data, which is not CMOR compliant and so carries a smaller set of metadata.
The shared obs4MIPs-style column block used by obs4MIPs, PMPClimatology and now obs4REF was extracted into a ReferenceDatasetMixin. (#805)
- Adds dataset adapters for the obs4REF and ESMValTool reference dataset types, so both can now be ingested with ref datasets ingest --source-type obs4ref and ref datasets ingest --source-type esmvaltool-reference. obs4REF datasets also participate in solving, while ESMValTool reference datasets are ingest-only for now. (#814)
- Reworked the ILAMB provider to draw its reference observations from the ingested obs4MIPs and obs4REF datasets rather than its own bundled registry.
The few observations not yet published there (currently WangMao, GLEAMv3 and CRU4.02) are still fetched from the provider registry at execute time. ILAMB region masks moved to their own support registry.
Each dataset registry now declares which source type its files map to, if any, and whether it holds reference or support data. This is what lets the solver provision reference observations from the database. (#815)
- Added a --exclude-diagnostic option to ref solve.
Diagnostics can be excluded by slug or by provider/diagnostic pair, matched exactly and case-insensitively.
This uses the same token format as the REF_TEST_CASES_SKIP environment variable. (#820)
Improvements#
- Reworked the cloud radiative effects diagnostic so it runs on CMIP7.
Both source types now use the
ref/recipe_ref_cre_cmip7.ymlrecipe, with the time range rewritten per source type: 2002-2014 for CMIP6 and 2002-2021 for CMIP7. (#683) - Adds CMIP7 support to the ozone diagnostics.
All five now build from the new
ref/recipe_ref_ozone_cmip7.ymlrecipe. The lat-time, SH October, NH March and annual-cycle diagnostics run on both CMIP6 and CMIP7, and the stratospheric zonal-mean profile stays on CMIP6 with its reference switched to the ESACCI-OZONEL3-SAGE-OMPSproduct. (#704) - Switched the PMP extratropical modes-of-variability diagnostics that use sea-level pressure
(NAO, NAM, PNA, NPO, SAM) to the obs4MIPs
20CR-V2reference, fetched from ESGF, in place of the obs4REF20CRcopy. Their regression baselines were re-minted against the new reference. The sharedExtratropicalModesOfVariabilityversion bumps from 2 to 3. The surface-temperature modes (PDO, NPGO) share that class, so their baselines were re-minted under the new version too. Their reference is unchanged and only the last few significant figures move. (#801) - ESMValTool reference data (the OBS, OBS6, native6 and obs4MIPs files in the provider's
data.txtregistry) is now ingested as theesmvaltool-referencesource type, so it is recorded in the database. Selecting it at solve time is a separate follow-up. This also adds the CMIP5-eraOImonsea-ice table to the MIP-table to frequency mapping, which the OSI-450 sea-ice references use. (#815) - Reduced the
climate-refunit test runtime by reusing migrated database templates, parallelising pull request checks, and moving full archive solver regressions to the main slow test suite. (#825)
Bug Fixes#
- Made the transient climate response to cumulative CO2 emissions (TCRE) diagnostic compatible with CMIP7 data. (#702)
- Moved the test-case
catalog.paths.yamlsidecar out of the provider packages and into the dataset cache, at$REF_DATASET_CACHE_DIR/test-case-paths/{provider}/{diagnostic}/{test_case}/. (#806) - Fixed
ref test-cases fetchaborting the whole fetch loop when the solver found no matching datasets for a single test case. The failure is now logged and the remaining test cases are still fetched, so one broken case no longer leaves later diagnostics without catalogs and paths sidecars.
Fixed the global-sst-bias example diagnostic to accumulate its global means in float64.
The float32 reduction differed by ~1e-6 between platforms,
which tripped the regression gate when baselines were minted on a different machine.
Its regression baselines were re-minted and the diagnostic version bumps from 1 to 2.
Loosened the regression comparison's default relative tolerance from 1e-6 to 1e-5,
so platform-level floating-point noise no longer trips the gate.
Committed bundles keep their 7 significant figures,
which is now two digits finer than the tolerance. (#810)
- Handles an ESGF search that returns no results by raising DatasetResolutionError.
This lets ref test-cases fetch log and skip a test case whose dataset has been de-indexed,
instead of crashing the whole prefetch. (#811)
- Enables the CEDA Solr index as an additional ESGF search index when fetching data.
The obs4MIPs ozone reference C3S-GTO-ECV-9-0 is currently missing from the default
ESGF2-US-1.5-Catalog index but is still served by CEDA,
so this lets ref test-cases fetch and scripts/fetch-esgf.py resolve it again. (#812)
- Fixed the PMP regression baselines drifting on CI.
The annual-cycle climatology version stamp is now derived from the diagnostic version instead of the run date,
the conda platform in CMEC provenance is redacted to a placeholder,
and the affected baselines were re-minted on linux-64 to match the CI platform. (#816)
- Fixed the ozone-zonal diagnostic by shipping the ESACCI-OZONE L3-SAGE-OMPS ozone profile file
that the updated ESMValTool recipe references in the dataset registry.
Also re-minted the ESMValTool regression baselines that drifted after the ESMValTool version update
and improved the error message for catalogs built from unfinalised CMIP7 datasets. (#817)
- Fixes the ILAMB lai-avh15c1 test case running the CI runner out of memory and getting killed with no logs.
- Coarsens the very fine AVHRR LAI reference one time block at a time instead of loading the whole ~0.05 degree cube into memory at once
- Caches the coarsened reference to disk, keyed on the source data and target resolution, because every model comparison in a solve reuses the same coarsened field. The first execution computes it, later ones read the cached file. Set
REF_ILAMB_COARSEN_NO_CACHEto disable. - Logs wall clock, CPU time, and peak memory after each test case in
ref test-cases runand the per-provider drift tests. - Re-mints the
lai-avh15c1baselines against the chunked coarsening, and thegpp-fluxnet2015baselines to adopt the CFlong_namea newer ILAMB now sets on the region traces.
(#818)
- Fixed the GitHub release listing a stray default.gitignore alongside the wheels and sdists.
uv build writes a .gitignore into the build output directory,
and the release workflow's dist/* glob was uploading it as a release asset.
The upload is now restricted to the built wheels and source distributions. (#819)
- Captures the pressure level for the PMP annual cycle diagnostic's 3D variables.
The ta, ua, va and zg variables are evaluated at fixed pressure levels,
but the level was dropped when the scalar values were written to the database.
Every level landed with identical dimensions, so the per-level values could not be told apart.
- Adds a
leveldimension to the controlled vocabulary so it is stored and queryable. - The annual cycle diagnostic reports
levelas a facet and its version bumps to 5. - Fixes the plots and data files for 3D variables, which were never captured because the output was globbed from a directory that never existed.
(#821)
- Re-dirties the execution group when a run fails retryably, so the failure is actually retried on the next solve.
A group that reran because its input hash changed starts from dirty=False,
so a retryable failure (timeout, OOM, worker death, missing log file) left the group clean
and the next solve never picked it back up.
The stale-execution reaper had the same gap and now marks the group dirty too.
Guards the success-path output copy so a diagnostic that reports success but under-declares
a referenced output file fails only its own execution.
copy_execution_outputs raising FileNotFoundError used to abort the whole result-collection loop
inside the local executor and strand every not-yet-processed execution in the batch at successful=None. (#822)
- Made long_name nullable on the obs4MIPs, obs4REF and PMP climatology dataset tables.
Every other dataset table already treated long_name as optional. (#824)
Improved Documentation#
- Rewrites the architecture background page to describe the system as implemented, with C4 context, container and component diagrams, a package layering diagram and sequence diagrams for the main runtime flows. (#813)
climate-ref 0.16.1 (2026-07-08)#
Bug Fixes#
default_ignore_datasets.yamlis now shipped inside theclimate_refpackage and resolved withimportlib.resources. Theconfigandsolve_configpytest fixtures previously located it relative to the monorepo source tree, so they raisedValueErrorfor any project that installsclimate-reffrom a wheel. (#800)
climate-ref 0.16.0 (2026-07-08)#
Features#
- Metric values now carry a first-class
kindof eithermodelorreference, so model and reference (observation) values can be distinguished and filtered directly rather than inferred from provider-specific conventions. Reference series additionally record a stablereference_idcontent hash, so an identical observation produced by different executions is recognised as the same value. Series can also carry typed presentation metadata (value_units,value_long_name,index_units, andcalendar). (#772) - Every PMP scalar metric value now keys its reference identity on
reference_source_idand declares its role with the first-classkindofmodel. The ENSO diagnostics previously keyed reference identity onreference_datasets, so a consumer now reads reference identity the same way for every PMP diagnostic. ENSO references that the metrics package scores against two observation datasets jointly (for exampleHadISST_GPCP-Monthly-3-2) are kept as a single combinedreference_source_idvalue, because the score is one value that depends on both references and cannot be split per reference. (#775) - ESMValTool series now carry an explicit
kindof eithermodelorreference, so an observation curve can be distinguished from a model curve directly rather than inferred from the presence of a reference source. (#776) - ILAMB metric values now carry the shared
kindfield, distinguishing reference (observation) series from model series without the previous"Reference"sentinel. Series units are presented in a clean CF form so a model series and its reference agree, the typed presentation fields (value_units,value_long_name,index_units,calendar) are populated, and comparison scalars record the reference they were scored against so they can be grouped by it. (#777) - The obs4REF reference-data registry gained 33 new files covering 12 CEDA
obs_for_ref_v2observational datasets, including cloud and radiation observations (ESACCI-CLOUD), burnt-area (GFED-5-0), ozone (SAGE-CCI-OMPS), FLUXNET site fluxes, soil carbon (HWSD-2-0), CALIPSO ice cloud, WOA-23 ocean biogeochemistry, RAPID AMOC transport, snow cover (CCI-CryoClim), and ocean carbon fluxes (Hoffman). Older WECANN and WOA2023 entries were replaced by their newer versions. (#778) - Ten ILAMB standard diagnostics now evaluate models against the obs4REF reference datasets rather than the bundled ILAMB files.
This updates the version of:
- gross primary productivity (WECANN, FLUXNET2015)
- runoff (LORA)
- soil carbon (HWSD-2-0)
- net biome productivity (Hoffman)
- snow cover (CCI-CryoClim),
- burnt area (GFED-5-0)
- sea-surface temperature and salinity (WOA-23),
- RAPID Atlantic overturning transport.
(#779)
- Added climate_ref.results, a typed read layer for querying results from notebooks and other tools.
A single Reader exposes per-domain readers for metric values, executions, datasets, diagnostics, and output artifacts, each returning frozen objects that remain usable after the database session closes and can be turned into a pandas DataFrame.
Added a ref diagnostics list command that lists the registered diagnostics with their promoted version and per-diagnostic successful, in-flight, and total execution-group counts.
Metric values now expose a first-class kind (model or reference) field, and ref datasets list returns only the latest version of each dataset. (#780)
- Added REF_TEST_CASES_SKIP,
a comma-separated list of diagnostic slugs or provider/diagnostic pairs
that are excluded from both ref test-cases fetch and the per-provider no-drift test.
This keeps diagnostics whose full-resolution ESGF input cannot be fetched within CI limits
(for example ilamb/thetao-woa2023-surface, a 3D ocean field)
from being downloaded or executed.
ESGF fetching is now confined to the populate-cache CI job. (#797)
- The ESGF fetch script can now retrieve the obs4MIPs reference datasets used by the PMP diagnostics,
namely GPCP-Monthly-3-2, TropFlux-1-0, CERES-EBAF-4-2, HadISST-1-1 and 20CR-V2.
Previously these were only available through the obs4REF registry.
Note that the psl modes of variability diagnostics request 20CR,
which ESGF publishes as 20CR-V2,
so those diagnostics continue to source their reference data from the obs4REF registry. (#799)
Improvements#
- Added a
registerargument toProviderRegistry.build_from_config. Registration is the only step that writes to the database, so read-only consumers such as the API can now build the registry withregister=Falseand serve a database mounted read-only. (#791) - Removed the ERA-20C and ERA-Interim reference datasets from the obs4REF and sample data registries,
reducing the volume of data fetched by
ref datasets fetch-data --registry obs4ref. No diagnostic used either dataset, and neither is available from obs4MIPs. The unrelated ERA-5 dataset is unaffected.
If you have already fetched the sample data,
re-fetch it with fetch_sample_data(force_cleanup=True) so that the removed files are cleared from your local copy. (#799)
Bug Fixes#
- Made ECS diagnostic work with CMIP7 data. (#671)
- Made TCR diagnostic work with CMIP7 data. (#686)
- Widened the contiguous-timerange tolerance so monthly datasets split across multiple files are no longer dropped as false-positive gaps. (#774)
- Committed regression baselines no longer embed the machine-specific source-checkout path or the
host operating system in provider command-line provenance.
Both are now redacted to portable
<SOURCE_DIR>and<OS>placeholders, so a baseline minted on one platform (for example macOS) reproduces one re-derived on another (for example Linux in CI) instead of drifting. (#775) - Fixed execution listings duplicating an execution group when two of its executions shared the same timestamp; each group now resolves to exactly one latest execution. (#786)
- Decoupled fetching of the ignore datasets file from configuration loading;
the file is now fetched lazily at solve time rather than while the configuration is loaded,
so read-only commands no longer perform network I/O.
Added an
ignore_datasets_urlsetting (envREF_IGNORE_DATASETS_URL) whose empty-string value disables fetching entirely for offline and air-gapped deployments. A failed download no longer creates an empty placeholder file; an existing cached copy is reused unchanged, and a solve with no cached copy fails loudly rather than running without ignore-dataset protections. (#792) - Fixed the
LocalExecutorper-task timeout counting time spent waiting in the process-pool queue as execution time. The budget is now measured from when a worker actually starts a task, so a large backlog of executions queued behind other work is no longer culled en masse once it ages past the timeout. (#795) - Fixed
ref datasets ingest --n-jobsgreater than 1 crashing (SIGSEGV/SIGBUS) with the defaultcompleteCMIP6 parser. Parsing previously fanned out across threads, but the netCDF4/HDF5 wheels published on PyPI are not thread-safe. Parsing now runs using a process pool. (#796) - Fixed three integration-test failures caused by the self-hosted runner's shared cache and stable output directory leaking into tests that assumed a clean, default environment. (#797)
- The ESGF fetch script now exits with a non-zero status if any request failed,
rather than reporting
0 datasetsand exiting successfully. Transient ESGF access errors are retried with an exponential backoff, configurable via--max-attemptsand--retry-delay. A request that matches no datasets is reported but is not treated as a failure. (#799)
Trivial/Internal Changes#
climate-ref 0.15.0 (2026-06-30)#
Breaking Changes#
- The hash that identifies a diagnostic's input datasets is now computed deterministically across pandas versions and platforms.
It previously relied on
pandas.util.hash_pandas_object, whose output changes between pandas releases and platforms.
Because the underlying hash values change, the committed regression catalog_hash values have been regenerated,
and existing databases will re-run each execution once on first use after upgrading. (#741)
- The minimum supported Python version is now 3.12; Python 3.11 is no longer supported.
Update your environment to Python 3.12 or newer before upgrading. (#745)
- Removed the legacy central regression-testing framework in favour of the native baselines managed by ref test-cases.
The DiagnosticValidator and ExecutionRegression helpers, the diagnostic_validation, execution_regression, and regression_data_dir pytest fixtures (from climate_ref.conftest_plugin), the central tests/test-data/regression/ baseline tree, and the per-provider test_diagnostics / test_build_results tests have all been removed.
Provider regression baselines are now recorded and verified entirely through the ref test-cases CLI and the per-package committed bundles (see Regression baselines). (#757)
Features#
- Added
ref test-cases mint,replay, andsynccommands for managing native regression baselines.mintrecords a test case's native outputs into the baseline store,replayregenerates a bundle from the stored outputs and checks it against the committed copy, andsyncdownloads the native outputs referenced by committed baselines into the local cache. (#724) - Added the
ref test-cases ci-gatecommand, which decides how CI should verify each test case's regression baseline against the base branch: skip it, replay the cached native baseline, re-run the diagnostic in full, or fail an unauthorised change. Baselines are now coupled to their input catalogs, so changing a test case's inputs without regenerating its baseline is reported as a failure rather than passing silently. (#727) - Added a Cloudflare R2 object-store backend for diagnostic regression baselines, so
ref test-cases mintcan publish a test case's native outputs to the shared baseline store rather than only a local directory. The store endpoint and bucket are configured through theREF_NATIVE_STORE_*settings (defaulting to the project's public baseline bucket), and write credentials are supplied via the environment or a named AWS/R2 profile, never the persisted configuration.
Minting is also easier to get right.
ref test-cases mint now checks the store credentials and bucket up front and stops with a clear message when they are misconfigured, the new ref test-cases check-store command verifies store connectivity without minting, ref test-cases mint --dry-run previews what would be minted, and ref test-cases fetch restores a missing local paths file on its own so a fresh checkout no longer needs --force. (#732)
- Regression baselines are now verified automatically in CI.
Every pull request runs the coupling gate and replays each test case's cached native baseline, a nightly job re-checks every baseline for silent drift, and a manually gated workflow mints new baselines to the shared object store behind required reviewers — so write credentials never run on untrusted pull-request code.
The diagnostic testing guide also gained a "pull request workflow" section explaining what the gate decides for each change and how to publish a new baseline. (#733)
- ILAMB regression baselines are migrated to the per-package Framework B layout, starting with the mrsos-wangmao, gpp-fluxnet2015 and lai-avh15c1 cmip6 test cases. Each carries a committed CMEC bundle and a manifest; native blobs are published separately through the gated mint workflow. ILAMB executions now also declare their scalar CSV and netCDF outputs in the CMEC output bundle so they are persisted with the results and can be replayed. (#738)
- Extended the ILAMB cmip6 regression-baseline migration to the per-package Framework B layout with nine more test cases:
gpp-wecann, mrro-lora, csoil-hwsd2, nbp-hoffman, snc-esacci, burntfractionall-gfed, emp-gleamgpcp2.3, thetao-woa2023-surface and so-woa2023-surface. (#743)
- ref test-cases runs now materialise their outputs into an inspectable, gitignored output slot
(<case>/output/<label>/) holding the curated native files, the rebuilt committed bundle, and a source stamp,
so you can see and diff exactly what a run produced.
Added a build verb that rebuilds the committed bundle from an existing slot without re-executing the diagnostic,
a mint --from-replay option that re-authors a baseline from the stored native instead of re-running it,
and a --label option that keeps several runs side by side for comparison. (#748)
- Added a --format json option to ref datasets list, ref providers list, ref executions list-groups, and ref config list for machine-readable output with full, untruncated identifiers and paths. (#765)
- Added ref config init, get, set, unset, edit, and validate commands to make configuration onboarding and management easier from the CLI. (#768)
- Added a five-minute Quick Start guide
and a new model-vs-observation diagnostic to the example provider,
global-sst-bias,
which compares modelled global-mean sea surface temperature (tos, area-weighted by areacello)
against the HadISST-1-1 observations and emits scalar metrics, the model/reference/bias time series, and figures.
A curated quickstart data registry provides the single reference dataset it needs,
avoiding the multi-gigabyte obs4REF download. (#769)
- The CI coupling gate now couples each regression baseline to its diagnostic's Diagnostic.version.
A test case fails the gate when the diagnostic's in-code version no longer matches the version recorded
when its baseline was minted, so an execution-affecting change can no longer slip through as a green replay
against a stale committed bundle.
The new author-declared diagnostic_version field lives in the test case manifest.json (schema version 2),
and the new ref test-cases migrate-manifests command backfills it across existing baselines.
A legitimate revert is still permitted: lower the diagnostic version and re-mint the baseline. (#770)
Improvements#
- ILAMB reference (observational) series are now tagged with their source via the existing
reference_source_iddimension (for examplereference_source_id="WangMao"), so a reference series is self-identifying without adding a new schema field. (#743) - Committed regression baseline bundles (
series.jsonanddiagnostic.json) now round floating-point values to seven significant figures when written, giving stable, reviewable bytes that no longer churn between local and CI runs while staying well within the regression comparison tolerance. (#744) - Reduced database size by deduplicating the index of series metric values into a shared
index_axistable, referenced by each series rather than stored inline on every row. A database migration backfills existing data losslessly; on SQLite, runVACUUMafterwards to release the freed space on disk. (#747) - Sped up
ref executions reingestby eager-loading datasets and batching the database writes, so bulk reingests now scale with the number of executions rather than the number of data files. (#750)
Bug Fixes#
- Regression
series.jsonbaselines are now written with a stable, dimension-sorted order. Previously a diagnostic could emit its series in a platform-dependent order, causing a committed bundle minted on one platform to falsely fail the regression gate on another even when every value matched within tolerance. (#738) - Selecting the latest dataset version now compares version numbers numerically, so versions like v10 are correctly preferred over v2 (previously the older version could be chosen). (#739)
- Fixed a bug where a diagnostic whose results failed to ingest was incorrectly recorded as successful with no metric values; such executions are now marked failed and retried on the next solve. (#740)
- ILAMB regression
output.jsonbundles are now written with a stable, sorted block order (the plot and data sections are sorted by filename), so a baseline regenerated on one machine reproduces byte-for-byte on another instead of reordering and producing a spurious diff. (#743) - Fixed
ref executions reingestreporting a reingest as successful when the result failed to re-ingest; a failed re-ingestion is now rolled back (leaving no orphaned execution row or output directory) and reported as a failure instead. (#749) - ILAMB metric bundles now always report the
unitsattribute as a string (for example"1"for dimensionless metrics) instead of sometimes emitting the number1, keeping the metric schema consistent across diagnostics and stable for regression comparison. (#755) - PMP climatology reference datasets whose institution identifier contains a hyphen (for example the
GPCP-3-3precipitation climatology fromNASA-GISS) are now resolved by the registry instead of being silently skipped, so diagnostics that depend on them no longer fail with "No datasets found matching facets". (#761) - Committed regression baselines are now reproducible from their stored native blobs on any machine: native outputs are rewritten to portable placeholders (
<OUTPUT_DIR>/<TEST_DATA_DIR>/<SOFTWARE_ROOT_DIR>) before snapshotting, so re-minting on a different host yields identical digests instead of churning the baseline store. Diagnostics can also declarereconstruction_inputs— extra output globs such as ESMValTooldiagnostic_provenance.ymlor the PMP driver*_cmec.json— that are persisted into the baseline, soref test-cases replayrebuilds the bundle by re-runningbuild_execution_resultagainst the captured inputs. (#761) - Fixed minting of the ESMValTool ECS, TCR, TCRE and ZEC diagnostics, which failed with
can only concatenate str (not "datetime.timedelta") to strwhen their datasets were loaded from a committed catalog.get_child_and_parent_datasetnow parses the YAML-serialised stringstart_timeback to a calendar-aware cftime datetime before deriving the parent timerange. (#764) ref --quiet --helpno longer prints debug log messages to stderr, and successful solves no longer log a spuriousERRORwhen an optional dataset source type (such as CMIP7) is not present in the data catalog. (#765)ref solve --dry-runis now a read-only preview and no longer creates execution groups in the database. It still reports execution groups with stale, abandoned in-progress executions as runnable, matching what the next real solve would pick up.
ref solve now exits with an error and remediation advice when no diagnostic providers are configured, or when a --provider filter matches none of the configured providers, instead of exiting successfully without doing anything. (#766)
Improved Documentation#
- Corrected and expanded the regression-testing documentation, fixing inaccuracies in the diagnostic testing guide and filling gaps in the regression baselines reference. (#744)
- Consolidated the In a Nutshell, Basic Concepts, and Explanation documentation into a single Concepts page, and corrected the documented default dataset parser (the complete parser is the default, not the DRS parser). (#751)
- Corrected command examples in the documentation and package READMEs that used invalid flags (for example
---output-directoryand--output directory) or linked to a renamed helper script (fetch-esgf.py). (#765)
Trivial/Internal Changes#
climate-ref 0.14.7 (2026-06-18)#
Bug Fixes#
- Fixed
ref solve --timeout 0discarding completed work instead of waiting for it. A timeout of0(or any non-positive value) now means "wait with no time limit", so executions are collected, copied to the results directory, and ingested as expected; previously they ran but their outputs were left in the scratch directory. Use--no-waitto queue executions and exit immediately, and recover any orphaned scratch outputs from an earlier run withref executions reingest --include-failed. (#735) - Fixed
ref executions reingestfailing for ESMValTool diagnostics. Reingest copies an execution's outputs to a new directory, but the absolute paths recorded in ESMValTool's provenance files still pointed at the original location, which caused an error that silently rolled back the reingest. These embedded paths are now re-pointed at the new output directory so the results are reingested as expected. (#736)
climate-ref 0.14.6 (2026-06-17)#
Bug Fixes#
- Fixed loading the default configuration in environments where the user cache directory cannot be written (for example, read-only or restricted HPC systems).
Previously, running a
refcommand without an existing configuration file could fail withError loading configurationbecause the default ignore-datasets file could not be cached. The REF now degrades gracefully and continues without the cached ignore-datasets file. (#734)
climate-ref 0.14.5 (2026-06-17)#
Bug Fixes#
- Made ESMValTool regression baselines deterministic by giving each captured run a stable execution directory instead of a timestamped one, so re-running a test case no longer churns the committed output. (#714)
- Fixed fetching datasets from ESGF failing with
Must have equal len keys and value when setting with an iterableunder pandas 3.0, by running the intake-esgf catalogue build with the legacy object-string dtype. (#726)
Trivial/Internal Changes#
climate-ref 0.14.4 (2026-06-04)#
Improvements#
- Added GPCP dataset version to 3.3 for the PMP precipitation climatology reference dataset. (#672)
- Updated GPCP dataset version to 3.3 for the PMP precipitation climatology reference dataset. (#684)
- Removed the pandas upper bound and fixed pandas 3 compatibility in catalog and ILAMB result handling. (#708)
Bug Fixes#
- Updates the PMP provider data directory (#685)
- Fixed a
KeyError: 'branded_variable'when solving diagnostics against CMIP7 datasets. Thebranded_variablefacet is now reconstructed when a data catalog is loaded from the database, so it is available to data requirement filters. (#712)
Improved Documentation#
- Added a Zenodo DOI badge to the README and documentation landing page so users can easily find how to cite the REF. (#634)
- Added a getting-started guide for modelling centres that explains how to evaluate local or pre-publication CMOR-compliant model output with Climate-REF, including deployment options, accessing results, and a suggested first run. (#709)
Trivial/Internal Changes#
climate-ref 0.14.3 (2026-05-18)#
Bug Fixes#
- Fixed a race condition when trying to create the same
ExecutionGroupconcurrently.Database.get_or_createandDatabase.update_or_createnow wrap the INSERT in a SAVEPOINT and re-fetch the winning row on conflict instead of aborting the transaction. (#679)
climate-ref 0.14.2 (2026-05-15)#
Bug Fixes#
- Changed the behaviour for
register_datasettreatment of files absent from the current ingest slice as kept-in-place with a warning, instead of raising NotImplementedError. (#677)
climate-ref 0.14.1 (2026-05-14)#
Features#
- Added a
--chunk-sizeoption toref datasets ingest(CMIP6 and CMIP7) that streams the catalog in directory-aligned batches instead of loading the whole archive into memory at once. Peak memory is now bounded bychunk_sizerather than by the total number of files in the input tree. (#674)
Improvements#
- Skip the redundant per-dataset
validate_data_catalogcall insideregister_dataset. The production ingest path already validates the catalog (and each streamed chunk) once up-front, so the inner re-validation only duplicated work. Cuts ~20% off ingest wall time on a 50k-file / 500-dataset synthetic CMIP6 archive. A cheap per-slice guard (no groupby) remains so callers that bypass the upstream validation contract still get a clear error instead of silently registering inconsistent metadata. (#675)
Bug Fixes#
- Stopped re-ingesting unchanged CMIP6 files on every run.
Previously,
ref datasets ingestreported every file as updated and emitted a "Updating file metadata" warning per file even when nothing on disk had changed, because the file-metadata comparison treated astrloaded from the database as unequal to a freshly parsedcftime.datetime. Re-ingesting an unchanged directory is now a no-op. (#673)
climate-ref 0.14.0 (2026-05-12)#
Features#
- Write the diagnostic
versionto the database.
Bumping a diagnostic's version now creates a fresh execution group
(preserving the prior version's group and results) instead of overwriting it,
and each execution is stamped with the provider version that produced it. (#667)
Improvements#
- Reworked the layout of execution output directories.
New executions now write to
<provider>/<diagnostic>/<group_short>/<execution_id>/instead of<provider>/<diagnostic>/<dataset_hash>/, so reruns of the same diagnostic group no longer overwrite earlier outputs. Existing rows on disk continue to resolve through their storedExecution.output_fragment. (#655) - Added diagnostic versioning read-path foundations.
New database columns track the diagnostic version for each execution group,
the promoted version for each diagnostic, and the provider version for each execution.
Default queries now return only results at the promoted version, and the
ref executions statscommand reflects the same filter. (#665)
Bug Fixes#
ref datasets ingestnow exits with a non-zero status when one or more input directories fail to ingest, so cron- and Kubernetes-based deployments can detect failures. The CMIP6, CMIP7, and obs4MIPs adapters also now tolerate individual files with missing DRS components: those files are logged and skipped instead of aborting the whole batch. (#668)
climate-ref 0.13.2 (2026-05-10)#
Features#
- Added support for Python 3.14. (#625)
Improvements#
- Add extra options for users to ingest options other than base to parsl functions (#651)
- Updated ESMValCore to v2.14.0 and ESMValTool to 2.15.0.dev15+gdead90ca8. (#652)
Bug Fixes#
- Fixed
ref test-cases fetchaborting the entire run when a single test case could not be parsed (for example because of aPermissionErroron a cached CMIP6 file); the failing test case is now logged as a warning and the loop continues. (#639) - Fixed
ref solveperiodically reporting executions as never finishing. Stuck executions left in the in-progress state by a crashed worker, walltime kill, or hung diagnostic are now reaped and retried on the next solve, the CLI--timeoutdefault has been raised to 6 hours to match the worker time limit, and per-task timeouts in the local executor cancel hung diagnostics instead of blocking the whole pool. (#641)
Improved Documentation#
- Updated CITATION.cff with extra author names and orcid ids. (#636)
Trivial/Internal Changes#
climate-ref 0.13.1 (2026-04-13)#
Features#
- Added first-class read-only support and a migration-status helper to the
DatabaseAPI.
Database.from_config(..., read_only=True) rewrites file-based SQLite URLs to read-only URI form and skips migrations,
and Database.migration_status(config) reports the current/head revisions and state. (#624)
climate-ref 0.13.0 (2026-04-11)#
Features#
- Added timeseries extraction for ESMValTool regional historical and ozone polar cap diagnostics, and updated file patterns to match the current ESMValTool output directory structure. (#607)
- Added
ref executions reingestcommand to re-ingest existing execution results without re-running diagnostics. Creates a new immutable execution record with a timestamped output fragment, leaving the original execution untouched. (#610) - Added
ref dbCLI subcommand group for database management. Includes commands for running migrations, checking schema status, viewing migration history, creating backups, executing SQL queries, and listing tables. (#615)
Improvements#
- Fetch ESMValTool recipes when installing the provider. (#582)
- Unify facet filter parsing across CLI commands.
--filterinexecutions list-groupsanddelete-groupsnow supports multiple values for the same key with OR semantics (e.g.,--filter source_id=A --filter source_id=B), consistent with--dataset-filterindatasets listandsolve. (#613) - Improved solver performance by batching dataset finalisation before grouping, avoiding redundant file I/O when multiple groups share overlapping datasets. (#616)
Bug Fixes#
- Fixed CLI test isolation by making the
invoke_clifixture depend on theconfigfixture, ensuring tests use an isolated database rather than the user's real one. Also markedRLIMIT_AStests as expected failures on macOS where this resource limit is not supported. (#611) - Copied the scratch directory for the previous execution when reingesting (#612)
climate-ref 0.12.3 (2026-03-30)#
Features#
- Added
ref datasets statsandref executions statsCLI commands for viewing summary statistics without listing individual records. (#584)
Bug Fixes#
- Fixed zero-emission-commitment diagnostic failures caused by space-separated
activity_idvalues creating path mismatches, incorrect parent timerange computation when datasets are split across multiple files, and models withesm-1pctCO2as parent experiment being incorrectly scheduled (see #586). (#585) - Fixed experiment selection for computing ZEC. (#589)
- Solve regression tests now use the local
default_ignore_datasets.yamlinstead of downloading from themainbranch on GitHub, ensuring tests reflect the current ignore list. (#606) - Use flexible time stamp for PMP annual cycle. (#465)
Improvements#
- Enable multiple-file-input for PMP's variability modes diagnostics. (#583)
Improved Documentation#
- Added a how-to guide on controlling memory use and parallism during diagnostic execution. (#591)
climate-ref 0.12.2 (2026-03-06)#
No significant changes.
climate-ref 0.12.1 (2026-03-06)#
Bug Fixes#
- Improved conda error logging by capturing stderr in solve logs and avoiding unnecessary stacktraces. (#580)
Trivial/Internal Changes#
climate-ref 0.12.0 (2026-03-03)#
Features#
- Added ESMValTool ozone diagnostics. (#473)
- Metric for diagnostics of double ITCZ was added: Spatial corrleation of simulated DJF precipitation climatology against reference dataset over the area of 20S to 0 latitude and 100 to 210 longitude. (#557)
- Added lazy loading and finalisation support for CMIP7 datasets via DRS and complete parsers, matching the existing CMIP6 pattern.
This removes the realm filter for CMIP7 executions as it cannot be parsed from the DRS.
We strongly recommend the use of branded variables when filtering CMIP7 datasets to properly constrain the expected variables. (#571)
- Update ilamb3, remove the ohc-noaa diagnostic and add the evspsbl-pr diagnostic. (#573)
- Added time_units and calendar metadata to CMIP6 and CMIP7 datasets, enabling proper handling of non-standard CF calendars such as 360_day and noleap. Time values are now stored as cftime.datetime objects instead of datetime.datetime. (#574)
- Added CMIP7 data catalog and ESMValTool recipe variants for CMIP7 diagnostics. (#577)
Improvements#
- Implemented a memory constraint using the environment variable
MEMORY_LIMIT_PARSL_JOB_GBto set the memory limit (units: GB) for a PARSL worker. (#464) - Extracted shared finalisation logic into
FinaliseableDatasetAdapterMixin, reducing code duplication between CMIP6 and CMIP7 adapters. (#571) - Use the metadata from the data catalog instead of reading netCDF files when determining ESMValTool branch times (#577)
Bug Fixes#
- Fixed fire diagnostic CMIP7 data selection and recipe writing. (#540)
- Migrated to use a different post-processed JSON file for mapping CMIP6 compound names to CMIP7. This fixes some errors when dealing with tasmax/tasmin and the removes the out_name attribute which was correctly included in #530. (#547)
- Apply same version filtering logic to the regression tests as loading the data catalog (#570)
climate-ref 0.11.1 (2026-02-24)#
Bug Fixes#
- Fixed DRS re-ingestion from crashing or regressing already-finalised datasets. Previously, re-ingesting the same directory with the DRS parser would either crash with a
TypeErrordue topd.NAcomparisons, or overwrite finalised metadata with empty values. Finalised datasets are now skipped during DRS ingestion while still adding any new files.
Reduced memory usage during dataset ingestion by releasing ORM objects from the SQLAlchemy session after each dataset commit, preventing unbounded memory growth on large archives. (#567)
climate-ref 0.11.0 (2026-02-24)#
Breaking Changes#
- Changed
get_branding_suffix,get_realm, andget_cmip7_compound_nameto require atable_idparameter in addition tovariable_id, enabling Data Request compound name lookups. (#530) - Failed diagnostic executions now clear the execution group's dirty flag, preventing automatic retry on subsequent solves.
Previously, failed executions were retried indefinitely.
Use ref solve --rerun-failed or ref executions flag-dirty to explicitly retry failed diagnostics.
The solver also now skips duplicate submissions when an execution with the same dataset hash is already in progress. (#552)
Features#
- Added a constraint to add the parent experiment. (#214)
- Added lazy dataset ingestion with two-phase finalisation. Datasets are now bootstrapped from directory structure metadata only (no file I/O), with full metadata extracted lazily at solve time after filtering narrows candidates. This dramatically reduces ingestion time for large CMIP6 archives on HPC parallel file systems. (#515)
- Added CMIP7 support to all ESMValTool diagnostics using OR-logic data requirements, enabling automatic evaluation of CMIP7 datasets alongside existing CMIP6 support. (#519)
- Added CMIP7 data requirements and test data specifications for all PMP diagnostics (annual cycle, ENSO, and variability modes). (#526)
- Added
esgf_data_catalogtest fixture and per-provider solver regression baselines using pre-generated parquet catalogs, enabling solver regression testing without requiring sample data downloads. (#529) - Added structured CMIP6-to-CMIP7 variable mappings from the CMIP7 Data Request, with a typed
DReqVariableMappingclass for reliable branding suffix, realm, and output name lookups. (#530) - TOML file for QAQC requirement from REF added. (#532)
- Added dimensions to files produced by ESMValTool diagnostics. (#534)
- Added CMIP7 data support to ILAMB diagnostics, enabling dual CMIP6/CMIP7 data requirements with branded variable name lookups and dynamic source type detection. (#535)
- Distinguished system errors (OOM, disk full, worker crash) from diagnostic logic errors when handling execution failures. System errors leave the execution group dirty so they are automatically retried on the next solve, while diagnostic errors clear the dirty flag to prevent retrying indefinitely with the same data.
The solver also now skips duplicate submissions when an execution is already in progress for the same dataset hash.
Added --rerun-failed and --no-wait flags to ref solve, and a new ref executions fail-running command for marking stuck executions as failed. (#552)
- Added validation that prevents DataRequirements from filtering or grouping on columns that require dataset finalisation, raising a clear error instead of silently producing empty results. (#561)
- Added --limit flag to the solve command to cap the number of executions, and --dataset-filter option to both solve and datasets list commands to filter input datasets by facet values before solving. (#562)
Improvements#
- Improved solver performance by caching slug column lookups and avoiding expensive DataFrame string representation in debug logging. (#533)
- Removed the
ecgtoolsdependency and 16 transitive dependencies (intake, intake-esm, joblib, zarr, etc.) by replacing it with a focused internal catalog builder module. This also eliminates the pydantic v1 deprecation warning that ecgtools was causing. File parsing now shows a tqdm progress bar. (#558) - Replaced xarray with netCDF4 for metadata-only reads during dataset ingestion, significantly reducing per-file parsing overhead. (#559)
- Parallelised the
finalise_datasetsoperation for CMIP6 datasets, mirroring the threaded approach used during ingest. The number of worker threads is controlled by the existingn_jobsparameter onCMIP6DatasetAdapter. (#564)
Bug Fixes#
- Removed deprecated
mix_stderrparameter fromCliRunnerin the test fixture, fixing compatibility with Click 8.3+. (#528) - Fixed a
ValueErrorinAddSupplementaryDatasetwhen the data catalog contained duplicate index labels for supplementary datasets. (#537) - Fixed confusion between variable_id and out_name in fake CMIP7 data. (#539)
- Improved the resiliance of the celery worker configuration to failures (#550)
- Mount a
celeryconfig.pyvia ConfigMap for Flower soaccept_contentis read correctly by Celery's config loader. TheCELERY_ACCEPT_CONTENTenv var is not picked up by Flower/Kombu directly. The config is user-configurable viaflower.celeryConfigin Helm values. (#556) - Fixed
REF_CMIP6_PARSERandREF_LOG_FORMATenvironment variables not being applied because theConfigclass was missing the post-init hook for environment variable overrides. (#561) - Resolved pandas FutureWarnings to support both pandas 2 and 3, including fixes for DataFrame concatenation with empty or all-NA entries and null-type mismatches in parquet round-trips. (#565)
Trivial/Internal Changes#
climate-ref 0.10.0 (2026-02-10)#
Features#
- Added database support for CMIP7 datasets based on the CMIP7 Global Attributes v1.0 specification. (#503)
- Added CMIP7 data requirements support, enabling providers to fetch CMIP6 data from ESGF and translate it to CMIP7 format using the CMIP7 CV converter. (#510)
- Added diagnostic summary introspection and auto-generated documentation for all providers. The
ref providers showcommand now defaults to detailed list format and supports--columnsfor filtering table output. (#518)
Improvements#
- Separated model from observation runs for regional historical diagnostics. (#460)
- Made listing the changed files from a regression test faster. (#514)
- Prepared the monorepo for splitting diagnostic provider packages into independent repositories. Extracted shared test fixtures into a
climate-ref[test]pytest plugin, decoupledclimate-ref-corefrom application-level types, and added API surface documentation, versioning strategy, provider compatibility CI, and a copier template for bootstrapping new provider repositories. (#520)
Bug Fixes#
- Fixed CMEC bundle dimension validation to use a subset check instead of exact equality, allowing diagnostics with multiple data requirements to have varying output dimensions. (#523)
Trivial/Internal Changes#
climate-ref 0.9.1 (2026-02-05)#
Features#
- Added
ingest_data()lifecycle hook to providers, enabling automatic dataset ingestion duringref providers setup. PMP climatology data is now ingested automatically, eliminating the need for a separate manual ingestion step. (#508)
Improvements#
- Implemented coupled versioning for Helm chart: chart version, appVersion, and default image tag now stay in sync with the application version and are updated automatically by bump-my-version. (#507)
- Improved CLI performance by skipping database backup for read-only commands like
config listanddatasets list. (#511) - Improved CLI startup time by deferring heavy imports until needed. (#512)
Bug Fixes#
- Fixed Helm chart CI to use correct image tag override path (
defaults.image.taginstead of invalidclimate-ref.image.tag). (#507)
Improved Documentation#
- Updated getting started documentation with clearer configuration and dataset download instructions. (#508)
climate-ref 0.9.0 (2026-02-03)#
Features#
-
Added test data management infrastructure for diagnostic development:
-
New
ref test-casesCLI commands (fetch,list,run) for managing and running diagnostic test cases. - ESGF data fetching utilities with support for CMIP6 and obs4MIPs datasets.
TestDataSpecificationandTestCaseclasses for defining reproducible test scenarios.
(#475)
- Add functionality to translate a CMIP6 dataset to the new CMIP7 conventions (#484)
- Added test automation infrastructure for diagnostic testing using test-cases (#485)
- Added CMIP6 to CMIP7 format conversion command-line script to translate CMIP6 datasets into CMIP7-compatible format. (#489)
- Added RegistryRequest class for fetching datasets from pooch registries (pmp-climatology, obs4ref) instead of ESGF. (#490)
- Added Helm chart for Kubernetes deployment with automated CI/CD pipeline for building and publishing the chart to GitHub Container Registry, including deployment templates for provider workloads (ESMValTool, PMP, ILAMB), Flower monitoring UI, Dragonfly Redis dependency, and comprehensive integration testing in minikube. (#492)
- Add CI workflow to verify solve works without network access (ci-offline-solve.yaml).
This test uses Docker with --network none to block all network access including
subprocesses. Runs every other day and can be triggered manually. (#497)
- Added provider lifecycle hooks for offline execution setup. Providers can now implement setup_environment(), fetch_data(), and post_setup() methods to prepare for execution on HPC compute nodes without internet access. A new ref providers setup CLI command runs all provider setup hooks, fetching required reference data to the local cache before offline solving. (#498)
Improvements#
- Clean up the open database connections in the test suite (#482)
- Improved catalog handling with hash-based change detection and multi-file dataset support. Enhanced CLI
test-casescommands with new flags:--only-missing,--force,--dry-run,--if-changed, and--clean. (#490) - Updated ESMValTool to v2.13.0 (#500)
Trivial/Internal Changes#
climate-ref 0.8.1 (2026-01-06)#
Bug Fixes#
- Add a pin for fastprogress (dependency of intake-esm) to work around bug in newer versions (#476)
Trivial/Internal Changes#
climate-ref 0.8.0 (2026-01-05)#
Features#
- Enable spatial 3-d variables that has levels for PMP annual cycle. (#411)
- Added ignore datasets constraint and configuration file.
If no path is configured for the ignore_datasets_file in the configuration file,
the default ignore datasets file is downloaded from the Climate-REF GitHub repository
if it does not exist or is older than 6 hours. (#447)
Improvements#
- Validated the slurm configurations of the HPCExecutor using pydantic (#375)
- Added reference values to ESMValTool series output. (#452)
Bug Fixes#
- General fixes found when rerunning, including handling an edge case where no log output is written, ignoring empty input directories and increased logging of the number of executions. (#444)
- Worked around pydata/xarray#2742 by always replacing the default fillvalue for the data type with NaN in arrays read with Xarray. (#454)
- Excluded piControl from PMP annual cycle and variability metrics owing to non-overlapping periods with observations. (#463)
Improved Documentation#
- Add a Jupyter notebook showing how to use the CMIP7 Assessment Fast Track website OpenAPI. (#466)
climate-ref 0.7.0 (2025-10-01)#
Breaking Changes#
- Use the directory structure template from the obs4MIPs specification to define the instance_id. (#383)
- Used logical or instead of logical and to combine multiple facet filters.
This means that the selected input datasets will include items that match any of
the specified FacetFilters, rather than only those that match all of them.
This change was made to improve the usability of the option to use multiple facet
filters, allowing users to be more specific with their filtering criteria. (#414)
- Changed all constraints so they return a filtered dataframe instead of a boolean.
This allows using the constraints in multi-model diagnostics. (#416)
Features#
- Add a timerange constraint to ensure the required data is available. (#399)
- Add a human readable representation for
climate_ref_core.datasets.ExecutionDatasetCollection. (#401) - Allow execution outputs to have dimensions. These dimensions allow the outputs to be filtered in the same manner as the metric values (#434)
- Adds filter options to the
ref executions list-groupscommand. This allows users to filter by facet, provider and diagnostic, as well as whether the latest executions was successful or if the exection group was has been marked as dirty (#438) - Added a
ref executions delete-groupscommand for removing unneeded results from the database. This will delete the execution group and any executions and outputs associated with the group. (#441)
Improvements#
- Updated the sample data to v0.7.3. (#402)
- PMP version updated to v3.9.2 (#404)
- Specified the "table_id" for all ESMValTool diagnostics to improve data selection. (#407)
- Added a groupby operation to the RequireFacets constraint to reduce failed runs because of missing data. (#408)
- Adds extra ESMValTool dimensions to the output series (#410)
- The dataset ingestion process now supports updating datasets if new files become available (#412)
- Added series output for ESMValTool diagnostics. (#413)
- Gracefully handles empty directories when ingesting. The default behaviour is now to skip any invalid datasets as that is a sane default when ingesting many datasets. (#419)
- Include obs4MIPs and thetao in the ESGF download script and turn into a CLI tool.
This script was renamed to
fetch-esgf.pytofetch-esgf.pyas it now includes more than just CMIP6 data. (#420) - Improved support for concatenating historical and future experiments in the ESMValTool recipe (#431)
- Log the ESMValTool recipe and configuration for easier debugging (#432)
- Adds output dimensions to ILAMB outputs. This adds additional dimensions and attributes to the series values (#435)
- Replaced instance_id by individual facets in group_by of ESMValTool diagnostics. (#439)
Bug Fixes#
- Resolves missing index and index names for series (#410)
- Added missing column to the PMP dataset model (#418)
- Removes the ERA-5 ta data Obs4REF in preference for Obs4MIPs (#421)
- Avoid selecting the areacella variable when filling the recipe for cloud scatterplots. (#427)
- Fixes to the ocean heat content IOMB diagnostic (#433)
Trivial/Internal Changes#
climate-ref 0.6.6 (2025-09-10)#
Features#
- Added a diagnostic to create scatterplots of two cloud-relevant variables. (#261)
- Added support for serialising series metric values from an execution. These series metric values are also ingested into the database for later retrieval.
Currently, only the ESMValTool example diagnostic supports this feature,
but this will be extended to other diagnostics in the future. (#374)
- Added regional historical changes diagnostics. (#380)
- Shifts ILAMB executions to be per experiment/model/ensemble/grid as opposed to all models for a given experiment. Also dumps out series information into the database for all time traces found in the output files. (#391)
- Made it possible to run PMP diagnostics without having conda installed. (#392)
- Added climate drivers for fire diagnostic. (#393)
- Added a sub-command to ref executions that allows you to flag an execution as dirty. (#396)
Improvements#
- Use a self-hosted CI runner for the integration tests. The GitHub runner do not have enough disk space to store the required datasets, and these data are downloaded over the internet on each run. (#365)
- Updated the sample data to v0.7.1. (#385)
- Adds additional indexes for some slow queries in the API (#395)
- Better error message when a user tries to use the HPCExecutor on Windows (#397)
Bug Fixes#
- Fix bug in writing ESMValTool recipes that loses order of preprocessing steps. This bug was introduced in #378 and included in the v0.6.5 release. (#384)
climate-ref 0.6.5 (2025-08-25)#
Features#
- Add support for NCI PBS Scheduler to the HPCExecutor (#358)
- Added sea ice sensitivity diagnostic. (#378)
- Added data required for regional historical changes diagnostics. (#379)
Improvements#
- Added a dataset ingestor for CMIP6 data that only uses the DRS information present in the filename. This greatly speeds up the process of ingesting a large number of files, as it does not require reading the file contents to extract the DRS information.
This method can be opted into by specifying cmip6_parser: "drs" in the REF configuration file. The default parser remains the complete parser, which reads the file contents to all the required metadata, but this will change in a future PR. (#366)
Improved Documentation#
- Address documentation review from the Model Benchmarking Task Team (#371)
climate-ref 0.6.4 (2025-08-04)#
Deprecations#
- The
--packageoption of theref celery start-workercommand has been deprecated and scheduled for removal. This functionality is now handled by the--provideroption which uses entry points declared in the provider packages. (#367)
Features#
- Use entrypoints to register provider plugins. (#360)
- Support celery workers to consume tasks for multiple providers (#367)
Improvements#
- Add additional dimensions to the example metric for testing purposes. (#372)
- Added a basic script to the CMIP6 data targetted by the current set of diagnostics for the Assessment Fast Track. (#373)
climate-ref 0.6.3 (2025-07-17)#
Improvements#
- Use a new URL for serving the reference data. This should now support older versions of TLS which may help some users. (#364)
climate-ref 0.6.2 (2025-07-09)#
Improvements#
- Implemented the parsl retry function (#341)
- Allow arbitrary environment variables to be used in paths in the configuration file. (#349)
- No longer automatically try to create the conda environment for a provider when running diagnostics. (#354)
- Use provider conda environments from the configured location when running tests. (#357)
- Remove the dependency on
ruamel.yamlinclimate-ref-core(#361) - Clarify that we don't technically support Windows at the moment, but it is possible to use WSL or a VM. (#362)
Improved Documentation#
- Add documentation for the CLI tool (#343)
climate-ref 0.6.1 (2025-05-28)#
Features#
- Implemented a HPCExecutor. It could let users run REF under HPC workflows by submitting batch jobs and compute diagnostics on the computer nodes. Only the slurm scheduler is supported now. (#305)
Bug Fixes#
- Remove keys with their value None from the output JSON for CMEC validation of PMP extratropical variability modes (#337)
Improved Documentation#
- Add Getting Started section for ingesting and solving (#342)
climate-ref 0.6.0 (2025-05-27)#
Breaking Changes#
- Updated the group by dimensions for the PMP diagnostics. This will cause duplicate runs to appear if an existing database is used. We recommend starting with a new database if using the next release. (#321)
Features#
- Implemented PMP ENSO metrics (#273)
- Added ESMValTool ENSO diagnostics. (#320)
- Add the creation of verbose debug logs.
These logs will be created in the
$REF_CONFIGURATION/logdirectory (or overriden via theconfig.paths.logsetting). (#323) - Data catalogs now only contain the latest version of a dataset. This will trigger new executions when a new version of a dataset is ingested.
Some additional datasets have been added to the obs4REF dataset registry.
These datasets should be fetched and reingested. (#330)
- Added a comparison of burntFractionAll to the ILAMB list of diagnostics (#332)
- Adds --diagnostic and --provider arguments to the ref solve command.
This allows users to subset a specific diagnostic or provider that they wish to run.
Multiple --diagnostic or --provider arguments can be used to specify multiple diagnostics or providers.
The diagnostic or provider slug must contain one of the filter values to be included in the calculations. (#338)
Improvements#
- Raise the ilamb3 version to 2025.5.20 and add remaining ILAMB/IOMB metrics. (#317)
- Adds Ocean Heat Content and snow cover datasets to the ilamb/iomb registry (#318)
- Updated the ESMValTool version to include updated recipes and diagnostics. (#325)
- Add obs4MIPs ERA-5 ta sample data as obs4REF. (#334)
- Enable more variables for the annual cycle diagnostics via PMP. (#335)
- Verify the checksum of downloaded datasets by default (#336)
Bug Fixes#
- Depth selects properly in mrsos, added regression data (#331)
climate-ref 0.5.5 (2025-05-21)#
Improvements#
- Added additional dimensions to the ILAMB and ESMValTool metric values. This includes additional information about the execution group that will be useful to end-users. (#308)
- Move the ILAMB datasets to S3 (#309)
- Clean ECS diagnostic (remove unused keys in ESMValTool recipes and avoid "cmip6" in diagnostic name) (#310)
- Clean TCR diagnostic (remove unused keys in ESMValTool recipes and avoid "cmip6" in diagnostic name) (#311)
Improved Documentation#
- Updated documentation to include more information about concepts within the REF. (#312)
climate-ref 0.5.4 (2025-05-19)#
Bug Fixes#
- Add additional dependencies to the
climate-ref-coreso that it is self-contained (#307)
climate-ref 0.5.3 (2025-05-19)#
Features#
- Diagnostic's have been split into two phases, executing which generates any outputs and then building a result object from the outputs. This split makes it easier to make modifications to how the results are translated into the CMEC outputs. (#303)
Improvements#
- Added automatic backup of SQLite database files before running migrations.
Backups are stored in a
backupsdirectory adjacent to the database file and are named with timestamps. The number of backups to retain can be configured via thedb.max_backupssetting in the database configuration, with a default of 5 backups. (#301) - Update to v0.6.0 of the sample data. (#302)
- Add a smoke test for the Celery deployment (#304)
- Add tests that the pypi releases are installable (#306)
Improved Documentation#
- Added page describing the required reference datasets (#298)
climate-ref 0.5.2 (2025-05-15)#
Bug Fixes#
- Fix missing dependency in migrations (#297)
Improved Documentation#
- Added documentation for configuration options (#296)
climate-ref 0.5.1 (2025-05-14)#
Features#
- Added an ESMValTool metric to compute climatologies and zonal mean profiles of cloud radiative effects. (#241)
- Add additional dimensions to the metrics produced by PMP.
Added
climate_ref_core.pycmec.metric.CMECMetric.prepend_dimensions(#275) - Ensure that selectors are always alphabetically sorted (#276)
- Add data model for supporting series of metric values. This allows diagnostic providers to supply a collection of climate_ref_core.metric_values.SeriesMetricValue extracted from an execution. (#278)
- The default executor (climate_ref.executor.LocalExecutor) uses a process pool to enable parallelism. An alternative climate_ref.executor.SynchronousExecutor is available for debugging purposes, which runs tasks synchronously in the main thread. (#286)
Improvements#
- Bumps the ilamb3 version to now contain all analysis modules and reformats its CMEC output bundle. (#262)
- Adds the ability to capture the output of an execution for regression testing. (#274)
- Update to v0.5.1 of the sample data. (#279)
- Update to v0.5.2 of the sample data. (#282)
- Add a CITATION.cff to the repository to make it easier to cite. (#283)
- Added the Assessment Fast Track-related services to the
docker-composestack alongside improved documentation for how to use the REF via docker containers. (#287) - Added support for ingesting multiple directories at once. This is useful for ingesting large datasets that are split into multiple directories or via glob patterns. An example of this is importing the monthly and fx datasets from an archive of CMIP6 data:
bash
ref datasets ingest --source-type cmip6 path_to_archive/CMIP6/*/*/*/*/*/*mon path_to_archive/CMIP6/*/*/*/*/*/fx (#291)
- Update the default log level to INFO from WARNING.
Added the -q option to decrease the log level to WARNING. (#292)
Bug Fixes#
- Resolves an issue that was blocking some PMP executions from completing. Any additional dimensions are now logged and ignored rather than causing the execution to fail. (#274)
- Sets the environment variable
FI_PROVIDER=tcpto use the TCP provider for libfabric (part of MPICH). The defaults were causing MPICH errors on some systems (namely macOS). This also removes the PMP provider's direct dependency on the source ofpcmdi_metric. (#281) - Support the use of empty metric bundles (#284)
- Reworked the lifetimes of the database transactions during the solve process. This is a fix for out of process executors where the transaction was not being committed until the end of a solve. (#288)
- Requery an Execution from the database when handling the result from the LocalExecutor. This ensures that the execution isn't stale and that the result is still valid. (#293)
climate-ref 0.5.0 (2025-05-03)#
Breaking Changes#
- Renamed packages to start with
climate_ref_and removedmetricsfrom the package name to avoid confusion. This changes the root name of the PyPi packages fromcmip_reftoclimate-refand will require updating yourrequirements.txt,pyproject.toml,setup.py, or other dependency management files to listclimate-refinstead ofcmip_ref. (#270) - Clarified the difference between a diagnostic and a metric. This caused significant refactoring of names of classes and functions throughout the codebase, as well as renaming of database tables.
| Package | Old Name | New Name |
|---|---|---|
| climate_ref_core.diagnostics | Metric | Diagnostic |
| climate_ref_core.diagnostics | MetricExecutionDefinition | ExecutionDefinition |
| climate_ref_core.diagnostics | MetricExecutionResult | ExecutionResult |
| climate_ref.models.execution | MetricExecutionResultß | Execution |
| climate_ref.models.execution | MetricExecutionGroup | ExecutionGroup |
| climate_ref.models.execution | ResultOutput | ExecutionOutput |
| climate_ref_core.datasets | MetricDataset | ExecutionDatasetCollection |
| climate_ref_core.solver | MetricSolver | ExecutionSolver |
| climate_ref_core.providers | MetricsProvider | DiagnosticProvider |
| climate_ref_core.providers | CommandLineMetricsProvider | CommandLineDiagnosticProvider |
| climate_ref_core.providers | CondaMetricsProvider | CondaDiagnosticProvider |
| climate_ref.config | MetricsProviderConfig | DiagnosticProviderConfig |
This removes any previous database migrations and replaces them with a new clean migration. If you have an existing database, you will need to delete it and re-create it. (#271)
cmip_ref 0.4.1 (2025-05-02)#
Breaking Changes#
- Removed unnecessary prefixes in the metric slugs. This will cause duplicate results to be generated so we recommend starting with a clean REF installation. (#263)
Features#
- Added PMP's annual cycle metrics (#221)
- Add a
facetsattribute to a metric. This attribute is used to define the facets of the values that the metric produces. (#255) - Added a diagnostic to calculate climate variables at global warming levels. (#257)
- Support multiple sets of data requirements (#266)
Bug Fixes#
- Retry downloads if they fail (#267)
- PMP annual cycle output JSON tranformed to be more comply with CMEC (#268)
Improved Documentation#
- Add deprecation notices to PyPi package README's (#269)
cmip_ref 0.4.0 (2025-04-29)#
Breaking Changes#
- Removed
climate_ref.solver.MetricSolver.solve_metric_executionsin preference for a standalone functionclimate_ref.solver.solve_metric_executionswith identical functionality. (#229) - Updated the algorithm to generate the unique identifier for a Metric Execution Group. This will cause duplicate entries in the database if the old identifier was used. We recommend removing your existing database and starting fresh. (#233)
- Removed the implicit treatment of the deepest dimension. The change will cause a validation error if the deepest dimension in the
RESULTSis a dictionary. (#246) - Ensure that the order of the source dataset types in the MetricExecutionGroup dataset key are stable (#248)
Deprecations#
- Removes support for Python 3.10. The minimum and default supported Python version is now 3.11. (#226)
Features#
- Add the basic framework for enforcing a controlled vocabulary for the results in a CMEC metrics bundle. This is still in the prototype stage and is not yet integrated into post-metric execution processing. (#183)
- Scalar values from the metrics are now stored in the database if they pass validation. The controlled vocabulary for these metrics is still under development. (#185)
- Added Zero Emission Commitment (ZEC) metric to the ESMValTool metrics package. (#204)
- Added Transient Climate Response to Cumulative CO2 Emissions (TCRE) metric to the ESMValTool metrics package. (#208)
- Add
ref datasets fetch-obs4ref-dataCLI command to fetch datasets that are in the process of being published to obs4MIPs and are appropriately licensed for use within the REF. The CLI command fetches the datasets and writes them to a local directory. These datasets can then be ingested into the REF as obs4MIPs datasets. (#219) - Enabled metric providers to register registries of datasets for download.
This unifies the fetching of datasets across the REF via the
ref datasets fetch-dataCLI command. Added registries for the datasets that haven't been published to obs4MIPs yet (obs4REF) as well as PMP annual cycle datasets. (#227) - Capture log output for each execution and display via
ref executions inspect. (#232) - Added the option to install development versions of metrics packages. (#236)
- Added seasonal cycle and time series of sea ice area metrics. (#239)
- The unique group identifiers for a MetricExecutionGroup are now tracked in the database. These values are used for presentation. (#248)
- Added a new dataset source type to track PMP climatology data (#253)
Improvements#
- Refactored
MetricSolver.solve_metric_executionsto be a standalone function for easier testing. Also added some additional integration tests for the Extratropical Modes of Variability metric from PMP. (#229) - The configuration paths are now all resolved to absolute paths (#230)
- Verified support for PostgreSQL database backends (#231)
- Updated the ESMValTool metric and output bundles. (#235)
- Update to v0.5.0 of the sample data (#264)
Bug Fixes#
- Relax some of the requirements for the availability of metadata in CMIP6 datasets. (#245)
- Added a missing migration and tests to ensure that the migration are up to date. (#247)
- Fixed how the path to ESMValTool outputs was determined, and added support for outputs in nested directories. (#250)
- Marked failing tests as xfail as a temporary solution. (#259)
- Fetch ESMValTool reference data in the integration test suite (#265)
Improved Documentation#
- Now following SPEC-0000 for dependency support windows. Support for Python versions will be dropped after 3 years and support for key scientific libraries will be dropped after 2 years. (#226)
- Migrate from the use of ‘AR7 Fast Track’ to ‘Assessment Fast Track’ in response to the CMIP Panel decision to change the name of the CMIP7 fast track. (#251)
Trivial/Internal Changes#
cmip_ref 0.3.1 (2025-03-28)#
Trivial/Internal Changes#
cmip_ref 0.3.0 (2025-03-28)#
Breaking Changes#
- We changed the
refCommand Line Interface to make the distinction between execution groups and individual executions clear. A metric execution is the evaluation of a specific metric for a specific set of input datasets. We group together all executions for the same set of input datasets which are re-run because the metric or the input datasets were updated or because a metric execution failed. For showing results, it is more useful to think in terms of execution groups. In particular, theref executions listcommand was re-named toref executions list-groups. (#165)
Features#
- Support ingesting obs4MIPs datasets into the REF (#113)
- Add extratropical modes of variability analysis using PMP (#115)
- Added management of conda environments for metrics package providers.
Several new commands are available for working with providers:
- ref providers list - List the available providers
- ref providers create-env - Create conda environments for providers
(#117) - Added CMECMetric.create_template method to create an empty CMEC metric bundle. (#123) - Outputs from a metric execution and their associated metadata are now tracked in the database. This includes HTML, plots and data outputs.
Metric providers can register outputs via the CMEC output bundle.
These outputs are then ingested into the database if the execution was successful. (#125)
- Build and publish container images to Github Container Registry (#156)
- Enable more variability modes for PMP modes of variability metrics (#173)
- Add a --timeout option to the solve cli command.
This enables the user to set a maximum time for the solver to run. (#186)
Improvements#
- Cleanup of ilamb3 interface code, enabling IOMB comparisons. (#124)
- Migrate the PMP provider to use a REF-managed conda environment.
For non-MacOS users, this should be created automatically. MacOS users will need to create the environment using the following command:
bash
MAMBA_PLATFORM=osx-64 uv run ref providers create-env --provider pmp (#127)
- Fixed issue with mypy not being run across the celery package (#128)
- Added the
fetch-ref-datamake command to download reference data while it's not in obs4mips, yet. (#155) - Improvements:
- Drop the metric plugin version number in the environment name because the environment may not change between releases
- Avoid calling micromamba update as this may not work for everyone
- Print out the location of environments even when they are not installed
- Do not mention conda as a requirement on the Hackathon page
(#160)
- Add activity and institute to ESMValTool recipes to allow running with models
and experiments that are not in the CMIP6 controlled vocabulary. (#166)
- Add an integration test for the CMIP7 AFT metric providers.
This will be run nightly as part of the Climate-REF CI pipeline. (#187)
- Improved the error message when running ref datasets list with the --column argument.
If a column is specified that is not available, the error message now only mentions
the invalid column name(s) and shows a list of available columns. (#203)
- Do not list duplicate entries in dataframes shown from the command line. (#210)
Bug Fixes#
- Remove example metric for ILAMB (#121)
- Removed the "SCHEMA" attribute from the CMEC metric bundle as it is not part of the EMDS specification and unused. (#123)
- Fixed the validation error when 'attributes' value is a dict (#133)
- Fixed PMP's modes of variablity PDO metrics driver to use obs4MIP-complying reference dataset. Also update PMP's version to 3.9, which include turning off direct usage of conda as a part of the driver (to capture provenance info) (#154)
- Enforce the use of relative paths when copying files after an execution. This resolves an issue where files were not being copied to the correct location causing failures in PMP. (#170)
- If no obs4mips-compliant reference dataset is found in specified directory, give a meaningful error message. (#174)
- Fixed the behaviour of FacetFilter with
keep=Falseso all facets need to match before excluding a file. (#209)
Improved Documentation#
- Renamed CMIP-REF to Climate-REF (#119)
- Add a landing page for hackathon attendees (#120)
- Fix the incorrect capitalisation of GitHub organisation (#122)
- Updated the getting started documentation. (#126)
- Update the roadmap to reflect progress as of 2025/03/10 (#134)
- Clarified language and other small fixes in the documentation. (#178)
Trivial/Internal Changes#
cmip_ref 0.2.0 (2025-03-01)#
Breaking Changes#
- Refactor
climate_ref.envmodule toclimate_ref_core.env(#60) - Removed
climate_ref.executor.ExecutorManagerin preference to loading an executor using a fully qualified package name.
This allows the user to specify a custom executor as configuration
without needing any change to the REF codebase. (#77)
- Renamed the $.paths.tmp in the configuration to $.paths.scratch to better reflect its purpose.
This requires a change to the configuration file if you have a custom configuration. (#89)
- The REF now uses absolute paths throughout the application.
This removes the need for a config.paths.data directory and the config.paths.allow_out_of_tree_datasets configuration option.
This will enable more flexibility about where input datasets are ingested from.
Using absolute paths everywhere does add a requirement that datasets are available via the same paths for all nodes/container that may run the REF. (#100)
- An Executor now supports only the asynchronous processing of tasks.
A result is now not returned from the run_metric method,
but instead optionally updated in the database.
The run_metric method also now requires a provider argument to be passed in. (#104)
Features#
- Adds a
cmip-ref-celerypackage to the REF that provides a Celery task queue.
Celery is a distributed task queue that allows you to run tasks asynchronously.
This package will be used as a test bed for running the REF in a distributed environment,
as it can be deployed locally using docker containers. (#60)
- Add metric_providers and executor sections to the configuration which loads the metric provider and executor using a fully qualified package name. (#77)
- Implemented Pydantic data models to validate and serialize CMEC metric and output bundles. (#84)
- Add the climate_ref_celery CLI commands to the ref CLI tool.
These commands should be available when the climate_ref_celery package is installed.
The commands should be available in the ref CLI tool as ref celery .... (#86)
- Add fetch-sample-data to the CLI under the datasets command.
bash
ref datasets fetch-sample-data --version v0.3.0 --force-cleanup (#96)
- Add a Celery-based executor
to enable asynchronous processing of tasks. (#104)
- Add ref executions list and ref executions inspect CLI commands for interacting with metric executions. (#108)
Improvements#
- Move ILAMB/IOMB reference data initialization to a registry-dependent script. (#83)
- ILAMB gpp metrics added with full html output and plots. (#88)
- Saner error messages for configuration errors (#89)
- Centralised the declaration of environment variable overrides of configuration values.
Renamed the REF_OUTPUT_ROOT environment variable to REF_RESULTS_ROOT to better reflect its purpose.
It was previously unused. (#92)
- Sample data is now copied to the test/test-data/sample-data instead of symlinked.
This makes it easier to use the sample data with remote executors as the data is now self-contained without any links to other parts of the file system. (#96) - Integrated the pycmec validation models into ref core and metric packages (#99) - Added ILAMB relationship analysis to the current metrics and flexibility to define new metrics in ILAMB via a yaml file. (#101) - Sped up the test suite execution (#103)
Improved Documentation#
- Added an excerpt from the architecture design document (#87)
- Adds a roadmap to the documentation (#98)
Trivial/Internal Changes#
cmip_ref 0.1.6 (2025-02-03)#
Features#
- Added Equilibrium Climate Sensitivity (ECS) to the ESMValTool metrics package. (#51)
- Added Transient Climate Response (TCS) to the ESMValTool metrics package. (#62)
- Added the possibility to request datasets with complete and overlapping timeranges. (#64)
- Added a constraint for selecting supplementary variables, e.g. cell measures or ancillary variables. (#65)
- Added a sample metric to the ilamb metrics package. (#66)
- Added a sample metric to the PMP metrics package. (#72)
-
- Added the standard ILAMB bias analysis as a metric. (#74)
Bug Fixes#
-
- Added overlooked code to fully integrate ilamb into ref. (#73)
- Correct the expected configuration name to
ref.tomlas per the documentation. (#82)
Improved Documentation#
- Update the package name in the changelog.
This will simplify the release process by fixing the extraction of changelog entries. (#61)
Trivial/Internal Changes#
cmip_ref 0.1.5 (2025-01-13)#
Trivial/Internal Changes#
cmip_ref 0.1.4 (2025-01-13)#
Breaking Changes#
- Adds an
ingestCLI command to ingest a new set of data into the database.
This breaks a previous migration as alembic's render_as_batch attribute should have been set
to support targeting sqlite. (#14)
- Renames ref ingest to ref datasets ingest (#30)
- Prepend package names with cmip_ to avoid conflicting with an existing PyPI package.
This is a breaking change because it changes the package name and all imports.
All package names will now begin with cmip_ref. (#53)
Features#
- Migrate to use UV workspaces to support multiple packages in the same repository.
Adds a
climate-ref-examplepackage that will be used to demonstrate the integration of a metric package into the REF. (#2) - Adds the placeholder concept of
Executor's which are responsible for running metrics in different environments. (#4) - Adds the concept of MetricProvider's and Metrics to the core. These represent the functionality that metric providers must implement in order to be part of the REF. The implementation is still a work in progress and will be expanding in follow-up PRs. (#5)
- Add a collection of ESGF data that is required for test suite.
Package developers should run make fetch-test-data to download the required data for the test suite. (#6)
- Adds the ref package with a basic CLI interface that will allow for users to interact with the database of jobs. (#8)
- Add SqlAlchemy as an ORM for the database alongside alembic for managing database migrations. (#11)
- Added a DataRequirement class to declare the requirements for a metric.
This provides the ability to:
- filter a data catalog
- group datasets together to be used in a metric calculation
- declare constraints on the data that is required for a metric calculation
(#15)
- Add a placeholder iterative metric solving scheme (#16)
- Extract a data catalog from the database to list the currently ingested datasets (#24)
- Translated selected groups of datasets into MetricDatasets.
Each MetricDataset contains all of the dataset's needed for a given execution of a metric.
Added a slug to the MetricDataset to uniquely identify the execution
and make it easier to identify the execution in the logs. (#29)
- Adds ref datasets list command to list ingested datasets (#30)
- Add database structures to represent a metric execution.
We record previous executions of a metric to minimise the number of times we need to run metrics. (#31)
- Added option to skip any datasets that fail validation and to specify the number of cores to
use when ingesting datasets.
This behaviour can be opted in using the --skip-invalid and --n-jobs options respectively. (#36)
- Track datasets that were used for different metric executions (#39)
- Added an example ESMValTool metric. (#40)
- Support the option for different assumptions about the root paths between executors and the ref CLI.
Where possible path fragments are stored in the database instead of complete paths. This allows the ability to move the data folders without needing to update the database. (#46)
Improvements#
- Add a bump, release and deploy flow for automating the release procedures (#20)
- Migrate test data into standalone Climate-REF/ref-sample-data repository.
The sample data will be downloaded by the test suite automatically into tests/test-data/sample-data,
or manually by running make fetch-test-data. (#49)
Bug Fixes#
- Adds
versionfield to theinstance_idfield for CMIP6 datasets (#35) - Handle missing branch times. Fixes #38. (#42)
- Move alembic configuration and migrations to
cmip_refpackage so that they can be included in the distribution. (#54)
Improved Documentation#
- Deployed documentation to https://climate-ref.readthedocs.io/en/latest/ (#16)
- General documentation cleanup.
Added notebook describing the process of executing a notebook locally (#19) - Add Apache licence to the codebase (#21) - Improved developer documentation. (#47)