Skip to content

Configuration#

The REF uses a tiered configuration model, where options can be sourced from different places. Then configuration is loaded from a .toml file which overrides any default values. However, some configuration variables can be overridden at runtime using environment variables, which always take precedence over any other configuration values set by default or found in a .toml file.

The default values for these environment variables are generally suitable, but if you require updating these values we recommend the use of a .env file to make the changes easier to reproduce in future.

Configuration File Discovery#

The REF will look for a configuration file in the following locations, taking the first one it finds:

  • ${REF_CONFIGURATION}/ref.toml
  • ~/.config/climate_ref/ref.toml (Linux)
  • $XDG_CONFIG_HOME/climate_ref/ref.toml (Linux)
  • ~/Library/Application Support/climate_ref/ref.toml (macOS)
  • %USERPROFILE%\AppData\Local\climate_ref\ref.toml (Windows)

If no configuration file is found, the REF will use the default configuration.

This directory may contain significant amounts of data, so for HPC systems it is recommended to set the REF_CONFIGURATION environment variable to a directory on a scratch filesystem.

This default configuration is equivalent to the following:

log_level = "INFO"

[paths]
log = "${REF_CONFIGURATION}/log"
scratch = "${REF_CONFIGURATION}/scratch"
software = "${REF_CONFIGURATION}/software"
results = "${REF_CONFIGURATION}/results"

[db]
database_url = "sqlite:///${REF_CONFIGURATION}/db/climate_ref.db"
run_migrations = true

[executor]
executor = "climate_ref.executor.LocalExecutor"
measure_resources = true

[executor.config]

[[diagnostic_providers]]
provider = "climate_ref_esmvaltool:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_ilamb:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_pmp:provider"

[diagnostic_providers.config]

Managing configuration from the CLI#

Use ref config init to create a supported starter ref.toml in REF_CONFIGURATION. The command creates parent directories and refuses to overwrite an existing file unless you pass --force.

Individual scalar values can be inspected or changed with dotted keys:

ref config get paths.scratch
ref config set log_level DEBUG
ref config unset log_level

ref config get prints the effective value the REF will use at runtime, so environment variables such as REF_DATABASE_URL take precedence over values in ref.toml. When an environment variable shadows a requested key, the CLI keeps stdout script-friendly and writes the notice to stderr.

Run ref config validate after hand-editing the file. For CI or editor integrations, use ref config validate --format json and rely on the exit code: 0 means valid, 1 means invalid.

ref config set and ref config unset rewrite ref.toml from the parsed configuration model. This is convenient for simple scalar changes, but it does not preserve comments or custom key ordering in a hand-edited file. Edit structured values such as diagnostic_providers and executor.config directly in TOML.

Additional Environment Variables#

Environment variables are used to control some aspects of the framework outside of the configuration file.

REF_CELERY_ROUTES#

Path to a TOML routing table for the Celery executor. The table maps diagnostics to queue names, so that an execution lands on a queue such as esmvaltool-large instead of the bare provider queue. Differently sized worker pools can then consume the queues independently.

default = "{provider}"

[esmvaltool]
default = "esmvaltool-medium"
rules = [
  { match = "portrait-*", queue = "esmvaltool-large" },
  { match = "sea-ice-basic", queue = "esmvaltool-small" },
]

[ilamb]
default = "ilamb-small"

Rules are matched against the diagnostic slug in order, first match wins. Patterns support exact strings and glob wildcards. Queue names are templates in which {provider} expands to the provider slug. A provider default applies when no rule matches, and the top-level default applies when the provider has no entry. With no default and no match, the bare provider queue is used, equivalent to a default of "{provider}".

If this is not set, every execution uses the bare provider queue. A malformed file fails the solve at startup rather than silently misrouting jobs. Workers consuming a size-specific queue must be started with that queue name explicitly, for example ref-celery start-worker --provider climate_ref_esmvaltool -- --queues=esmvaltool-large.

REF_DATASET_CACHE_DIR#

Path where any datasets that are fetched via the ref datasets fetch-data command are stored. This directory will be several GB in size, so it is recommended to set this to a directory on a scratch filesystem rather than a directory on your home filesystem.

This is used to cache the datasets so that they are not downloaded multiple times. It is not recommended to ingest datasets from this directory (see --output-dir argument for ref datasets fetch-data).

This defaults to the following locations:

  • ~/Library/Caches/climate_ref (MacOS)
  • ~/.cache/climate_ref or the value of the $XDG_CACHE_HOME/climate_ref environment variable, if defined. (Linux)
  • %USERPROFILE%\AppData\Local\climate_ref\Cache (Windows)

REF_DATASET_FETCH_WORKERS#

Maximum number of files fetched concurrently by ref datasets fetch-data. This must be a positive integer and defaults to 4. Set it to 1 to fetch files sequentially, or reduce it when network bandwidth or the remote server limits concurrent transfers.

REF_EXECUTOR_MEASURE_RESOURCES#

Whether to record what each execution costs in wall time, CPU time and peak memory. This defaults to true and accepts true/false, 1/0, yes/no and on/off.

The measurement is what ref executions resources aggregates, so turning it off leaves that command without new samples. Set it to false where the sampling thread is unwelcome, for example in a memory-constrained container or a profiling run.

REF_TEST_DATA_DIR#

Override the location of the test data directory. If this is not set, the test data directory will be inferred from the location of the test suite.

If this is set, then the sample data won't be updated.

REF_TEST_OUTPUT#

Path where the test output is stored. This is used to store the output of the tests that are run in the test suite for later inspection.

Celery environment variables#

These apply when the Celery executor is in use. The full set of tuning knobs is listed in the Helm chart README, and the ones below govern the wire format.

Tasks and results are encoded as JSON (ref-json). What a process sends is fixed and cannot be changed by configuration, so no deployment can put pickle back on the wire.

CELERY_TASK_COMPRESSION and CELERY_RESULT_COMPRESSION#

Codec used to compress task and result message bodies, defaulting to gzip. Bodies are dominated by the datasets DataFrame carried in the execution definition and shrink by roughly 80%, which cuts broker memory on a full solve. Set either to an empty string to send uncompressed bodies.

CELERY_ACCEPT_CONTENT#

Comma separated content types a worker will decode, defaulting to json,ref-json. This only widens what is accepted. It will not revive messages queued by a release that still used pickle because those carry a pickled diagnostic and provider that current releases no longer define. Purge the queues when upgrading from such a release and re-solve.

Grey list#

The REF maintains a grey list: datasets that are known to cause problems for particular diagnostics and should be excluded from solving until the underlying issues are resolved. The grey list is a YAML file listing facets to exclude per provider, diagnostic and source type.

!!! note "Naming"

The configuration values below are currently named `ignore_datasets_*` for historical reasons.
They will be renamed to `grey_list_*` in a future release.
The old names will continue to work for a deprecation period.

The grey list is resolved from the first of three layers that is available:

  1. ignore_datasets_file, if it is set.
  2. A copy refreshed from ignore_datasets_url into the local cache.
  3. The copy shipped inside the climate_ref package.

The third layer is always available, so a solve never depends on the network or on a writable filesystem.

Two configuration values control this behaviour:

  • ignore_datasets_file (env REF_IGNORE_DATASETS_FILE): a path to a grey list you manage yourself. Leave it unset to use the packaged copy. Setting it also disables fetching, because an explicit file is yours to manage.
  • ignore_datasets_url (env REF_IGNORE_DATASETS_URL): the URL the grey list is refreshed from. It defaults to the copy served from the main branch of the Climate-REF repository.

Refreshing happens lazily during solving, not while the configuration is loaded, so read-only commands such as ref providers list never perform network I/O. When a solve runs, the cached file is refreshed only if it is missing or older than six hours, so at most one download happens per six-hour window.

Refreshing is best effort. An unreachable network, an unwritable cache directory, and an HTTP error are all non-fatal. The solve logs a warning and falls back to the cached copy if there is one, and to the packaged copy otherwise. Each provider logs which layer it read the grey list from at debug level.

A cached copy that has not been refreshed for 30 days is ignored in favour of the packaged copy. Without that bound, a cache left behind by an older release would shadow the newer packaged copy indefinitely on a host that can never reach the network. On such a host the cache can still shadow a newer packaged copy for up to 30 days after an upgrade. Delete the cached file to read the packaged copy immediately.

Offline and air-gapped deployments#

Nothing needs to be configured. The grey list and the dimensions controlled vocabulary are both read straight out of the installed packages, so the REF runs with no outbound network access and on a read-only filesystem.

On a host with no route to the internet, set REF_IGNORE_DATASETS_URL= (an empty string) to skip the refresh attempt. This avoids waiting for the request to time out on every solve. It changes nothing else, since a failed refresh already falls back to the packaged copy.

To pin the grey list to a version you control, point REF_IGNORE_DATASETS_FILE at your own copy.

Configuration Options#

Top-level#

Configuration that is used by the REF

cmip6_parser#

Parser to use for CMIP6 datasets

This can be either drs or complete.

  • drs: Use the DRS parser, which parses the dataset based on the DRS naming conventions.
  • complete: Use the complete parser, which parses the dataset based on all available metadata.

Default: 'complete'

Type: Literal

Environment Variable: REF_CMIP6_PARSER

cmip7_parser#

Parser to use for CMIP7 datasets

This can be either drs or complete.

  • drs: Use the DRS parser, which parses the dataset based on the DRS naming conventions.
  • complete: Use the complete parser, which parses the dataset based on all available metadata.

Default: 'complete'

Type: Literal

Environment Variable: REF_CMIP7_PARSER

ignore_datasets_file#

Path to a file containing the grey list

This file is a YAML file that contains a list of facets to ignore per diagnostic.

The format is: 

provider:
  diagnostic:
    source_type:
      - facet: value
      - another_facet: [another_value1, another_value2]

Leave this unset to use the grey list shipped inside the climate_ref package, refreshed from ignore_datasets_url when that is possible. Setting it pins the grey list to a file you manage, and disables fetching.

Type: Path | None

Environment Variable: REF_IGNORE_DATASETS_FILE

ignore_datasets_url#

URL to refresh the grey list from at solve time.

The download happens during solving only, at most once every 6 hours, and never during configuration loading. A failed download is not an error, since the copy shipped in the package is used instead.

Set to an empty string (e.g. REF_IGNORE_DATASETS_URL=) to skip the attempt entirely, which avoids the request timeout on a host with no route to the internet.

Default: 'https://raw.githubusercontent.com/Climate-REF/climate-ref/refs/heads/main/default_ignore_datasets.yaml'

Type: str

Environment Variable: REF_IGNORE_DATASETS_URL

log_format#

Format of the log messages that are displayed by the REF via the CLI

Examples of the formatting options are available in the loguru documentation.

Default: '{time:YYYY-MM-DD HH:mm:ss.SSS Z} | {level: <8} | {name} - {message}'

Type: str

Environment Variable: REF_LOG_FORMAT

log_level#

Log level of messages that are displayed by the REF via the CLI

This value is overridden if a value is specified via the CLI.

Default: 'INFO'

Type: str

n_jobs#

Number of worker processes used to parse dataset files

1 parses serially, -1 uses all available CPUs, and any other positive value is used as-is.

Parsing opens netCDF files, so it is bound by (highly parallel) filesystem latency rather than by CPU. This applies both to ref datasets ingest (where --n-jobs overrides it) and to the finalisation of datasets that the solver performs when they were ingested with the drs parser.

Default: 1

Type: int

Environment Variable: REF_N_JOBS

db#

Database configuration

We support SQLite and PostgreSQL databases. The default is to use SQLite, which is a file-based database that is stored in the REF_CONFIGURATION directory. This is a good option for testing and development, but not recommended for production use.

For production use, we recommend using PostgreSQL.

database_url#

Database URL that describes the connection to the database.

Defaults to sqlite:///{config.paths.db}/climate_ref.db. This configuration value will be overridden by the REF_DATABASE_URL environment variable.

Schemas

The following schemas are supported: 

postgresql://USER:PASSWORD@HOST:PORT/NAME

sqlite:///RELATIVE_PATH or sqlite:////ABS_PATH or sqlite:///:memory:

Default: 'sqlite:///$REF_CONFIGURATION/db/climate_ref.db'

Type: str

Environment Variable: REF_DATABASE_URL

max_backups#

Maximum number of database backups to keep.

When running migrations for on-disk SQLite databases, a backup of the database is created. This setting controls how many of these backups are retained. The oldest backups are automatically removed when this limit is exceeded.

Default: 5

Type: int

Environment Variable: REF_MAX_BACKUPS

run_migrations#

No description provided.

Default: True

Type: bool

diagnostic_providers#

Defining the diagnostic providers used by the REF.

Each diagnostic provider is a package that contains the logic for running a specific set of diagnostics. This configuration determines which diagnostic providers are loaded and used when solving.

Multiple diagnostic providers can be specified as shown in the example below.

[[diagnostic_providers]]
provider = "climate_ref_esmvaltool:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_ilamb:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_pmp:provider"

[diagnostic_providers.config]
config#

Additional configuration for the diagnostic provider.

See the documentation for the diagnostic package for the available configuration options.

Default: {}

Type: dict

provider#

Package that contains the diagnostic provider

This should be the fully qualified name of the diagnostic provider.

Default: 'climate_ref_ilamb:provider'

Type: str

executor#

Configuration to define the executor to use for running diagnostics

config#

Additional configuration for the executor.

See the documentation for the executor for the available configuration options. These options will be passed to the executor class when it is created.

Default: {}

Type: dict

executor#

Executor class to use for running diagnostics

This should be the fully qualified name of the executor class (e.g. climate_ref.executor.LocalExecutor). The default is to use the local executor which runs the executions locally, in-parallel using a process pool.

This class will be used for all executions of diagnostics.

Default: 'climate_ref.executor.LocalExecutor'

Type: str

Environment Variable: REF_EXECUTOR

measure_resources#

Whether to record what each execution costs in wall time, CPU time and peak memory.

The measurement is what ref executions resources aggregates, so turning it off leaves that command without new samples. Set it to false where the sampling thread is unwelcome, for example in a memory-constrained container or a profiling run.

Default: True

Type: bool

Environment Variable: REF_EXECUTOR_MEASURE_RESOURCES

native_store#

Configuration for the content-addressed native-bundle object store.

The native store holds the curated native outputs (NetCDF, PNG, ...) produced by each test case, keyed by their sha256 digest. Read operations (has, fetch) are always anonymous and credential-free. Write operations are gated to the mint verb only.

bucket#

Name of the writable (Cloudflare R2) bucket.

Non-secret routing config, consumed only by the mint verb. Set REF_NATIVE_STORE_BUCKET to override.

Write credentials are not stored here: the access-key id and secret-access-key are read from REF_NATIVE_STORE_ACCESS_KEY_ID / REF_NATIVE_STORE_SECRET_ACCESS_KEY (falling back to boto3's default credential chain) at upload time only, so secrets never land in a serialised config.

Default: 'ref-baselines-public'

Type: str

Environment Variable: REF_NATIVE_STORE_BUCKET

cache_dir#

Local pooch cache directory for downloaded native blobs.

Defaults via :func:~climate_ref_core.dataset_registry.resolve_cache_dir, so the REF_DATASET_CACHE_DIR environment variable applies here too.

Default: PosixPath('/home/docs/.cache/climate_ref/native-baselines')

Type: Path

Environment Variable: REF_NATIVE_STORE_CACHE_DIR

s3_endpoint_url#

S3 API endpoint for the writable (Cloudflare R2) backend, without the bucket.

Non-secret routing config, consumed only by the mint verb. Defaults to the production Climate-REF R2 account endpoint (default jurisdiction — note there is no .eu in the host). Anonymous read (fetch / has) uses :attr:url instead and never touches this. Set REF_NATIVE_STORE_S3_ENDPOINT_URL to override (e.g. a staging account).

Default: 'https://2aa5172b2bba093c516027d6fa13cdc8.r2.cloudflarestorage.com'

Type: str

Environment Variable: REF_NATIVE_STORE_S3_ENDPOINT_URL

url#

Base URL of the native-bundle object store.

Blobs are served at {url}/{digest}. Defaults to the production Climate-REF baselines endpoint.

Set REF_NATIVE_STORE_URL to a local file:///path/to/dir (or a plain filesystem path) for offline development and testing.

Default: 'https://baselines.climate-ref.org'

Type: str

Environment Variable: REF_NATIVE_STORE_URL

paths#

Common paths used by the REF application

Warning

These paths must be common across all systems that the REF is being run. Generally, this means that they should be mounted in the same location on all systems.

If any of these paths are specified as relative paths, they will be resolved to absolute paths. These absolute paths will be used for all operations in the REF.

dimensions_cv#

Path to a file containing the controlled vocabulary for the dimensions in a CMEC diagnostics bundle

Leave this unset to use the controlled vocabulary for the CMIP7 Assessment Fast Track diagnostics, which is shipped inside the climate_ref_core package. That copy is read straight out of the installed package, so it needs no network access and no writable filesystem.

This controlled vocabulary is used to validate the dimensions in the diagnostics bundle. If custom diagnostics are implemented, point this at a copy that has been extended with any new dimensions.

Type: Path | None

Environment Variable: REF_DIMENSIONS_CV_PATH

log#

Directory to store log files from the compute engine

This is not currently used by the REF, but is included for future use.

Default: '$REF_CONFIGURATION/log'

Type: Path

Environment Variable: REF_LOG_ROOT

results#

Path to store the executions

Default: '$REF_CONFIGURATION/results'

Type: Path

Environment Variable: REF_RESULTS_ROOT

scratch#

Shared scratch space for the REF.

This directory is used to write the intermediate executions of a diagnostic execution. After the diagnostic has been run, the executions will be copied to the executions directory.

This directory must be accessible by all the diagnostic services that are used to run the diagnostics, but does not need to be mounted in the same location on all the diagnostic services.

Default: '$REF_CONFIGURATION/scratch'

Type: Path

Environment Variable: REF_SCRATCH_ROOT

software#

Shared software space for the REF.

This directory is used to store software environments.

This directory must be accessible by all the diagnostic services that are used to run the diagnostics, and should be mounted in the same location on all the diagnostic services.

Default: '$REF_CONFIGURATION/software'

Type: Path

Environment Variable: REF_SOFTWARE_ROOT