Configuration#

The REF uses a tiered configuration model, where options can be sourced from different places. Then configuration is loaded from a .toml file which overrides any default values. However, some configuration variables can be overridden at runtime using environment variables, which always take precedence over any other configuration values set by default or found in a .toml file.

The default values for these environment variables are generally suitable, but if you require updating these values we recommend the use of a .env file to make the changes easier to reproduce in future.

Configuration File Discovery#

The REF will look for a configuration file in the following locations, taking the first one it finds:

${REF_CONFIGURATION}/ref.toml
~/.config/climate_ref/ref.toml (Linux)
$XDG_CONFIG_HOME/climate_ref/ref.toml (Linux)
~/Library/Application Support/climate_ref/ref.toml (macOS)
%USERPROFILE%\AppData\Local\climate_ref\ref.toml (Windows)

If no configuration file is found, the REF will use the default configuration.

This directory may contain significant amounts of data, so for HPC systems it is recommended to set the REF_CONFIGURATION environment variable to a directory on a scratch filesystem.

This default configuration is equivalent to the following:

log_level = "INFO"

[paths]
log = "${REF_CONFIGURATION}/log"
scratch = "${REF_CONFIGURATION}/scratch"
software = "${REF_CONFIGURATION}/software"
results = "${REF_CONFIGURATION}/results"
dimensions_cv = "${REF_INSTALLATION_DIR}/packages/climate-ref-core/src/climate_ref_core/pycmec/cv_cmip7_aft.yaml"

[db]
database_url = "sqlite:///${REF_CONFIGURATION}/db/climate_ref.db"
run_migrations = true

[executor]
executor = "climate_ref.executor.LocalExecutor"

[executor.config]

[[diagnostic_providers]]
provider = "climate_ref_esmvaltool:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_ilamb:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_pmp:provider"

[diagnostic_providers.config]

Managing configuration from the CLI#

Use ref config init to create a supported starter ref.toml in REF_CONFIGURATION. The command creates parent directories and refuses to overwrite an existing file unless you pass --force.

Individual scalar values can be inspected or changed with dotted keys:

ref config get paths.scratch
ref config set log_level DEBUG
ref config unset log_level

ref config get prints the effective value the REF will use at runtime, so environment variables such as REF_DATABASE_URL take precedence over values in ref.toml. When an environment variable shadows a requested key, the CLI keeps stdout script-friendly and writes the notice to stderr.

Run ref config validate after hand-editing the file. For CI or editor integrations, use ref config validate --format json and rely on the exit code: 0 means valid, 1 means invalid.

ref config set and ref config unset rewrite ref.toml from the parsed configuration model. This is convenient for simple scalar changes, but it does not preserve comments or custom key ordering in a hand-edited file. Edit structured values such as diagnostic_providers and executor.config directly in TOML.

Additional Environment Variables#

Environment variables are used to control some aspects of the framework outside of the configuration file.

`REF_DATASET_CACHE_DIR`#

Path where any datasets that are fetched via the ref datasets fetch-data command are stored. This directory will be several GB in size, so it is recommended to set this to a directory on a scratch filesystem rather than a directory on your home filesystem.

This is used to cache the datasets so that they are not downloaded multiple times. It is not recommended to ingest datasets from this directory (see --output-dir argument for ref datasets fetch-data).

This defaults to the following locations:

~/Library/Caches/climate_ref (MacOS)
~/.cache/climate_ref or the value of the $XDG_CACHE_HOME/climate_ref environment variable, if defined. (Linux)
%USERPROFILE%\AppData\Local\climate_ref\Cache (Windows)

`REF_TEST_DATA_DIR`#

Override the location of the test data directory. If this is not set, the test data directory will be inferred from the location of the test suite.

If this is set, then the sample data won't be updated.

`REF_TEST_OUTPUT`#

Path where the test output is stored. This is used to store the output of the tests that are run in the test suite for later inspection.

Grey list#

The REF maintains a grey list: datasets that are known to cause problems for particular diagnostics and should be excluded from solving until the underlying issues are resolved. The grey list is a YAML file listing facets to exclude per provider, diagnostic and source type.

!!! note "Naming"

The configuration values below are currently named `ignore_datasets_*` for historical reasons.
They will be renamed to `grey_list_*` in a future release;
the old names will continue to work for a deprecation period.

Two configuration values control this behaviour:

ignore_datasets_file (env REF_IGNORE_DATASETS_FILE) — the path to the grey list file. It defaults to a location under the user cache directory and is coerced to a filesystem path.
ignore_datasets_url (env REF_IGNORE_DATASETS_URL) — the URL the grey list is fetched from. It defaults to the copy shipped in the Climate-REF repository.

The grey list is fetched lazily during solving, not while the configuration is loaded. Read-only commands such as ref providers list and ref datasets list never perform network I/O. When a solve runs, the cached file is refreshed from ignore_datasets_url only if it is missing or older than six hours, so at most one download happens per six-hour window.

If the download fails and a cached copy already exists, the cached copy is reused unchanged and a warning is logged. If the download fails and no cached copy exists, the solve fails with an error rather than silently running without the grey list protections.

Offline and air-gapped deployments#

On systems without outbound network access (for example an air-gapped HPC cluster), seed the grey list manually and disable fetching:

Copy default_ignore_datasets.yaml to a writable location on the target system.
Point ignore_datasets_file at that copy, for example REF_IGNORE_DATASETS_FILE=/shared/config/default_ignore_datasets.yaml.
Set REF_IGNORE_DATASETS_URL= (an empty string) to disable fetching entirely.

With an empty URL the solver never touches the network and uses whatever grey list already exists at ignore_datasets_file.

Configuration Options#

Top-level#

Configuration that is used by the REF

`cmip6_parser`#

Parser to use for CMIP6 datasets

This can be either drs or complete.

drs: Use the DRS parser, which parses the dataset based on the DRS naming conventions.
complete: Use the complete parser, which parses the dataset based on all available metadata.

Default: 'complete'

Type: Literal

Environment Variable: REF_CMIP6_PARSER

`cmip7_parser`#

Parser to use for CMIP7 datasets

This can be either drs or complete.

drs: Use the DRS parser, which parses the dataset based on the DRS naming conventions.
complete: Use the complete parser, which parses the dataset based on all available metadata.

Default: 'complete'

Type: Literal

Environment Variable: REF_CMIP7_PARSER

`ignore_datasets_file`#

Path to the file containing the ignore datasets

This file is a YAML file that contains a list of facets to ignore per diagnostic.

The format is:

provider:
  diagnostic:
    source_type:
      - facet: value
      - another_facet: [another_value1, another_value2]

Defaults to a path under the user cache directory. The file is fetched lazily from ignore_datasets_url at solve time if it is missing or older than 6 hours, so this location must be writable unless the file is seeded manually.

Default: PosixPath('/home/docs/.cache/climate_ref/default_ignore_datasets.yaml')

Type: Path

Environment Variable: REF_IGNORE_DATASETS_FILE

`ignore_datasets_url`#

URL to fetch the ignore datasets file from at solve time.

The download happens during solving only, at most once every 6 hours, and never during configuration loading.

Set to an empty string (e.g. REF_IGNORE_DATASETS_URL=) to disable fetching entirely; in that case the solver uses whatever file already exists at ignore_datasets_file, which supports offline and air-gapped deployments.

Default: 'https://raw.githubusercontent.com/Climate-REF/climate-ref/refs/heads/main/default_ignore_datasets.yaml'

Type: str

Environment Variable: REF_IGNORE_DATASETS_URL

`log_format`#

Format of the log messages that are displayed by the REF via the CLI

Examples of the formatting options are available in the loguru documentation.

Default: '{time:YYYY-MM-DD HH:mm:ss.SSS Z} | {level: <8} | {name} - {message}'

Type: str

Environment Variable: REF_LOG_FORMAT

`log_level`#

Log level of messages that are displayed by the REF via the CLI

This value is overridden if a value is specified via the CLI.

Default: 'INFO'

Type: str

`db`#

Database configuration

We support SQLite and PostgreSQL databases. The default is to use SQLite, which is a file-based database that is stored in the REF_CONFIGURATION directory. This is a good option for testing and development, but not recommended for production use.

For production use, we recommend using PostgreSQL.

`database_url`#

Database URL that describes the connection to the database.

Defaults to sqlite:///{config.paths.db}/climate_ref.db. This configuration value will be overridden by the REF_DATABASE_URL environment variable.

Schemas

The following schemas are supported:

postgresql://USER:PASSWORD@HOST:PORT/NAME

sqlite:///RELATIVE_PATH or sqlite:////ABS_PATH or sqlite:///:memory:

Default: 'sqlite:///$REF_CONFIGURATION/db/climate_ref.db'

Type: str

Environment Variable: REF_DATABASE_URL

`max_backups`#

Maximum number of database backups to keep.

When running migrations for on-disk SQLite databases, a backup of the database is created. This setting controls how many of these backups are retained. The oldest backups are automatically removed when this limit is exceeded.

Default: 5

Type: int

Environment Variable: REF_MAX_BACKUPS

`run_migrations`#

No description provided.

Default: True

Type: bool

`diagnostic_providers`#

Defining the diagnostic providers used by the REF.

Each diagnostic provider is a package that contains the logic for running a specific set of diagnostics. This configuration determines which diagnostic providers are loaded and used when solving.

Multiple diagnostic providers can be specified as shown in the example below.

[[diagnostic_providers]]
provider = "climate_ref_esmvaltool:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_ilamb:provider"

[diagnostic_providers.config]

[[diagnostic_providers]]
provider = "climate_ref_pmp:provider"

[diagnostic_providers.config]

`config`#

Additional configuration for the diagnostic provider.

See the documentation for the diagnostic package for the available configuration options.

Default: {}

Type: dict

`provider`#

Package that contains the diagnostic provider

This should be the fully qualified name of the diagnostic provider.

Default: 'climate_ref_esmvaltool:provider'

Type: str

`executor`#

Configuration to define the executor to use for running diagnostics

`config`#

Additional configuration for the executor.

See the documentation for the executor for the available configuration options. These options will be passed to the executor class when it is created.

Default: {}

Type: dict

`executor`#

Executor class to use for running diagnostics

This should be the fully qualified name of the executor class (e.g. climate_ref.executor.LocalExecutor). The default is to use the local executor which runs the executions locally, in-parallel using a process pool.

This class will be used for all executions of diagnostics.

Default: 'climate_ref.executor.LocalExecutor'

Type: str

Environment Variable: REF_EXECUTOR

`native_store`#

Configuration for the content-addressed native-bundle object store.

The native store holds the curated native outputs (NetCDF, PNG, ...) produced by each test case, keyed by their sha256 digest. Read operations (has, fetch) are always anonymous and credential-free. Write operations are gated to the mint verb only.

`bucket`#

Name of the writable (Cloudflare R2) bucket.

Non-secret routing config, consumed only by the mint verb. Set REF_NATIVE_STORE_BUCKET to override.

Write credentials are not stored here: the access-key id and secret-access-key are read from REF_NATIVE_STORE_ACCESS_KEY_ID / REF_NATIVE_STORE_SECRET_ACCESS_KEY (falling back to boto3's default credential chain) at upload time only, so secrets never land in a serialised config.

Default: 'ref-baselines-public'

Type: str

Environment Variable: REF_NATIVE_STORE_BUCKET

`cache_dir`#

Local pooch cache directory for downloaded native blobs.

Defaults via :func:~climate_ref_core.dataset_registry.resolve_cache_dir, so the REF_DATASET_CACHE_DIR environment variable applies here too.

Default: PosixPath('/home/docs/.cache/climate_ref/native-baselines')

Type: Path

Environment Variable: REF_NATIVE_STORE_CACHE_DIR

`s3_endpoint_url`#

S3 API endpoint for the writable (Cloudflare R2) backend, without the bucket.

Non-secret routing config, consumed only by the mint verb. Defaults to the production Climate-REF R2 account endpoint (default jurisdiction — note there is no .eu in the host). Anonymous read (fetch / has) uses :attr:url instead and never touches this. Set REF_NATIVE_STORE_S3_ENDPOINT_URL to override (e.g. a staging account).

Default: 'https://2aa5172b2bba093c516027d6fa13cdc8.r2.cloudflarestorage.com'

Type: str

Environment Variable: REF_NATIVE_STORE_S3_ENDPOINT_URL

`url`#

Base URL of the native-bundle object store.

Blobs are served at {url}/{digest}. Defaults to the production Climate-REF baselines endpoint.

Set REF_NATIVE_STORE_URL to a local file:///path/to/dir (or a plain filesystem path) for offline development and testing.

Default: 'https://baselines.climate-ref.org'

Type: str

Environment Variable: REF_NATIVE_STORE_URL

`paths`#

Common paths used by the REF application

Warning

These paths must be common across all systems that the REF is being run. Generally, this means that they should be mounted in the same location on all systems.

If any of these paths are specified as relative paths, they will be resolved to absolute paths. These absolute paths will be used for all operations in the REF.

`dimensions_cv`#

Path to a file containing the controlled vocabulary for the dimensions in a CMEC diagnostics bundle

This defaults to the controlled vocabulary for the CMIP7 Assessment Fast Track diagnostics, which is included in the climate_ref_core package.

This controlled vocabulary is used to validate the dimensions in the diagnostics bundle. If custom diagnostics are implemented, this file may need to be extended to include any new dimensions.

Default: '$REF_INSTALL_DIRECTORY/cv_cmip7_aft.yaml'

Type: Path

Environment Variable: REF_DIMENSIONS_CV_PATH

`log`#

Directory to store log files from the compute engine

This is not currently used by the REF, but is included for future use.

Default: '$REF_CONFIGURATION/log'

Type: Path

Environment Variable: REF_LOG_ROOT

`results`#

Path to store the executions

Default: '$REF_CONFIGURATION/results'

Type: Path

Environment Variable: REF_RESULTS_ROOT

`scratch`#

Shared scratch space for the REF.

This directory is used to write the intermediate executions of a diagnostic execution. After the diagnostic has been run, the executions will be copied to the executions directory.

This directory must be accessible by all the diagnostic services that are used to run the diagnostics, but does not need to be mounted in the same location on all the diagnostic services.

Default: '$REF_CONFIGURATION/scratch'

Type: Path

Environment Variable: REF_SCRATCH_ROOT

`software`#

Shared software space for the REF.

This directory is used to store software environments.

This directory must be accessible by all the diagnostic services that are used to run the diagnostics, and should be mounted in the same location on all the diagnostic services.

Default: '$REF_CONFIGURATION/software'

Type: Path

Environment Variable: REF_SOFTWARE_ROOT

Configuration#

Configuration File Discovery#

Managing configuration from the CLI#

Additional Environment Variables#

REF_DATASET_CACHE_DIR#

REF_TEST_DATA_DIR#

REF_TEST_OUTPUT#

Grey list#

Offline and air-gapped deployments#

Configuration Options#

Top-level#

db#

diagnostic_providers#

executor#

native_store#

paths#

`REF_DATASET_CACHE_DIR`#

`REF_TEST_DATA_DIR`#

`REF_TEST_OUTPUT`#

`db`#

`diagnostic_providers`#

`executor`#

`native_store`#

`paths`#