Skip to content

Installation#

There are several ways to install and use the climate-ref package and associated CLI tool (ref). If you find the various options to be confusing, we recommend reading this overview about the state of python package installation.

The REF itself is a pure Python package and does not require any conda-based dependencies. This was a deliberate decision to make it easy to make the framework easy to install in a range of different environments. Some of the diagnostic providers require additional dependencies in order to run an execution. For these providers, the REF can automatically create a new Conda environment and install the required dependencies in this standalone environment. Each of these provider-specific environments are decoupled to allow for potentially incompatible dependencies. This uses a bundled version of micromamba to create and manage the environments so no additional dependencies are required.

HPC Users

HPC users are recommended to install the REF inside a conda environment to ensure that key system dependencies are available.

This can be achieved using the following commands:

conda create -n climate-ref python=3.11
conda activate climate-ref
pip install "climate-ref[aft-providers]"

Installing climate-ref#

PyPI#

You can install climate-ref using pip:

pip install climate-ref

The PyPI package contains some extras that bundle some additional dependencies. Depending on your use case you may want to install:

  • celery - for running the REF in a distributed manner
  • aft-providers - diagnostic providers for Assessment Fast Track
  • postgres - additional dependencies required for using a PostgresSQL database

Extras can be installed by including the name inside square brackets after the package name:

pip install climate-ref[celery,aft-providers]

Note

pip install "climate-ref[aft-providers]" additionally installs the diagnostic providers that are used for the Assessment Fast Track. These diagnostic providers declare the available diagnostics and the rules for when they should be run. The actual execution occurs within a provider-specific conda environment.

Conda#

conda-forge

The conda-forge packages are a work in progress and are not yet available. See #80 for more information.

We intend on providing a recipe on conda-forge. Once complete you can install climate-ref using mamba or conda:

mamba install -c conda-forge climate-ref
conda install -c conda-forge climate-ref

A modern alternative to using conda as package a manager is Pixi. Pixi uses the same packages as conda but is faster and can create reproducible environments out of the box.

Docker#

For production use, we recommend using the climate-ref Docker image to provide a consistent environment for running the REF. Not all users may support Docker directly, instead requiring the use of Apptainer to provide some additional isolation.

docker pull ghcr.io/climate-ref/climate-ref:latest

or

apptainer pull docker://ghcr.io/climate-ref/climate-ref:latest

If you want to use the latest development version, you can build the Docker image from the source code:

git clone https://github.com/Climate-REF/climate-ref.git
cd climate-ref
docker build

If you require the full-stack of services recommended for a production deployment, you can use the docker-compose file to start the services. This requires an initialisation step to fetch some required datasets and environments before starting the services.

bash scripts/initialise-docker.sh
docker-compose up

From Source#

To install climate-ref from the source code, clone the repository and install it:

git clone https://github.com/Climate-REF/climate-ref.git
cd climate-ref
make virtual-environment

See the development documentation for more information on how to contribute to the project.

Setting up diagnostic providers#

Windows support

Windows doesn't support some of the packages required by the diagnostic providers, so we only support MacOS and Linux. Windows users are recommended to use WSL or a Linux VM if they wish to use the REF.

Diagnostic providers may require additional setup before they can be used:

  • Conda environments: Some providers run diagnostics in isolated conda environments
  • Reference data: Some providers need to download reference datasets or auxiliary files

The REF can manage all of this for you with a single command:

ref providers setup

This command:

  1. Creates conda environments for providers that need them (using bundled micromamba)
  2. Downloads any required reference data to the local cache
  3. Performs any additional provider-specific setup

All operations are idempotent, so it's safe to run multiple times.

HPC users

If you're running the REF on an HPC system where compute nodes don't have internet access, run ref providers setup on a login node before submitting jobs. This ensures all environments and data are available for offline execution.

Note

The PMP conda environment is not yet available for arm-based MacOS users, so the automatic installation process will fail.

Arm-based MacOS users can use the following command to create the conda environment manually:

MAMBA_PLATFORM=osx-64 ref providers setup --provider pmp

The installed providers and their status can be viewed using:

ref providers list

Advanced options#

For fine-grained control, you can use additional flags:

# Setup only a specific provider
ref providers setup --provider esmvaltool

# Skip environment creation (only fetch data)
ref providers setup --skip-env

# Skip data fetching (only create environments)
ref providers setup --skip-data

# Validate setup without running it
ref providers setup --validate-only