Skip to content

How to control memory use and parallism#

The diagnostics packages used by the REF all use Dask to process data that is larger than memory in parallel. By default, Dask uses its threaded scheduler, which may not be optimal for more complicated computations, and it configures this threaded scheduler to use as many worker threads as there are CPU cores on the machine. Because the REF typically runs multiple executors in parallel (see Executors), and if unconfigured, each executor uses as many threads as there are CPU cores, this can lead to excessive memory use and too much parallism, which can cause the system to run out of memory or become slow because of excessive context switching. Inefficient scheduling by the threaded scheduler can also lead to excessive memory use and/or slow computations. Therefore, it is highly recommended that you take a moment to configure Dask for your system. For an in-depth introduction to these topics, see the Dask documentation on configuration and scheduling.

Configuring ESMValTool#

ESMValCore, the framework powering ESMValTool, works best with the Dask Distributed scheduler. It is recommended to set max_parallel_tasks (an ESMValCore setting), to a low number, e.g. 1, 2 or 3, because only one ESMValCore preprocessing task will submit jobs to the Distributed scheduler at a time to avoid overloading the workers. Therefore, the following settings are recommended for ESMValCore:

max_parallel_tasks: 2
dask:
  use: local_distributed
  profiles:
    local_distributed:
      cluster:
        type: distributed.LocalCluster
        n_workers: 2
        threads_per_worker: 2
        memory_limit: 4GiB

These settings should be put in a file with the extension .yaml in the directory ~/.config/esmvaltool, for example: ~/.config/esmvaltool/dask.yaml.

With the settings above, the total memory per REF diagnostic execution will be n_workers * memory_limit = 8GB. It is recommended to use at least 4GB of RAM per Dask Distributed worker. Some diagnostics may be able to run with 2GB per worker, but probably not all of them. You can tune the total memory / CPU use by specifying the number of workers. The number of CPU cores used will be n_workers * threads_per_worker.

Note that the REF may run multiple executors in parallel, and each executor running an ESMValTool diagnostic will use the resources specified above.

More information on how to configure ESMValCore is available in its documentation.

ESMValTool users

If you are using ESMValTool outside of the REF on the same computer, it is highly recommended that you create a separate ESMValTool configuration directory for the version of ESMValTool used by the REF to avoid conflicts, e.g. accidentally using input data that is not managed by the REF. You can do this by setting the ESMVALTOOL_CONFIG_DIR environment variable to a different directory, e.g. ~/.config/esmvaltool-ref, and then creating the Dask configuration file in that directory, e.g. ~/.config/esmvaltool-ref/dask.yaml with the settings described above.

Configuring PMP and ILAMB/IOMB#

Both ILAMB/IOMB and PMP use Dask through Xarray, but they do not expose their own Dask configuration options. Therefore, you need to configure Dask globally for these packages by creating a Dask configuration file, e.g. at ~/.config/dask/config.yaml.

While testing the REF, we have seen occasional crashes when running ILAMB/IOMB and PMP diagnostics with the threaded scheduler, so we recommend using the synchronous scheduler for these diagnostics by adding the following content to the global Dask configuration file:

scheduler: synchronous

For faster processing, you can can use the threaded scheduler with a limited number of worker threads (adjust to your system's resources):

scheduler: threads
num_workers: 4

Note that the REF may run multiple executors in parallel, and each executor running a PMP diagnostic will use the resources specified above.

ILAMB/IOMB diagnostics

The ILAMB/IOMB diagnostics are currently restricted to the synchronous scheduler, so they will not respect the global Dask settings.

Limiting OpenMP and BLAS threads#

Independently of Dask, the numerical libraries underneath the diagnostics (NumPy, SciPy, and the BLAS/LAPACK backends they link against, such as OpenBLAS or MKL) spin up their own thread pools. By default each of these pools grows to the number of CPU cores on the machine. Because the REF runs many diagnostic executions in parallel (the LocalExecutor launches up to n worker processes), an unconfigured numerical backend multiplies out to n_processes * n_cores threads, which oversubscribes the CPUs and causes heavy context-switching that slows the whole run down.

Cap the per-process thread count by exporting the following environment variables before running ref solve (a small number such as 4 is a good starting point, but tune it against the executor's n and your core count):

export OMP_NUM_THREADS=4
export OPENBLAS_NUM_THREADS=4
export MKL_NUM_THREADS=4
export NUMEXPR_NUM_THREADS=4
export VECLIB_MAXIMUM_THREADS=4
export BLIS_NUM_THREADS=4

Each variable targets a different backend, so it is safest to set all of them:

  • OMP_NUM_THREADS (OpenMP)
  • OPENBLAS_NUM_THREADS (OpenBLAS)
  • MKL_NUM_THREADS (Intel MKL)
  • NUMEXPR_NUM_THREADS (NumExpr)
  • VECLIB_MAXIMUM_THREADS (Apple Accelerate/vecLib)
  • BLIS_NUM_THREADS (BLIS)

They must be set in the environment before NumPy is imported, so export them in the shell (or your job script) that launches the solve rather than from within Python.

Note

As a rough guide, keep n * OMP_NUM_THREADS at or below the number of physical cores so the executor's worker processes and the BLAS threads do not compete for the same CPUs.