climate_ref_core.esmvaltool_reference

Path conventions for ESMValTool reference (observational/reanalysis) data.

ESMValCore locates this data from its own DRS directory templates rather than by instance_id, so a reference file is identified by where its DRS path begins.

This data is not CMOR/obs4MIPs compliant, so its metadata cannot be read from global attributes. It is parsed from the path and filename templates that ESMValTool itself uses to find the data:

• OBS / OBS6 (metadata from the filename): OBS/Tier{tier}/{dataset}/{project}_{dataset}_{type}_{version}_{mip}_{short_name}_{timerange}.nc
• native6 (metadata from the directory, raw non-CMOR filename): native6/Tier{tier}/{dataset}/{version}/{frequency}/{short_name}/*.nc
• obs4MIPs: obs4MIPs/{dataset}/{version}/{short_name}_*.nc

Both the ingest adapter and the registry fetcher parse through :func:parse_reference_path, so a request cannot name a facet value that ingest would spell differently.

PROJECT_ANCHORS = tuple(_PROJECT_LAYOUTS) module-attribute

Top-level project directories, relative to the ESMValTool data root, that begin a DRS path.

OBS6 data lives under the OBS directory, so the anchor for both is OBS. The project itself is recovered from the filename.

ReferenceFacets

Bases: NamedTuple

Metadata carried by the path of one ESMValTool reference file.

timerange is the raw DRS token rather than a parsed date range, because it is not a facet anything selects on.

Source code in packages/climate-ref-core/src/climate_ref_core/esmvaltool_reference.py

class ReferenceFacets(NamedTuple):
    """
    Metadata carried by the path of one ESMValTool reference file.

    ``timerange`` is the raw DRS token rather than a parsed date range,
    because it is not a facet anything selects on.
    """

    project: str
    source_id: str
    variable_id: str
    frequency: str
    version: str
    data_type: str | None
    tier: int | None
    timerange: str | None

drs_relative_parts(path)

Split a reference file path into its DRS-relative components.

A data root may itself contain a directory named after a project, and a directory inside the tree may in turn be named after another project, so neither the first nor the last matching component is reliable on its own. The rightmost candidate whose remaining components fit its project's layout wins.

Parameters:

Name	Type	Description	Default
path	str | Path	Path to a reference file, absolute or relative.	required

Returns:

Type	Description
tuple[str, ...]	The path components from the project anchor onward, so .../ESMValTool/OBS/Tier2/CERES-EBAF/x.nc becomes ("OBS", "Tier2", "CERES-EBAF", "x.nc").

Raises:

Type	Description
ValueError	If no component of path names a project, or if none of them begins a path that fits that project's layout.

Source code in packages/climate-ref-core/src/climate_ref_core/esmvaltool_reference.py

def drs_relative_parts(path: str | Path) -> tuple[str, ...]:
    """
    Split a reference file path into its DRS-relative components.

    A data root may itself contain a directory named after a project, and a directory
    inside the tree may in turn be named after another project, so neither the first nor
    the last matching component is reliable on its own. The rightmost candidate whose
    remaining components fit its project's layout wins.

    Parameters
    ----------
    path
        Path to a reference file, absolute or relative.

    Returns
    -------
    :
        The path components from the project anchor onward,
        so ``.../ESMValTool/OBS/Tier2/CERES-EBAF/x.nc`` becomes
        ``("OBS", "Tier2", "CERES-EBAF", "x.nc")``.

    Raises
    ------
    ValueError
        If no component of ``path`` names a project,
        or if none of them begins a path that fits that project's layout.
    """
    parts = Path(path).parts
    candidates = [index for index, part in enumerate(parts) if part in PROJECT_ANCHORS]

    if not candidates:
        raise ValueError(
            f"{path} is not under a known ESMValTool reference project ({', '.join(PROJECT_ANCHORS)})"
        )

    for index in reversed(candidates):
        if _fits_layout(parts[index:]):
            return parts[index:]

    raise ValueError(f"unexpected {parts[candidates[0]]} path structure: {path}")

parse_reference_path(path)

Read the metadata a reference file's DRS path encodes.

Parameters:

Name	Type	Description	Default
path	str | Path	Path to a reference file, absolute or relative. It may equally be a registry key, which is the same path relative to the data root.	required

Returns:

Type	Description
ReferenceFacets	The facets the path carries.

Raises:

Type	Description
ValueError	If the path does not fit any project layout, or if the filename does not fit the template of the project it sits under.

Source code in packages/climate-ref-core/src/climate_ref_core/esmvaltool_reference.py

def parse_reference_path(path: str | Path) -> ReferenceFacets:
    """
    Read the metadata a reference file's DRS path encodes.

    Parameters
    ----------
    path
        Path to a reference file, absolute or relative.
        It may equally be a registry key, which is the same path relative to the data root.

    Returns
    -------
    :
        The facets the path carries.

    Raises
    ------
    ValueError
        If the path does not fit any project layout,
        or if the filename does not fit the template of the project it sits under.
    """
    file = Path(path)
    rel = drs_relative_parts(file)
    anchor = rel[0]

    if anchor == "OBS":
        return _parse_obs(rel, file.name)
    if anchor == "native6":
        return _parse_native6(rel)
    return _parse_obs4mips(rel, file.name)

tier_from_segment(segment)

Parse a TierN directory name into its tier number.

Parameters:

Name	Type	Description	Default
segment	str	A single path component.	required

Returns:

Type	Description
int | None	The tier number, or None if segment does not name a tier.

Source code in packages/climate-ref-core/src/climate_ref_core/esmvaltool_reference.py

def tier_from_segment(segment: str) -> int | None:
    """
    Parse a ``TierN`` directory name into its tier number.

    Parameters
    ----------
    segment
        A single path component.

    Returns
    -------
    :
        The tier number, or ``None`` if ``segment`` does not name a tier.
    """
    if segment.startswith("Tier") and segment[4:].isdigit():
        return int(segment[4:])
    return None