Extending xclim

xclim tries to make it easy for users to add their own indices and indicators. The following goes into details on how to create indices and document them so that xclim can parse most of the metadata directly. We then explain the multiple ways new Indicators can be created and, finally, how we can regroup and structure them in virtual submodules.

Central to xclim are the Indicators, objects computating indices over climate variables, but xclim also provides other modules:

modules

Where subset is a phantom module, kept for legacy code, as it only redirects the calls to clisops.core.subset.

This introduction will focus on the Indicator/Indice part of xclim and how one can extend it by implementing new ones.

Indices vs Indicators

Internally and in the documentation, xclim makes a distinction between “indices” and “indicators”.

indice

A python function accepting DataArrays and other parameters (usually bultin types)
Returns one or several DataArrays.
Handles the units : checks input units and set proper CF-compliant output units. But doesn’t usually prescribe specific units, the output will at minimum have the proper dimensionality.
Performs no other checks or set any (non-unit) metadata.
Accessible through xclim.indices.

indicator

An instance of a subclass of xclim.core.indicator.Indicator that wraps around an indice (stored in its compute property).
Returns one or several DataArrays.
Handles missing values, performs input data and metadata checks (see usage).
Always ouputs data in the same units.
Adds dynamically generated metadata to the output after computation.
Accessible through xclim.indicators

Most metadata stored in the Indicators is parsed from the underlying indice documentation, so defining indices with complete documentation and an appropriate signature helps the process. The two next sections go into details on the definition of both objects.

Call sequence

The following graph shows the steps done when calling an Indicator. Attributes and methods of the Indicator object relating to those steps are listed on the right side.

indicator

Defining new indices

The annotated example below shows the general template to be followed when defining proper indices. In the comments Ind is the indicator instance that would be created from this function.

Note that it is not needed to follow these standards when writing indices that will be wrapped in indicators. Problems in parsing will not raise errors at runtime, but might raise warnings and will result in Indicators with poorer metadata than expected by most users, especially those that dynamically use indicators in other applications where the code is inaccessible, like web services.

indice doc

The following code is another example.

[1]:

import xarray as xr

import xclim as xc
from xclim.core.units import convert_units_to, declare_units
from xclim.indices.generic import threshold_count


@declare_units(tasmax="[temperature]", thresh="[temperature]")
def tx_days_compare(
    tasmax: xr.DataArray, thresh: str = "0 degC", op: str = ">", freq: str = "YS"
):
    r"""Number of days where maximum daily temperature. is above or under a threshold.

    The daily maximum temperature is compared to a threshold using a given operator and the number
    of days where the condition is true is returned.

    It assumes a daily input.

    Parameters
    ----------
    tasmax : xarray.DataArray
      Maximum daily temperature.
    thresh : str
      Threshold temperature to compare to.
    op : {'>', '<'}
      The operator to use.
      # A fixed set of choices can be imposed. Only strings, numbers, booleans or None are accepted.
    freq : str
      Resampling frequency.

    Returns
    -------
    xarray.DataArray, [temperature]
      Maximum value of daily maximum temperature.

    Notes
    -----
    Let :math:`TX_{ij}` be the maximum temperature at day :math:`i` of period :math:`j`. Then the maximum
    daily maximum temperature for period :math:`j` is:

    .. math::

        TXx_j = max(TX_{ij})

    References
    ----------
    Smith, John and Tremblay, Robert, An dummy citation for examples in documentation. J. RTD. (2020).
    """
    thresh = convert_units_to(thresh, tasmax)
    out = threshold_count(tasmax, op, thresh, freq)
    out.attrs["units"] = "days"
    return out

Naming and conventions

Variable names should correspond to CMIP6 variables, whenever possible. The file xclim/data/variables.yml lists all variables that xclim can use when generating indicators from yaml files (see below), and new indices should try to reflect these also. For new variables, the xclim.testing.get_all_CMIP6_variables function downloads the official table of CMIP6 variables and puts everything in a dictionary. If possible, use variables names from this list, add them to variables.yml as needed.

Generic functions for common operations

The xclim.indices.generic submodule contains useful functions for common computations (like threshold_count or select_resample_op) and many basic indice functions, as defined by clix-meta. In order to reduce duplicate code, their use is recommended for xclim’s indices. As previously said, the units handling has to be made explicitly when non trivial, xclim.core.units also exposes a few helpers for that (like convert_units_to, to_agg_units or rate2amount).

Defining new indicators

xclim’s Indicators are instances of (subclasses of) xclim.core.indicator.Indicator. While they are the central to xclim, their construction can be somewhat tricky as a lot happens backstage. Essentially, they act as self-aware functions, taking a set of input variables (DataArrays) and parameters (usually strings, integers or floats), performing some health checks on them and returning one or multiple DataArrays, with CF-compliant (and potentially translated) metadata attributes, masked according to a given missing value set of rules. They define the following key attributes:

the identifier, as string that uniquely identifies the indicator, usually all caps.
the realm, one of “atmos”, “land”, “seaIce” or “ocean”, classifying the domain of use of the indicator.
the compute function that returns one or more DataArrays, the “indice”,
the cfcheck and datacheck methods that make sure the inputs are appropriate and valid.
the missing function that masks elements based on null values in the input.
all metadata attributes that will be attributed to the output and that document the indicator:
- Indicator-level attribute are : title, abstract, keywords, references and notes.
- Ouput variables attributes (respecting CF conventions) are: var_name, standard_name, long_name, units, cell_methods, description and comment.

Output variables attributes are regrouped in Indicator.cf_attrs and input parameters are documented in Indicator.parameters.

A particularity of Indicators is that each instance corresponds to a single class: when creating a new indicator, a new class is automatically created. This is done for easy construction of indicators based on others, like shown further down.

See the class documentation for more info on the meaning of each attribute. The indicators module contains over 50 examples of indicators to draw inspiration from.

Identifier vs python name

An indicator’s identifier is not the same as the name it has within the python module. For example, xc.atmos.relative_humidity has hurs as its identifier. As explained below, indicator classes can be accessed through xc.core.indicator.registry with their identifier.

Metadata parsing vs explicit setting

As explained above, most metadata can be parsed from the indice’s signature and docstring. Otherwise, it can always be set when creating a new Indicator instance or a new subclass. When creating an indicator, output metadata attributes can be given as strings, or list of strings in the case of indicator returning multiple outputs. However, they are stored in the cf_attrs list of dictionaries on the instance.

Internationalization of metadata

xclim offers the possibility to translate the main Indicator metadata field and automatically add the translations to the outputs. The mechnanic is explained in the Internationalization page.

Inputs and checks

xclim decides which input arguments of the indicator’s call function are considered variables and which are parameters using the annotations of the underlying indice (the compute method). Arguments annotated with the xarray.DataArray type are considered variables and can be read from the dataset passed in ds.

Indicator creation

There a two ways for creating indicators:

By initializing an existing indicator (sub)class
From a dictionary

The first method is best when defining indicators in scripts of external modules and are explained here. The second is best used when building virtual modules through YAML files, and is explained further down and in the submodule doc.

Creating a new indicator that simply modifies a few metadata output of an existing one is a simple call like:

[2]:

from xclim.core.indicator import registry
from xclim.core.utils import wrapped_partial

# An indicator based on tg_mean, but returning Celsius and fixed on annual resampling
tg_mean_c = registry["TG_MEAN"](
    identifier="tg_mean_c",
    units="degC",
    title="Mean daily mean temperature but in degC",
    parameters=dict(freq="YS"),  # We inject the freq arg.
)

[3]:

print(tg_mean_c.__doc__)

Mean daily mean temperature but in degC (realm: atmos)

Resample the original daily mean temperature series by taking the mean over each period.

This indicator will check for missing values according to the method "from_context".
Based on indice :py:func:`~xclim.indices._simple.tg_mean`.
With injected parameters: freq=YS.

Parameters
----------
tas : str or DataArray
  Mean daily temperature.
  Default : `ds.tas`. [Required units : [temperature]]
ds : Dataset, optional
  A dataset with the variables given by name.
  Default : None.
indexer :
  Indexing parameters to compute the indicator on a temporal subset of the data. It accepts the same arguments as :py:func:`xclim.indices.generic.select_time`.
  Default : None.

Returns
-------
tg_mean : DataArray
  Mean daily mean temperature (air_temperature) [K]
  cell_methods: time: mean within days time: mean over days
  description: {freq} mean of daily mean temperature.

Notes
-----
Let :math:`TN_i` be the mean daily temperature of day :math:`i`, then for a period :math:`p` starting at
day :math:`a` and finishing on day :math:`b`:

.. math::

   TG_p = \frac{\sum_{i=a}^{b} TN_i}{b - a + 1}

The registry is a dictionary mapping indicator identifiers (in uppercase) to their class. This way, we could subclass tg_mean to create our new indicator. tg_mean_c is the exact same as atmos.tg_mean, but outputs the result in Celsius instead of Kelvins, has a different title and removes control over the freq argument, resampling to “YS”. The identifier keyword is here needed in order to differentiate the new indicator from tg_mean itself. If it wasn’t given, a warning would have been raised and further subclassing of tg_mean would have in fact subclassed tg_mean_c, which is not wanted!

By default, indicator classes are registered in xclim.core.indicator.registry, using their identifier which is prepended by the indicator’s module if that indicator is declared outisde xclim. An “child” indicator inherits it’s module from its parent:

[4]:

tg_mean_c.__module__ == xc.atmos.tg_mean.__module__

[4]:

True

To create indicators with a different module, for example, in a goal to differentiate them in the registry, two methods can be used : passing module to the constructor, or using conventional class inheritance.

[5]:

# Passing module
tg_mean_c2 = registry["TG_MEAN_C"](module="test")  # we didn't change the identifier!
print(tg_mean_c2.__module__)
"test.TG_MEAN_C" in registry

xclim.indicators.test

[5]:

True

[6]:

# Conventionnal class inheritance, uses the current module name
class TG_MEAN_C3(registry["TG_MEAN_C"]):
    pass  # nothing to change really


tg_mean_c3 = TG_MEAN_C3()

print(tg_mean_c3.__module__)
"__main__.TG_MEAN_C" in registry

__main__

[6]:

True

While the former method is shorter, the latter is what xclim uses internally as it provides some clean code structure. See the code in the github repo.

Extending xclim

Indices vs Indicators

indice

indicator

Call sequence

Defining new indices

Naming and conventions

Generic functions for common operations

Defining new indicators

Identifier vs python name

Metadata parsing vs explicit setting

Internationalization of metadata

Inputs and checks

Indicator creation

Virtual modules

YAML file

Validation of the YAML file

Loading the module and computating of the indices.

Mapping of indicators