Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Multiple group constraints fail with different constraint types

See original GitHub issue

Problem description

When there is more than one group constraint, and the group constraints have different constraint types, Calliope fails

Error message from test

calliope/test/common/util.py:30: in build_test_model
    scenario=scenario
calliope/core/model.py:70: in __init__
    self._init_from_model_run(model_run, debug_data)
calliope/core/model.py:92: in _init_from_model_run
    self._model_data_original = build_model_data(model_run)
calliope/core/preprocess/model_data.py:71: in build_model_data
    data.merge(xr.Dataset.from_dict(data_dict), inplace=True)
../../miniconda3/envs/calliope_dev/lib/python3.6/site-packages/xarray/core/dataset.py:2571: in merge
    join=join)
../../miniconda3/envs/calliope_dev/lib/python3.6/site-packages/xarray/core/merge.py:561: in dataset_merge_method
    return merge_core(objs, compat, join, priority_arg=priority_arg)
../../miniconda3/envs/calliope_dev/lib/python3.6/site-packages/xarray/core/merge.py:439: in merge_core
    aligned = deep_align(coerced, join=join, copy=False, indexes=indexes)
../../miniconda3/envs/calliope_dev/lib/python3.6/site-packages/xarray/core/alignment.py:216: in deep_align
    exclude=exclude)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

objects = (<xarray.Dataset>
Dimensions:                                               (carrier_tiers: 2, carriers: 1, constraint...e_max  (constraint_groups, carriers) float64 0.4
    group_demand_share_min  (constraint_groups, carriers) float64 0.6)
kwargs = {}, join = 'outer', copy = False, indexes = {}, exclude = frozenset()
all_indexes = defaultdict(<class 'list'>, {'carrier_tiers': [Index(['out', 'in'], dtype='object', name='carrier_tiers')], 'carriers'...21:00:00', '2005-02-01 22:00:00',
       '2005-02-01 23:00:00'],
      dtype='object', name='timesteps', length=768)]})
unlabeled_dim_sizes = defaultdict(<class 'set'>, {'carriers': {1}, 'constraint_groups': {1}, 'costs': {1}, 'loc_techs': {6}, 'loc_techs_finite_resource': {2}, 'loc_techs_investment_cost': {4}, 'techs': {3}})
dim = 'constraint_groups', joiner = functools.partial(<built-in function reduce>, <built-in function or_>)
joined_indexes = {'carriers': Index(['electricity'], dtype='object', name='carriers'), 'constraint_groups': Index(['demand_share_max', 'demand_share_min'], dtype='object', name='constraint_groups')}
unlabeled_sizes = {1}

    def align(*objects, **kwargs):
        """align(*objects, join='inner', copy=True, indexes=None,
                 exclude=frozenset())

        Given any number of Dataset and/or DataArray objects, returns new
        objects with aligned indexes and dimension sizes.

        Array from the aligned objects are suitable as input to mathematical
        operators, because along each dimension they have the same index and size.

        Missing values (if ``join != 'inner'``) are filled with NaN.

        Parameters
        ----------
        *objects : Dataset or DataArray
            Objects to align.
        join : {'outer', 'inner', 'left', 'right', 'exact'}, optional
            Method for joining the indexes of the passed objects along each
            dimension:
            - 'outer': use the union of object indexes
            - 'inner': use the intersection of object indexes
            - 'left': use indexes from the first object with each dimension
            - 'right': use indexes from the last object with each dimension
            - 'exact': instead of aligning, raise `ValueError` when indexes to be
              aligned are not equal
        copy : bool, optional
            If ``copy=True``, data in the return values is always copied. If
            ``copy=False`` and reindexing is unnecessary, or can be performed with
            only slice operations, then the output may share memory with the input.
            In either case, new xarray objects are always returned.
        exclude : sequence of str, optional
            Dimensions that must be excluded from alignment
        indexes : dict-like, optional
            Any indexes explicitly provided with the `indexes` argument should be
            used in preference to the aligned indexes.

        Returns
        -------
        aligned : same as *objects
            Tuple of objects with aligned coordinates.

        Raises
        ------
        ValueError
            If any dimensions without labels on the arguments have different sizes,
            or a different size than the size of the aligned dimension labels.
        """
        join = kwargs.pop('join', 'inner')
        copy = kwargs.pop('copy', True)
        indexes = kwargs.pop('indexes', None)
        exclude = kwargs.pop('exclude', _DEFAULT_EXCLUDE)
        if indexes is None:
            indexes = {}
        if kwargs:
            raise TypeError('align() got unexpected keyword arguments: %s'
                            % list(kwargs))

        if not indexes and len(objects) == 1:
            # fast path for the trivial case
            obj, = objects
            return (obj.copy(deep=copy),)

        all_indexes = defaultdict(list)
        unlabeled_dim_sizes = defaultdict(set)
        for obj in objects:
            for dim in obj.dims:
                if dim not in exclude:
                    try:
                        index = obj.indexes[dim]
                    except KeyError:
                        unlabeled_dim_sizes[dim].add(obj.sizes[dim])
                    else:
                        all_indexes[dim].append(index)

        # We don't reindex over dimensions with all equal indexes for two reasons:
        # - It's faster for the usual case (already aligned objects).
        # - It ensures it's possible to do operations that don't require alignment
        #   on indexes with duplicate values (which cannot be reindexed with
        #   pandas). This is useful, e.g., for overwriting such duplicate indexes.
        joiner = _get_joiner(join)
        joined_indexes = {}
        for dim, matching_indexes in iteritems(all_indexes):
            if dim in indexes:
                index = utils.safe_cast_to_index(indexes[dim])
                if (any(not index.equals(other) for other in matching_indexes) or
                        dim in unlabeled_dim_sizes):
                    joined_indexes[dim] = index
            else:
                if (any(not matching_indexes[0].equals(other)
                        for other in matching_indexes[1:]) or
                        dim in unlabeled_dim_sizes):
                    if join == 'exact':
                        raise ValueError(
                            'indexes along dimension {!r} are not equal'
                            .format(dim))
                    index = joiner(matching_indexes)
                    joined_indexes[dim] = index
                else:
                    index = matching_indexes[0]

            if dim in unlabeled_dim_sizes:
                unlabeled_sizes = unlabeled_dim_sizes[dim]
                labeled_size = index.size
                if len(unlabeled_sizes | {labeled_size}) > 1:
                    raise ValueError(
                        'arguments without labels along dimension %r cannot be '
                        'aligned because they have different dimension size(s) %r '
                        'than the size of the aligned dimension labels: %r'
>                       % (dim, unlabeled_sizes, labeled_size))
E                   ValueError: arguments without labels along dimension 'constraint_groups' cannot be aligned because they have different dimension size(s) {1} than the size of the aligned dimension labels: 2

../../miniconda3/envs/calliope_dev/lib/python3.6/site-packages/xarray/core/alignment.py:145: ValueError

Steps to reproduce the problem

Build a model with group constraints using different constraint types. For example:

group_constraints:
    demand_share_max:
        techs: ["cheap_supply"]
        demand_share_max:
            electricity: 0.4
    demand_share_min:
        techs: ["expensive_supply"]
        demand_share_min:
            electricity: 0.6

Full model

     model:
             name: Test model for group constraint mechanism
             timeseries_data_path: 'timeseries_data'

     run:
         mode: plan
         solver: glpk

     techs:
         cheap_supply:
             essentials:
                 name: Cheap supply tech
                 carrier: electricity
                 parent: supply
             constraints:
                 lifetime: 25
                 energy_cap_max: .inf
             costs:
                 monetary:
                     interest_rate: 0.1
                     energy_cap: 10
         expensive_supply:
             essentials:
                 name: Expensive supply
                 carrier: electricity
                 parent: supply
             constraints:
                 lifetime: 25
                 energy_cap_max: .inf
             costs:
                 monetary:
                     interest_rate: 0.1
                     energy_cap: 20
         electricity_demand:
             essentials:
                 name: Electricity demand
                 carrier: electricity
                 parent: demand
             constraints:
                 resource: file=demand_elec.csv

     locations:
         0,1:
             techs:
                 cheap_supply:
                 expensive_supply:
                 electricity_demand:

     overrides:
         more_than_one_group_constraint:
             group_constraints.demand_share_max:
                 techs: ["cheap_supply"]
                 demand_share_max:
                     electricity: 0.4
             group_constraints.demand_share_min:
                 techs: ["expensive_supply"]
                 demand_share_min:
                     electricity: 0.6