Speed up get_formatted_array

Problem description

Using get_formatted_array splits loc::techs and loc::tech::carriers string sets and interacts between xarray and pandas to produce a sparse matrix for easier indexing (e.g. summing over a single tech).

This can take a very long time for large DataArrays, and has been recorded as hitting memory limits for some devices.

So, it should be made more efficient. This could be a matter of defning loc::techs etc. as tuples instead of :: concatenated strings. Then they automagically are parsed as a MultiIndex, instead of needing to apply string operations.

Calliope version

0.6.3

[brynpickering]

@timtroendle, I actually just switched off postprocessing on the cluster in my runs, due to the same issue… Anyway, I had some stuff waiting to go on this, see PR #231 for a working branch that you could test with. It may still blow up on unstacking the MultiIndex (but my memory profiling suggests a much lower memory use than the previous incarnation of get_formatted_array.

I’m a but confused by your solution, how does it go from ("loctechscarriers", data_var_df.index) to being possible to select a location using ("loctechscarriers", data_var_df.index)? If it offers an even better solution, I’m happy to look at updating the PR in line with it.

[sjpfenninger]

Some ideas in here for further improvements of how we deal with arrays…