Compute forward difference with Dask DataFrame

I’m really excited about Dask. A huge THANK YOU to all the contributors 😃

I’d like to make a feature request for a new method: dask.DataFrame.diff(). It would work like pandas.DataFrame.diff().

Mathematically, the operation is very simple: subtract a column vector from a copy of itself shifted by one or more rows.

I have tried implementing diff() using Dask in the following ways, none of which works (yet):

• df - df.shift(periods=1) works in Pandas. But Dask DataFrame doesn’t have a shift() method.
• df.values[:-1] - df.values[1:] works in Pandas. But I can’t see how to index into a Dask DataFrame by position.

My current best idea for implementing diff would be to wrap some custom code in dask.dataframe.rolling.wrap_rolling, as suggested in this stack overflow answer (although I haven’t been able to find any documentation on how to do this). Or wrap some custom code using Dask Delayed? Any other thoughts?

[mrocklin]

It would be nice to have a higher level function around rolling that accepted user defined functions, much in the same way we do for reduction today.

[jcrist]

Sure, I can take this on. Definitely agree that a general purpose function would be good here.