RollingWindow on DataFrame

Hello again,

If you have a minute, would you mind helping me figure out why I can’t get rollingWindow to do what I expect?

I have a bunch of data formatted like so:

__index__   BTC      ETH
----------  -------  -------
2017-06-15  2261.25  333.499
2017-06-16  2481.74  371.358
2017-06-17  2591.41  373.975
2017-06-18  2568.92  377.562
2017-06-19  2578.25  371.813
2017-06-20  2657.32  376.359
2017-06-21  2709.33  338.292
2017-06-22  2714.58  340.326
2017-06-23  2760.61  340.811
2017-06-24  2688.44  334.167
2017-06-25  2627.95  321.283
2017-06-26  2476.72  262.385
2017-06-27  null     null
2017-06-28  2539.78  301.789
2017-06-29  2536.94  315.543
2017-06-30  2547.95  305.518
2017-07-01  2507.89  282.877
2017-07-02  2445.84  269.758
2017-07-03  2519.06  288.076
2017-07-04  2618.16  284.342

And I’m running the following:

const performance = df.rollingWindow(2).select(window => {
  return ((window.last() - window.first()) / window.first())
})
.withIndex(pair => pair[0])
.select(pair => pair[1])
console.log(performance.toString());

But I keep getting…

__index__  __value__
---------  ---------
0          NaN
1          NaN
2          NaN
3          NaN
4          NaN
5          NaN
6          NaN
7          NaN
8          NaN
9          NaN
10         NaN
// truncated for brevity
353        NaN
354        NaN
355        NaN
356        NaN
357        NaN
358        NaN
359        NaN
360        NaN
361        NaN
362        NaN
363        NaN

Not sure what I’m doing wrong. I’ve tried several variations based on the examples on the Guide.

Thank you!

[ashleydavis]

Yes I see the problem (although I haven’t tested the code myself).

You are doing this:

const performance = df.rollingWindow(2).select(window => {
  return ((window.last() - window.first()) / window.first()) // Return the value.
})
.withIndex(pair => pair[0]) // This doesn't work!
.select(pair => pair[1]) // This doesn't work!
console.log(performance.toString());

Instead you probably want this:

const performance = df.rollingWindow(2).select(window => {
  const dailyReturn = ((window.last() - window.first()) / window.first());
  
  // Note returning a pair containing the index and the value.
  return [window.getIndex().last(), dailyReturn]; 
})
.withIndex(pair => pair[0]) // Extract index.
.select(pair => pair[1]) // Extract value.
console.log(performance.toString());

The reason you return the pair (index + value) is because you want to reintegrate the new series with the original dataframe, like this:

const performance = originalDf.rollingWindow(2).select(window => {
  const dailyReturn = ((window.last() - window.first()) / window.first());
  // Note returning a pair containing the index and the value.
  return [window.getIndex().last(), dailyReturn]; 
})
.withIndex(pair => pair[0]) // Extract index.
.select(pair => pair[1]) // Extract value.

// Merge the new series into the orig dataframe:
const merged = orignalDf.withSeries({ Performance: performance });
console.log( merged.toString());

Merging the new series into the original dataframe gets you the combined results.

If you don’t want to merge you can simplify your code to this:

const performance = df.rollingWindow(2).select(window => {
  return ((window.last() - window.first()) / window.first()) // Only need the value, not the index.
});
console.log(performance.toString());

Note that you can simplify further by using the more concise anonymous function syntax as follows:

const performance = df.rollingWindow(2).select(w => ((w.last() - w.first()) / w.first());
console.log(performance.toString());

Also consider using either of the following convenience functions that are built in 😃

For example:

const performance = df.proportionChanged(2); // I think this does exactly what you are after

Or

const performance = df.percentChanged(2); // This converts the result to a percentage.

[ashleydavis]

You could do something like this:

function fillZero (value) {
    if (value === undefined || value === null) {
        return 0;
    }

    return value;
}

const columnTransformer = df.getColumns().toObject(column => column.name, column => fillZero);
const modifiedDf = df.transformSeries(columnTransformer);

Haven’t tested this code! Let me know if it helps!