`np.sum` lost precision for large array

A simple replicate (in CPU) is np.sum(np.ones(1000000) * 0.1).copy() which gives the result 100958.34 while it is expected to return a number around 1000000 +- 10.

[ekelsen]

np.sum uses a numerically stable pairwise summation algorithm (https://github.com/numpy/numpy/pull/3685)

Tensorflow’s CPU implementation should also now do the same thing (exact commit is somewhere in Eigen, ask Rasmus). As does the GPU implementation.

I think the issue here is that XLA does not do this. So basically, the problem has been fixed everywhere except XLA and probably XLA should implement numerically stable summation algorithms as well…

[mattjj]

@jlebar of the XLA team made this acute observation:

Huh, this is counterintuitive to me, but I also suspect it’s WAI.

It’s counterintuitive because I’d have expected precision loss can’t occur when the relative difference between the smallest summand (0.1) and the total sum (1e5) is less than 2^24 (number of mantissa bits in f32). In this case, log2(1e5/0.1) = 20 < 24, so…there should be no precision loss?

But the above analysis assumes that the summand and total sum are precisely-representable as floating-point numbers. My guess is that this behavior you’re seeing is related to the fact that 0.1 can’t be precisely expressed in float. If you change it to 0.125 or 0.0625, do you get a precise answer?

And indeed, as he predicts:

In [1]: import jax.numpy as np

In [2]: print np.sum(np.ones(int(1e6)) * 0.125)
jax/lib/xla_bridge.py:144: UserWarning: No GPU/TPU found, falling back to CPU.
  warnings.warn('No GPU/TPU found, falling back to CPU.')
125000.0

The fact that enabling fast math makes the precision go up seems like just coincidence, since fast math can do a lot of things.

I think this might be working as intended, but if you have a problem around this example, maybe we should figure out a workaround. What do you think?