`SpillBuffer.spilled_total` appears to return incorrect results

When spilling data to disk, the SpillBuffer appears to return the incorrect size of the data on disk. For example, when spilling an ~8 MB random matrix onto disk, the data file created by the SpillBuffer is also ~8 MB in size, yet the SpillBuffer only returns ~1 MB.

Reproducer:

from __future__ import annotations

import os
import tempfile

import numpy as np

from distributed.spill import SpillBuffer


def test_spill_size():
    tmpdir = tempfile.mkdtemp()
    buf = SpillBuffer(spill_directory=tmpdir, target=0, max_spill=False)
    data = np.random.random((1024, 1024))
    buf["data"] = data
    spill_size = os.stat(os.path.join(tmpdir,  "data")).st_size
    assert buf.spilled_total.disk == spill_size

fails with

>       assert buf.spilled_total.disk == spill_size
E       assert 1048808 == 8388840
E        +  where 1048808 = SpilledSize(memory=8388608, disk=1048808).disk
E        +    where SpilledSize(memory=8388608, disk=1048808) = Buffer<<LRU: 0/0 on dict>, <zict.cache.Cache object at 0x11f2413d0>>.spilled_total

[crusaderky]

I was not aware that len(memoryview)! = memoryview.nbytes. Good catch! No need to wait for my return.

[ncclementi]

That line was suggested by Guido when we were working on this. I’m sure he’ll have an idea of what could be happening here. I know he is on PTO, but he might be a good person to review this. He will be back next week.