Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Alignment issue when passing arrays though mmap

See original GitHub issue

Arrays that are transfered to subprocesses though mmap appear not always to be aligned:

import joblib
import numpy as np

def func(x):
    print(type(x))
    print(x.flags)
    print(x.ctypes.data / 8)

if __name__ == '__main__':
    # Make the array larger than 1Mbyte so that mmap is used
    n = 1024 ** 2 // 8 + 1
    x = np.random.randn(n)
    jobs = joblib.Parallel(2)
    print(x.nbytes)
    res = jobs([joblib.delayed(func)(x), joblib.delayed(func)(x)])

This returns

<class 'numpy.core.memmap.memmap'>
1048584
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : False
  ALIGNED : False
  UPDATEIFCOPY : False
570054683.375

Note the last line, the address of the first element of the array is indeed not aligned.

I’m using version 0.11, on OSx sierra.

This came up in https://github.com/pymc-devs/pymc3/issues/2640, and seems to happen on several OSs.

Issue Analytics

State:
Created 6 years ago
Comments:6 (5 by maintainers)

Top GitHub Comments

3reactions

lestevecommented, Oct 19, 2017

I can reproduce, this is a problem indeed. I’ll try to look at this, but probably not before next week.

1reaction

lorentzenchrcommented, Nov 20, 2021

A little history for fun and variety: Opening https://github.com/pymc-devs/pymc/issues/2640 was among my first contributions on github. This led to opening this issue in joblib. Now, years later, I stumble over this same error while working on https://github.com/scikit-learn/scikit-learn/pull/20567. Joblib memory alignment and me got off on the wrong foot😉

Top Results From Across the Web

mmap memory alignment in ARM - Stack Overflow

We've had a few memory alignment issues (SIGBUS error crashes) due to reading/writing with misaligned pointers.

Explanation of the assertion on unaligned arrays - Eigen

Cause 2: STL Containers or manual memory allocation ... If you use STL Containers such as std::vector, std::map, ..., with Eigen objects, or...

Data Alignment - an overview | ScienceDirect Topics

This directive will inform the compiler that the arrays in the loop are aligned and corresponding aligned memory read/write instructions can be used....

Structure Member Alignment, Padding and Data Packing

If the access is misaligned, the processor automatically issues sufficient memory read cycles and packs the data properly onto the data bus. The ......

JuliaArrays/UnalignedVectors.jl: Create arrays from memory ...

Create arrays from memory buffers that lack appropriate alignment ... A common usage might be memory mapping, using Julia's Mmap.mmap functionality.