Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

BUG: ARPACK's eigsh & OpenBLAS from Apple Silicon M1 (arm64) nightly build triggers macOS kernel panic / reboot

See original GitHub issue

Describe your issue.

Executing the following script makes all the 8 CPU cores run 100% for 1 or 2 seconds then macOS 11.5.1 (20G80) crashes (on a MacBook Air M1 from December 2020).

Note that this machine has 8 cores (4 performance cores and 4 efficiency cores). Running with OPENBLAS_NUM_THREADS=n set with n <= 4 runs fine and quickly.

With n == 5, the code runs significantly slower.

With n>=6, I get the full system crash almost 100% of the time, sometimes at the 2nd or 3rd iteration of the loop but mostly at the first iteration.

Note that I do not reproduce the crash when running the same code with scipy installed from conda-forge using the macos arm64 installer of miniforge.

Also note that there is probably a problem with the OpenBLAS binary of the numpy wheel because threadpoolctl cannot find the version information:

❯ python -m threadpoolctl -i scipy
[
  {
    "filepath": "/Users/ogrisel/miniforge3/envs/tmp/lib/python3.9/site-packages/numpy/.dylibs/libopenblas64_.0.dylib",
    "prefix": "libopenblas",
    "user_api": "blas",
    "internal_api": "openblas",
    "version": null,
    "num_threads": null,
    "threading_layer": "unknown",
    "architecture": null
  }
]

Reproducing Code Example

Install the scipy nightly build wheel (or download and install a local copy from this archive):

❯ pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy

from https://github.com/scipy/scipy/issues/13409#issuecomment-904548388.

Then try to execute this script:

from time import perf_counter
import numpy as np
from scipy.sparse.linalg import eigsh


n_samples, n_features = 2000, 10
rng = np.random.default_rng(0)
X = rng.normal(size=(n_samples, n_features))
K = X @ X.T

for i in range(10):
    print("running eigsh...")
    tic = perf_counter()
    s, _ = eigsh(K, 3, which="LA", tol=0)
    toc = perf_counter()
    print(f"computed {s} in {toc - tic:.3f} s")

Error message

Here is the panic report from macOS after system reboot:

panic(cpu 3 caller 0xfffffe0018fab460): "Spinlock timeout after 12585145 ticks, 0xfffffe001cd04588 = fffffe1668958661"
Debugger message: panic
Memory ID: 0xff
OS release type: User
OS version: 20G80
Kernel version: Darwin Kernel Version 20.6.0: Wed Jun 23 00:26:27 PDT 2021; root:xnu-7195.141.2~5/RELEASE_ARM64_T8101
Fileset Kernelcache UUID: E46841F89DC3FD7ACEC6F404AC995579
Kernel UUID: AC4A14A7-8A8E-3AE6-85A6-55E6B2502BF9
iBoot version: iBoot-6723.140.2
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x0000000011330000
KernelCache base:  0xfffffe0018334000
Kernel slide:      0x0000000011e78000
Kernel text base:  0xfffffe0018e7c000
Kernel text exec base:  0xfffffe0018f48000
mach_absolute_time: 0xa9b77fbf5
Epoch Time:        sec       usec
  Boot    : 0x613143f7 0x00087da9
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x61314b5c 0x00031a5d

CORE 0 recently retired instr at 0xfffffe00190bad6c
CORE 1 recently retired instr at 0xfffffe00190bad6c
CORE 2 recently retired instr at 0xfffffe00190bad6c
CORE 3 recently retired instr at 0xfffffe00190b96a4
CORE 4 recently retired instr at 0xfffffe00190bad70
CORE 5 recently retired instr at 0xfffffe00190bad70
CORE 6 recently retired instr at 0xfffffe00190bad70
CORE 7 recently retired instr at 0xfffffe00190bad70
CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 0: PC=0xfffffe0018f911f0, LR=0xfffffe0018f911f0, FP=0xfffffe307bbebd50
CORE 1: PC=0xfffffe0018f911f0, LR=0xfffffe0018f911f0, FP=0xfffffe30b50e3d50
CORE 2: PC=0xfffffe0018f911f0, LR=0xfffffe0018f911f0, FP=0xfffffe3087d13d50
CORE 3 is the one that panicked. Check the full backtrace for details.
CORE 4: PC=0xfffffe00190a7204, LR=0xfffffe00190a71e0, FP=0xfffffe3087ca3d50
CORE 5: PC=0xfffffe0018f91200, LR=0xfffffe00194bb268, FP=0xfffffe30b596bdb0
CORE 6: PC=0xfffffe0018f91200, LR=0xfffffe00194bb268, FP=0xfffffe30b5a1bdb0
CORE 7: PC=0x00000001034415bc, LR=0x0000000103441564, FP=0x000000016d7890d0
Panicked task 0xfffffe166ecfbac8: 4385 pages, 15 threads: pid 5061: python3.9
Panicked thread: 0xfffffe166ef73960, backtrace: 0xfffffe30b6a536d0, tid: 25548
		  lr: 0xfffffe0018f96b68  fp: 0xfffffe30b6a53740
		  lr: 0xfffffe0018f9694c  fp: 0xfffffe30b6a537b0
		  lr: 0xfffffe00190c01c8  fp: 0xfffffe30b6a537d0
		  lr: 0xfffffe00190b1674  fp: 0xfffffe30b6a53880
		  lr: 0xfffffe0018f4f7e8  fp: 0xfffffe30b6a53890
		  lr: 0xfffffe0018f965dc  fp: 0xfffffe30b6a53c20
		  lr: 0xfffffe0018f965dc  fp: 0xfffffe30b6a53c90
		  lr: 0xfffffe0019748e80  fp: 0xfffffe30b6a53cb0
		  lr: 0xfffffe0018fab460  fp: 0xfffffe30b6a53ce0
		  lr: 0xfffffe0018fab2cc  fp: 0xfffffe30b6a53d00
		  lr: 0xfffffe0018f91174  fp: 0xfffffe30b6a53d50
		  lr: 0xfffffe00194bb268  fp: 0xfffffe30b6a53db0
		  lr: 0xfffffe001959e0e8  fp: 0xfffffe30b6a53e40
		  lr: 0xfffffe00190b1350  fp: 0xfffffe30b6a53ef0
		  lr: 0xfffffe0018f4f7e8  fp: 0xfffffe30b6a53f00

last started kext at 694061903: com.apple.driver.AppleHIDKeyboard	224 (addr 0xfffffe00186a0000, size 16384)
loaded kexts:
com.apple.driver.AppleTopCaseHIDEventDriver	4050.1
com.apple.filesystems.autofs	3.0
com.apple.fileutil	20.036.15
com.apple.iokit.IOBluetoothSerialManager	8.0.5d7
com.apple.driver.AppleBiometricServices	1
com.apple.driver.usb.realtek8153patcher	5.0.0
com.apple.driver.usb.AppleUSBHostBillboardDevice	1.0
com.apple.driver.DiskImages.ReadWriteDiskImage	493.0.0
com.apple.driver.DiskImages.UDIFDiskImage	493.0.0
com.apple.driver.DiskImages.RAMBackingStore	493.0.0
com.apple.driver.DiskImages.FileBackingStore	493.0.0
com.apple.iokit.SCSITaskUserClient	436.140.1
com.apple.filesystems.apfs	1677.141.1
com.apple.driver.BCMWLANFirmware4378.Hashstore	1
com.apple.driver.CoreKDL	1
com.apple.driver.SEPHibernation	1
com.apple.driver.AppleUSBDeviceNCM	5.0.0
com.apple.driver.AppleSmartBatteryManager	161.0.0
com.apple.driver.AppleThunderboltIP	4.0.3
com.apple.driver.AppleALSColorSensor	1.0.0d1
com.apple.driver.AppleAOPVoiceTrigger	11.5
com.apple.driver.AppleFileSystemDriver	3.0.1
com.apple.nke.l2tp	1.9
com.apple.filesystems.tmpfs	1
com.apple.IOTextEncryptionFamily	1.0.0
com.apple.filesystems.hfs.kext	556.100.11
com.apple.security.BootPolicy	1
com.apple.BootCache	40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib	1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless	1.0.0d1
com.apple.driver.ApplePMP	1
com.apple.driver.AppleSmartIO2	1
com.apple.driver.ApplePMPFirmware	1
com.apple.AppleEmbeddedSimpleSPINORFlasher	1
com.apple.driver.AppleCS42L83Audio	442.26
com.apple.driver.AppleSPMIPMU	1.0.1
com.apple.driver.AppleDPDisplayTCON	1
com.apple.driver.AppleTAS5770LAmp	442.26
com.apple.driver.AppleT8020SOCTuner	1
com.apple.driver.AppleT8103CLPCv3	1
com.apple.AGXG13G	173.28.7
com.apple.driver.AppleAVD	385
com.apple.driver.AppleAVE2	401.73.4
com.apple.driver.AppleJPEGDriver	4.6.0
com.apple.driver.AppleMobileDispH13G-DCP	140.0
com.apple.driver.usb.AppleUSBHostT8103	1
com.apple.driver.AudioDMAController-T8103	1.60.5
com.apple.driver.AppleS5L8960XNCO	1
com.apple.driver.AppleT8103PMGR	1
com.apple.driver.AppleS8000AES	1
com.apple.driver.AppleS8000DWI	1.0.0d1
com.apple.driver.AppleS5L8960XWatchDogTimer	1
com.apple.driver.AppleInterruptController	1.0.0d1
com.apple.driver.AppleT8020DART	1
com.apple.driver.AppleBluetoothModule	1
com.apple.driver.AppleSamsungSerial	1.0.0d1
com.apple.driver.AppleBCMWLANBusInterfacePCIe	1
com.apple.driver.AppleS5L8920XPWM	1.0.0d1
com.apple.driver.AppleS5L8940XI2C	1.0.0d2
com.apple.driver.AppleSPIMC	1
com.apple.driver.AppleT8101	1
com.apple.driver.AppleM68Buttons	1.0.0d1
com.apple.iokit.IOUserEthernet	1.0.1
com.apple.driver.usb.AppleUSBUserHCI	1
com.apple.iokit.IOKitRegistryCompatibility	1
com.apple.iokit.EndpointSecurity	1
com.apple.driver.AppleDiskImages2	1
com.apple.AppleSystemPolicy	2.0.0
com.apple.nke.applicationfirewall	311
com.apple.kec.InvalidateHmac	1
com.apple.driver.AppleHIDKeyboard	224
com.apple.driver.AppleActuatorDriver	4440.3
com.apple.driver.AppleMultitouchDriver	4440.3
com.apple.driver.AppleHSBluetoothDriver	4050.1
com.apple.driver.IOBluetoothHIDDriver	8.0.5d7
com.apple.kext.triggers	1.0
com.apple.iokit.IOAVBFamily	940.4
com.apple.plugin.IOgPTPPlugin	985.2
com.apple.iokit.IOEthernetAVBController	1.1.0
com.apple.driver.AppleMesaSEPDriver	100.99
com.apple.iokit.IOBiometricFamily	1
com.apple.driver.usb.cdc.ecm	5.0.0
com.apple.driver.usb.cdc.acm	5.0.0
com.apple.driver.usb.serial	6.0.0
com.apple.driver.DiskImages.KernelBacked	493.0.0
com.apple.driver.usb.cdc	5.0.0
com.apple.driver.AppleUSBAudio	405.39
com.apple.iokit.IOAudioFamily	300.6.1
com.apple.vecLib.kext	1.2.0
com.apple.driver.AppleXsanScheme	3
com.apple.driver.AppleSEPHDCPManager	1.0.1
com.apple.driver.AppleTrustedAccessory	1
com.apple.iokit.AppleSEPGenericTransfer	1
com.apple.driver.IOBluetoothHostControllerPCIeTransport	8.0.5d7
com.apple.iokit.IOBluetoothHostControllerTransport	8.0.5d7
com.apple.driver.AppleConvergedIPCOLYBTControl	1
com.apple.driver.AppleConvergedPCI	1
com.apple.driver.AppleBluetoothDebug	1
com.apple.driver.usb.networking	5.0.0
com.apple.driver.AppleBTM	1.0.1
com.apple.driver.AppleThunderboltDPInAdapter	8.1.4
com.apple.driver.AppleThunderboltDPAdapterFamily	8.1.4
com.apple.driver.AppleThunderboltUSBDownAdapter	1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter	4.1.1
com.apple.driver.AppleAOPAudio	16.2
com.apple.nke.ppp	1.9
com.apple.driver.AppleHIDTransportSPI	4400.35
com.apple.driver.AppleHIDTransport	4400.35
com.apple.driver.AppleInputDeviceSupport	4400.35
com.apple.driver.AppleDCPDPTXProxy	1.0.0
com.apple.driver.DCPDPFamilyProxy	1
com.apple.driver.AppleBSDKextStarter	3
com.apple.filesystems.hfs.encodings.kext	1
com.apple.driver.AppleSPU	1
com.apple.AGXFirmwareKextG13GRTBuddy	173.28.7
com.apple.AGXFirmwareKextRTBuddy64	173.28.7
com.apple.iokit.IONVMeFamily	2.1.0
com.apple.driver.AppleDiagnosticDataAccessReadOnly	1.0.0
com.apple.driver.AppleNANDConfigAccess	1.0.0
com.apple.driver.AppleHPM	3.4.4
com.apple.driver.AppleCSEmbeddedAudio	442.26
com.apple.driver.AppleDialogPMU	1.0.1
com.apple.driver.AppleEmbeddedAudio	442.26
com.apple.iokit.AppleARMIISAudio	80.34.1
com.apple.driver.AppleStockholmControl	1.0.0
com.apple.driver.ApplePassthroughPPM	3.0
com.apple.iokit.IOGPUFamily	21.11
com.apple.driver.AppleH11ANEInterface	4.75.0
com.apple.driver.AppleH13CameraInterface	3.58.0
com.apple.driver.AppleH10PearlCameraInterface	16.15.1
com.apple.driver.DCPAVFamilyProxy	1
com.apple.iokit.IOMobileGraphicsFamily-DCP	343.0.0
com.apple.driver.AppleDCP	1
com.apple.driver.AppleFirmwareKit	1
com.apple.iokit.IOMobileGraphicsFamily	343.0.0
com.apple.driver.AppleSPMI	1.0.1
com.apple.driver.AppleUSBXDCIARM	1.0
com.apple.driver.AppleUSBXDCI	1.0
com.apple.iokit.IOUSBDeviceFamily	2.0.0
com.apple.driver.usb.AppleUSBXHCIARM	1
com.apple.driver.usb.AppleUSBXHCI	1.2
com.apple.driver.AppleEmbeddedUSBHost	1
com.apple.driver.usb.AppleUSBHub	1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice	1.2
com.apple.driver.AppleT8103TypeCPhy	1
com.apple.driver.AppleSART	1
com.apple.driver.ApplePMGR	1
com.apple.driver.watchdog	1
com.apple.driver.usb.AppleUSBHostPacketFilter	1.0
com.apple.driver.AppleDisplayCrossbar	1.0.0
com.apple.iokit.IODisplayPortFamily	1.0.0
com.apple.driver.AppleTypeCPhy	1
com.apple.driver.AppleThunderboltNHI	7.2.8
com.apple.driver.AppleT8103PCIeC	1
com.apple.iokit.IOThunderboltFamily	9.3.2
com.apple.driver.ApplePIODMA	1
com.apple.driver.AppleT8103PCIe	1
com.apple.driver.AppleMultiFunctionManager	1
com.apple.driver.AppleEmbeddedPCIE	1
com.apple.driver.AppleBluetoothDebugService	1
com.apple.driver.AppleBCMWLANCore	1.0.0
com.apple.iokit.IO80211FamilyV2	1200.12.2b1
com.apple.driver.IOImageLoader	1.0.0
com.apple.driver.corecapture	1.0.4
com.apple.driver.AppleMCA2-T8103	540.10
com.apple.driver.AppleGPIOICController	1.0.2
com.apple.driver.AppleFireStormErrorHandler	1
com.apple.driver.AppleMobileApNonce	1
com.apple.iokit.IOTimeSyncFamily	985.2
com.apple.driver.DiskImages	493.0.0
com.apple.iokit.IOGraphicsFamily	585.1
com.apple.iokit.IOBluetoothFamily	8.0.5d7
com.apple.iokit.IOBluetoothPacketLogger	8.0.5d7
com.apple.driver.FairPlayIOKit	68.7.1
com.apple.iokit.CoreAnalyticsFamily	1
com.apple.driver.AppleSSE	1.0
com.apple.driver.AppleSEPKeyStore	2
com.apple.driver.AppleUSBTDM	511.141.1
com.apple.iokit.IOUSBMassStorageDriver	184.140.2
com.apple.iokit.IOPCIFamily	2.9
com.apple.iokit.IOSCSIBlockCommandsDevice	436.140.1
com.apple.iokit.IOSCSIArchitectureModelFamily	436.140.1
com.apple.driver.AppleIPAppender	1.0
com.apple.driver.AppleFDEKeyStore	28.30
com.apple.driver.AppleEffaceableStorage	1.0
com.apple.driver.AppleCredentialManager	1.0
com.apple.driver.KernelRelayHost	1
com.apple.iokit.IOUSBHostFamily	1.2
com.apple.driver.AppleUSBHostMergeProperties	1.2
com.apple.driver.usb.AppleUSBCommon	1.0
com.apple.driver.AppleSMC	3.1.9
com.apple.driver.RTBuddy	1.0.0
com.apple.driver.AppleEmbeddedTempSensor	1.0.0
com.apple.driver.AppleARMPMU	1.0
com.apple.iokit.IOAccessoryManager	1.0.0
com.apple.driver.AppleOnboardSerial	1.0
com.apple.iokit.IOSkywalkFamily	1
com.apple.driver.mDNSOffloadUserClient	1.0.1b8
com.apple.iokit.IONetworkingFamily	3.4
com.apple.iokit.IOSerialFamily	11
com.apple.driver.AppleSEPManager	1.0.1
com.apple.driver.AppleA7IOP	1.0.2
com.apple.driver.IOSlaveProcessor	1
com.apple.driver.AppleBiometricSensor	2
com.apple.iokit.IOHIDFamily	2.0.0
com.apple.AUC	1.0
com.apple.iokit.IOAVFamily	1.0.0
com.apple.iokit.IOHDCPFamily	1.0.0
com.apple.iokit.IOCECFamily	1
com.apple.iokit.IOAudio2Family	1.0
com.apple.driver.AppleEmbeddedAudioLibs	1.17
com.apple.driver.AppleFirmwareUpdateKext	1
com.apple.driver.AppleM2ScalerCSCDriver	265.0.0
com.apple.iokit.IOSurface	290.8.1
com.apple.driver.IODARTFamily	1
com.apple.security.quarantine	4
com.apple.security.sandbox	300.0
com.apple.kext.AppleMatch	1.0.0d1
com.apple.driver.AppleMobileFileIntegrity	1.0.5
com.apple.kext.CoreTrust	1
com.apple.security.AppleImage4	3.0.0
com.apple.iokit.IOCryptoAcceleratorFamily	1.0.1
com.apple.driver.AppleARMPlatform	1.0.2
com.apple.iokit.IOStorageFamily	2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily	1.0.0
com.apple.iokit.IOReportFamily	47
com.apple.kec.pthread	1
com.apple.kec.corecrypto	11.1
com.apple.kec.Libm	1



** Stackshot Succeeded ** Bytes Traced 341829 (Uncompressed 886656) **

SciPy/NumPy/Python version information

1.8.0.dev0+1675.774941b 1.22.0.dev0+949.ga90677a0b sys.version_info(major=3, minor=9, micro=6, releaselevel=‘final’, serial=0)

Issue Analytics

State:
Created 2 years ago
Reactions:2
Comments:62 (52 by maintainers)

Top GitHub Comments

2reactions

ogriselcommented, Sep 3, 2021

Just in case, I archived a copy of the numpy and scipy nightly build wheels here:

https://github.com/ogrisel/tmpblobs/releases/tag/m1-dev-wheels

1reaction

sunilshahcommented, Oct 21, 2021

@ogrisel

Yes, it is feedback assistant bug report. The example referred in my post is a very simple spin loop checking time for one second at a time. So, it should be easy for the dev team to see the issue. Kernel dumps should get quicker response (potential for zero-day security bug). Expect to get MacOS12 in less than a week. Let us see if it has been fixed.