Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Segfault possibly caused by changes to `MaxOut` in v8.1.1

See original GitHub issue

Problem

We have recently (since Sep 9th) been experiencing intermittent seg faults in our (Alibi’s) Windows CI. These occur in some of our tests that use the en_core_web_md pipeline.

We believe we have narrowed down the cause to the Tok2Vec component. Since our errors started on the same day v8.1.1 was released we are wondering if the changes to MaxOut are the cause (https://github.com/explosion/thinc/pull/702)? Or perhaps the move to blis v0.9 (https://github.com/explosion/thinc/pull/736)?

We’ve struggled to come up with a MWE, but for a comparison, we’ve repeated our CI 30x with v8.1.0 and 30x with v8.1.1. With v8.1.0 we experience no failures, whilst with v8.1.1 we experience seg faults >20% of the time.

Error traceback

From this CI workflow:

Windows fatal exception: access violation

Thread 0x00001a38 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 316 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 581 in wait
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\tqdm\_monitor.py", line 60 in run
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 980 in _bootstrap_inner
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\threading.py", line 937 in _bootstrap

Current thread 0x00000760 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\spacy\ml\staticvectors.py", line 56 in forward
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\model.py", line 291 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\layers\concatenate.py", line 44 in <listcomp>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\layers\concatenate.py", line 44 in forward
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\model.py", line 291 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\layers\chain.py", line 55 in forward
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\model.py", line 291 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\layers\chain.py", line 55 in forward
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\thinc\model.py", line 315 in predict
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\spacy\pipeline\tok2vec.py", line 125 in predict
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\spacy\language.py", line 1020 in __call__
  File "D:\a\alibi\alibi\alibi\explainers\tests\test_anchor_text.py", line 326 in test_lm_stopwords_punctuation
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\python.py", line 192 in pytest_pyfunc_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\python.py", line [176](https://github.com/SeldonIO/alibi/actions/runs/3025334987/jobs/4867687688#step:7:177)1 in runtest
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 166 in pytest_runtest_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 259 in <lambda>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 338 in from_call
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 258 in call_runtest_hook
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 219 in call_and_report
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 130 in runtestprotocol
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\runner.py", line 111 in pytest_runtest_protocol
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 347 in pytest_runtestloop
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 322 in _main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 268 in wrap_session
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\main.py", line 315 in pytest_cmdline_main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_callers.py", line 39 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_manager.py", line 80 in _hookexec
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\pluggy\_hooks.py", line 265 in __call__
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\config\__init__.py", line 164 in main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\site-packages\_pytest\config\__init__.py", line [187](https://github.com/SeldonIO/alibi/actions/runs/3025334987/jobs/4867687688#step:7:188) in console_main
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts\pytest.exe\__main__.py", line 7 in <module>
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 87 in _run_code
  File "C:\hostedtoolcache\windows\Python\3.9.13\x64\lib\runpy.py", line 197 in _run_module_as_main
D:\a\_temp\511c6489-14db-4df4-820f-7b5[188](https://github.com/SeldonIO/alibi/actions/runs/3025334987/jobs/4867687688#step:7:189)c57acc.sh: line 2:  1[224](https://github.com/SeldonIO/alibi/actions/runs/3025334987/jobs/4867687688#step:7:225) Segmentation fault      pytest -m "not tf1" alibi
..........................................XXX......ssss.......
Error: Process completed with exit code 139.

Platform, versions etc

  • Platform: Github runner image windows-2022, version 20220905.1
  • Python versions tested: 3.9.12, 3.9.13
  • pip env: see here

Issue Analytics

  • State:open
  • Created a year ago
  • Reactions:1
  • Comments:11 (4 by maintainers)

github_iconTop GitHub Comments

2reactions
svlandegcommented, Sep 27, 2022

Hi! We’ve released thinc 8.1.2 today which restricts blis to <0.8.0 again instead of <0.10.0. Thanks again for your report!

1reaction
shadeMecommented, Sep 30, 2022

Thanks for the quick response! Yeah, the OOB access is in one of the haswell kernels, which confirms our suspicion.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Issues · explosion/thinc - GitHub
Tracking issue for Apple M-series GPU support. #792 opened ; Segfault possibly caused by changes to MaxOut in v8.1.1 third-party Related to third-party...
Read more >
Identify what's causing segmentation faults (segfaults)
A segmentation fault (aka segfault) is a common condition that causes programs to crash; they are often associated with a file named core...
Read more >
Segmentation Fault when wrapping V8 in a class?
Thy to change v8::Context::Scope context_scope(context); from the constructor (line 134) to internal_executeString (before script ...
Read more >
Common Causes of Segmentation Faults (Segfaults)
A segmentation fault (often called a segfault) can occur if a program you are running attempts to access an invalid memory location.
Read more >
Segfault possibly caused by changes to `MaxOut` in v8.1.1 ...
We believe we have narrowed down the cause to the Tok2Vec component. Since our errors started on the same day v8.1.1 was released...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

Slow to load

Next page

from_dlpack in spacy-transformers