Stuck on an issue?
Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Segfault in Z3 API for some inputs
See original GitHub issueSome of the recent Travis build failures indicate failed synthesis jobs with no further explanation. It appears that the synthesis jobs are segfaulting on Linux. In a Fedora 32 virtual machine, cozy examples/lsort.ds exhibits the segfault consistently. From dmesg:
[ 4969.897438] cozy[5274]: segfault at 0 ip 00007f8f339a3830 sp 00007ffdd6047460 error 4 in libffi.so.6.0.2[7f8f3399f000+5000]
[ 4969.897447] Code: ff ff 48 89 c7 48 8d 04 c1 48 8b 4d 88 01 f7 4a 89 04 e9 89 7d 84 49 83 c5 01 4c 39 6d 90 74 69 66 2e 0f 1f 84 00 00 00 00 00 <4b> 8b 3c ec 31 d2 4c 89 f1 4c 89 fe 4c 89 45 98 e8 eb f5 ff ff 4c
The dmesg output indicates that this is a null pointer dereference (segfault at 0). More info about segfault error codes.
After a great deal of work, I was able to obtain a stacktrace:
#0 0x00007fffea605830 in ffi_closure_unix64_inner () from /lib64/libffi.so.6
#1 0x00007fffea605c58 in ffi_closure_unix64 () from /lib64/libffi.so.6
#2 0x00007fffe87e9677 in Z3_model_get_func_interp ()
#3 0x00007fffea605af0 in ffi_call_unix64 () from /lib64/libffi.so.6
#4 0x00007fffea6052ab in ffi_call () from /lib64/libffi.so.6
#5 0x00007fffea38e4bc in _ctypes_callproc.cold ()
#6 0x00007fffea397fd7 in PyCFuncPtr_call ()
#7 0x00007ffff7bc3823 in _PyObject_MakeTpCall () from /lib64/libpython3.8.so.1.0
#8 0x00007ffff7bc0948 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#9 0x00007ffff7bba231 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#10 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#11 0x00007ffff7bbb744 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#12 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#13 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#14 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#15 0x00007ffff7bfff80 in call_method () from /lib64/libpython3.8.so.1.0
#16 0x00007ffff7bffd7c in slot_mp_subscript () from /lib64/libpython3.8.so.1.0
#17 0x00007ffff7bbbf1c in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#18 0x00007ffff7bba8d5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#19 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#20 0x00007ffff7bbb744 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#21 0x00007ffff7bba8d5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#22 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#23 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#24 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#25 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#26 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#27 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#28 0x00007ffff7bd8081 in gen_send_ex () from /lib64/libpython3.8.so.1.0
#29 0x00007ffff7c263c5 in chain_next () from /lib64/libpython3.8.so.1.0
#30 0x00007ffff7bbbd4b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#31 0x00007ffff7bba8d5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#32 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#33 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#34 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#35 0x00007ffff7bd17ec in method_vectorcall () from /lib64/libpython3.8.so.1.0
#36 0x00007ffff7bd1a90 in PyVectorcall_Call () from /lib64/libpython3.8.so.1.0
#37 0x00007ffff7bbd736 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#38 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#39 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#40 0x00007ffff7bba231 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#41 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#42 0x00007ffff7bd1603 in method_vectorcall () from /lib64/libpython3.8.so.1.0
#43 0x00007ffff7bbc60a in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#44 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#45 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#46 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#47 0x00007ffff7bc3247 in _PyObject_FastCallDict () from /lib64/libpython3.8.so.1.0
#48 0x00007ffff7bcf0bf in slot_tp_init () from /lib64/libpython3.8.so.1.0
#49 0x00007ffff7bc3afb in type_call () from /lib64/libpython3.8.so.1.0
#50 0x00007ffff7bc3823 in _PyObject_MakeTpCall () from /lib64/libpython3.8.so.1.0
#51 0x00007ffff7bc036c in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#52 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#53 0x00007ffff7bc04d7 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#54 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#55 0x00007ffff7bc04d7 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#56 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#57 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#58 0x00007ffff7bc92b8 in function_code_fastcall () from /lib64/libpython3.8.so.1.0
#59 0x00007ffff7bbbb3b in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#60 0x00007ffff7bba8d5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#61 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#62 0x00007ffff7bbb744 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#63 0x00007ffff7bba8d5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#64 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#65 0x00007ffff7bbc60a in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#66 0x00007ffff7bba8d5 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#67 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#68 0x00007ffff7bbb744 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#69 0x00007ffff7bba231 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#70 0x00007ffff7c351c9 in PyEval_EvalCodeEx () from /lib64/libpython3.8.so.1.0
#71 0x00007ffff7c3518b in PyEval_EvalCode () from /lib64/libpython3.8.so.1.0
#67 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () f
#72 0x00007ffff7c38ddc in builtin_exec () from /lib64/libpython3.8.so.1.0
#73 0x00007ffff7bc956b in cfunction_vectorcall_FASTCALL () from /lib64/libpython3.8.so.1.0
#74 0x00007ffff7bbb744 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#75 0x00007ffff7bba231 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#76 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#77 0x00007ffff7bbb744 in _PyEval_EvalFrameDefault () from /lib64/libpython3.8.so.1.0
#78 0x00007ffff7bba231 in _PyEval_EvalCodeWithName () from /lib64/libpython3.8.so.1.0
#79 0x00007ffff7bc8f59 in _PyFunction_Vectorcall () from /lib64/libpython3.8.so.1.0
#80 0x00007ffff7bd1a90 in PyVectorcall_Call () from /lib64/libpython3.8.so.1.0
#81 0x00007ffff7c52b90 in pymain_run_module () from /lib64/libpython3.8.so.1.0
#82 0x00007ffff7c51f24 in Py_RunMain () from /lib64/libpython3.8.so.1.0
#83 0x00007ffff7c28aa9 in Py_BytesMain () from /lib64/libpython3.8.so.1.0
#84 0x00007ffff7e0e042 in __libc_start_main () from /lib64/libc.so.6
#85 0x000055555555507e in _start ()
It looks like Z3 is the culprit (Z3_model_get_func_interp), but experience has shown me that it is much more likely to be Cozy’s fault somehow.
Notes:
- The segfault only seems to happen in child jobs after multiprocessing forks them off. This makes it very difficult to catch in a debugger.
- Bug reports elsewhere (e.g. JohannesBuchner/PyMultiNest#5) suggest that this might be due to Python’s garbage collector reclaiming some object that the native Z3 “ctypes” API needs, but I have very little evidence right now.
- This issue arose suddenly, without changes in Cozy. Either this is a new bug in one of Cozy’s dependencies or it has been a longstanding latent bug that is now revealed by a dependency update. I have ruled out Z3 as the culprit: the last successful Travis build used Z3 4.8.7.0, and I can still reproduce the segfault with that version.
Issue Analytics
- State:
- Created 3 years ago
- Comments:6
Top Results From Across the Web
Segmentation fault using the Z3 API with C++ - Stack Overflow
I have a project that generated big SMT-LIB v2 problems as strings and it solved them by writing the problems to a file...
Read more >File: README — Documentation for z3 (0.0.20221020)
As Z3 is a C library, doing anything weird with it will segfault your process. Ruby API tries its best to prevent such...
Read more >Exploit Development (z3-solver) - Medium
In this article, I'll describe how to analyze bugs and how to make exploits of well-known SMT (Satisfiability Modulo Theories) Solver named ...
Read more >z3/RELEASE_NOTES · master · Damián Barsotti / pef · GitLab
Now, it is possible to set parameter for Z3 internal modules. Several parameter names changed. Execute `z3 -p` for the new parameter list....
Read more >Automatically Testing String Solvers
widely-used SMT solvers, Z3-seq [7, 15], Z3str3 [13], and ... segmentation fault. ... input formulas by replacing some of the free variables in...
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
This problem is beyond my ability and patience to debug. Since it is due to memory corruption, the actual bug is probably very far from where the process eventually crashes.
Ultimately I think this is due to some subtlety of
fork()that Cozy or its dependencies does not handle properly. #119 seems to fix this problem by moving away fromfork(), so I am closing this issue.No, that doesn’t seem to fix the problem. I have no new hypothesis, but I do have new information. The Cozy line I mentioned earlier isn’t the only place that can segfault: I am occasionally seeing one in
exp_wf, as the procedure returns and the Z3 context gets deallocated. That could indicate any number of problems (double-free, memory corruption, …), none of which are good.