Segmentation fault when using threading

• Linux version: 64-bit Ubuntu

• PyQ version: PyQ 4.1.3 NumPy 1.13.3 KDB+ 3.5 (2018.02.26) l64 Python 2.7.12 (default, Dec 4 2017, 14:50:18) [GCC 5.4.0 20160609]

• kdb+ version: 64-bit kdb+ QLIC=/home/zakariyya/q/q64

• Not install via virtual environment

• QHOME=/home/zakariyya/q/q64

• Not using Conda

Hi guys,

I’m getting a segmentation fault when running a q process that loads in a Python script (.p) The use-case if fairly simple. I’m exposing Python subscription and unsubscription functions for Redis to q, then I call them so that they can run in a separate thread. That means that q can still work in “parallel”

$ cat redis.q
system "l /home/zakariyya/redis_thread.p";
.z.pc:{ if[x ~ hndl; -2" lost handle to tp"; if[`res in key `.;unsubscribe_redis enlist (res)] ] };
.z.exit:{ if[`res in key `.;unsubscribe_redis enlist (res)]; };
hndl:hopen 6006;
upd:{[t;d] data:d; neg[hndl](`upd;`tbl; data)};
show .z.i
$ q redis.q
KDB+ 3.5 2018.02.26 Copyright (C) 1993-2018 Kx Systems
l64/ 8()core 15999MB zakariyya zakariyya-pc-2193 127.0.1.1 EXPIRE 2018.07.01 zak********* KOD #51578

3645i
q)res:subscribe_redis("quote*";"quote")

And the Python script

$ cat redis_thread.p
import threading
import sys
import time
import redis
import signal
import os
from functools import partial
from pyq import q, K, _k

class PublisherThread(threading.Thread):

    def __init__(self, channel, r, kdb_table):
        super(PublisherThread, self).__init__()
        self._stopper = threading.Event()
        self.channel = channel
        self.kdb_table = kdb_table
        self.redis = r
        self.pubsub = self.redis.pubsub()
        self.redis.client_setname("kdb-feed-" + self.kdb_table)
        self.pubsub.psubscribe(self.channel)

    def stop(self):
        print('Closing Redis connection...')
        self.pubsub.close()
        self._stopper = True

    @property
    def stopped(self):
        return self._stopper.isSet()

    def run(self):
        while not self.stopped:
            try:
                msg = self.pubsub.get_message()
                if msg:
                    if msg['type'] in ('message', 'pmessage'):
                        #print(msg)
                        qmsg = K.string(msg)
                        q('upd', self.kdb_table, qmsg)
                        time.sleep(0.001)
            except _k.error as e:
                print('Caught Q error. Cannot insert data to table')
                self.stop()
            except Exception as e:
                print('Received unhandled exception. Cannot insert data to table')
                self.stop()


class RedisManager(object):

    def __init__(self, subscriber_init):
        self._subscribers_store = {}
        self.subscriber_init = subscriber_init

    def add(self, feed, kdb_table):
       print('Subscribing')
        key = ':'.join([feed, kdb_table])
        self._subscribers_store[key] = self.subscriber_init(feed, kdb_table)
        return key

    def remove(self, key):
        self._subscribers_store[key].stop()
        del self._subscribers_store[key]
        return True


def subscriber_init(feed, kdb_table, redis_client):
    t = PublisherThread(feed, redis_client, kdb_table)
    t.start()
    return t


# Create curried function of RedisManager's init
redis_manager = RedisManager(
    partial(subscriber_init,
         redis_client=redis.StrictRedis(host='XX.XXX.XXX.XXX', port=6399, db=0)))

# Create and expose Python functions a q callables

def q_subscribe_redis(feed, kdb_table):
    return redis_manager.add(str(feed), str(kdb_table))

def q_unsubscribe_redis(key):
    return redis_manager.remove(str(key))


q.subscribe_redis = q_subscribe_redis
q.unsubscribe_redis = q_unsubscribe_redis

And the segmentation fault:

q)Sorry, this application or an associated library has encountered a fatal error and will exit.
If known, please email the steps to reproduce this error to [e-mail]
with a copy of the kdb+ startup banner.
Thank you.
/home/zakariyya/q/q64/l64/q() [0x47a8b1]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fc0aa739390]
/home/zakariyya/q/q64/l64/q(r0+0) [0x41bd40]
/home/zakariyya/q/q64/l64/q() [0x408a9c]
/home/zakariyya/q/q64/l64/q() [0x40fb6f]
/home/zakariyya/q/q64/l64/q() [0x4042c8]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fc0aa37e830]
/home/zakariyya/q/q64/l64/q() [0x4043a1]
rlwrap: warning: q crashed, killed by SIGSEGV (core dumped).
rlwrap itself has not crashed, but for transparency,
it will now kill itself with the same signal


warnings can be silenced by the --no-warnings (-n) option
Segmentation fault (core dumped)

[abalkin]

… and finally, I have a proof that this issue has nothing to do with PyQ:

$ q -s 4
KDB+ 3.5 2018.04.25 Copyright (C) 1993-2018 Kx Systems
m32/ 20()core 65536MB **** NONEXPIRE

q)h:neg hopen 6006
q)upd:{h({n+::x};1)}
q)upd peach til 100000
[1]    74350 segmentation fault (core dumped)  q -s 4

(on the remote end, n got incremented to 4)

I am reassigning this issue to @awilson-kx to see what the Kx team has to say about this behavior.

For completeness, the back trace is

$ lldb -f $VIRTUAL_ENV/q/m32/q -c /cores/core.74350
(lldb) target create "/Users/a/.virtualenvs/3/q/m32/q" --core "/cores/core.74350"
Core file '/cores/core.74350' (i386) was loaded.
(lldb) process status
Process 0 stopped
* thread #1, stop reason = signal SIGSTOP
    frame #0: 0xa754745e libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
->  0xa754745e <+10>: retl
    0xa754745f <+11>: nop

libsystem_kernel.dylib`semaphore_wait_signal_trap:
    0xa7547460 <+0>:  movl   $0xffffffdb, %eax         ; imm = 0xFFFFFFDB
    0xa7547465 <+5>:  calll  0xa7552f64                ; _sysenter_trap
  thread #2, stop reason = signal SIGSTOP
    frame #0: 0x0000af67 q`___lldb_unnamed_symbol61$$q + 951
q`___lldb_unnamed_symbol61$$q:
->  0xaf67 <+951>: movl   (%eax), %eax
    0xaf69 <+953>: testl  %eax, %eax
    0xaf6b <+955>: jne    0xaf1b                    ; <+875>
    0xaf6d <+957>: jmp    0xad0d                    ; <+349>
  thread #3, stop reason = signal SIGSTOP
    frame #0: 0x0002011a q`r0 + 10
q`r0:
->  0x2011a <+10>: movl   0x4(%eax), %edx
    0x2011d <+13>: testl  %edx, %edx
    0x2011f <+15>: je     0x20140                   ; <+48>
    0x20121 <+17>: movl   0x82ffe(%ebx), %ecx
  thread #4, stop reason = signal SIGSTOP
    frame #0: 0xa754745e libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
->  0xa754745e <+10>: retl
    0xa754745f <+11>: nop

libsystem_kernel.dylib`semaphore_wait_signal_trap:
    0xa7547460 <+0>:  movl   $0xffffffdb, %eax         ; imm = 0xFFFFFFDB
    0xa7547465 <+5>:  calll  0xa7552f64                ; _sysenter_trap
  thread #5, stop reason = signal SIGSTOP
    frame #0: 0xa754745e libsystem_kernel.dylib`semaphore_wait_trap + 10
libsystem_kernel.dylib`semaphore_wait_trap:
->  0xa754745e <+10>: retl
    0xa754745f <+11>: nop

libsystem_kernel.dylib`semaphore_wait_signal_trap:
    0xa7547460 <+0>:  movl   $0xffffffdb, %eax         ; imm = 0xFFFFFFDB
    0xa7547465 <+5>:  calll  0xa7552f64                ; _sysenter_trap

[zak-b2c2]

Thanks a lot for taking the time to look into this, but also glad to know I was going crazy 😅

I’ve actually already implemented the main thread solution, which funnily enough needs to make sync calls to the “Tickerplant”, otherwise the packets get thrown God knows where. I guess the split thread solution was an interesting approach to having q and Python run in parallel to manage their bits together (Python for Redis management, q for IPC and reconnecting to TP…)

Interestingly, the processes are still running on the AWS RHEL 7.4 boxes after now 10 hours (will leave them overnight but looks like they’ve passed the threshold of 10,000 cycles).