Implement multithreading with subscriptions (concurrent execution) like Flask routes

Hi! I am moving my code from a flask app to NATS. Flask, by default is multi-threaded and I am trying to do that as well with nats but have struggled to run co-routines concurrently. As far as I understand, when you execute a co-routine, it blocks the event loop until it is completed even if that co-routine contains I/O operations.

The flask app had two routes:

1. /train: Large task of training a ML model and takes time.
2. /predict: Small high priority task.

In my flask app, I could call the /predict route while /train was running and get the result immediately. However, with the asyncio approach in NATS, the predict callback does not even start until the train callback is completed. They are always executed sequentially. I have tried to execute the LARGE TASK and SMALL TASK as threads, but in Python you cannot get a return value of the thread unless you use a Queue. However, Queue is not a good option when you start threads randomly.

Is there a possible solution to execute multiple callbacks concurrently? I have attached a sample code to better understand the terms I am using to describe my problem.

     async def train(msg):
        """
        The bot train requests come here
        :return:
        """
        subject = msg.subject
        reply = msg.reply
        data_train = json.loads(msg.data.decode())
        routes.logger.info("Received a message on '{subject} {reply}': {data}".format(
            subject=subject, reply=reply, data=data_train))

        routes.logger.debug('subscribe/train: Train data: {}'.format(data_train))
        data = routes.train(data_train) // LARGE TASK
        routes.logger.info(data)
        await nc.publish(msg.reply, json.dumps(data).encode())
        subscriber = 'nlu.{}.predict.{}'.format(lang, data['model'].split('_')[1])
        if subscriber not in current_subscribers:
            current_subscribers[subscriber] = await nc.subscribe(subscriber, cb=predict)
            routes.logger.info('Added subscription: {}'.format(subscriber))

    async def predict(msg):
        """
        The bot predict requests come here
        :return:
        """
        subject = msg.subject
        reply = msg.reply
        data_predict = json.loads(msg.data.decode())
        routes.logger.info("Received a message on '{subject} {reply}': {data}".format(
            subject=subject, reply=reply, data=data_predict))
        data = routes.predict(data_predict) // SMALL TASK
        routes.logger.info(data)
        await nc.publish(msg.reply, json.dumps(data).encode())

await nc.subscribe('{}.train'.format(connection), cb=train)
await nc.subscribe('{}.predict'.format(connection), cb=predict)

[charliestrawn]

It’s an interesting idea and worth looking into, but IMO that brings a lot of potential complexity that should probably not live in this repo. Flask is a (micro) web framework, while this is a library for talking to a nats-server or clusters. I think we want to keep this repo simple, focused, and flexible. This allows consumers to wrap or extend it in their own projects or custom frameworks that handle things like threading.

[wallyqs]

Hi @shayan09 yes this is something that can look into in the near future, like binding each handler into its own thread pool executor maybe. One thing to consider as well when there are blocking issues is to wrap the NATS connection into its own component and with its own thread (something like: https://github.com/nats-io/nats.py/blob/master/examples/coroutine_threadsafe.py#L31) and then whenever receive a task use syncio.run_coroutine_threadsafe so that the client can be called from another thread.