PostgresqlDatabase multithreading performance problem

Hello, I have a postgresql db connection:

database = PostgresqlDatabase(config_server['database_name'],
                              **{'host': config_server['host'], 'port': config_server['port'],
                                 'user': config_server['user'],
                                 'password': config_server['password']
                                 })

and each thread(4 thread) runs the below function with different date_time parameter, we must run the below function 100 times:

def get_estimations_v1(date_time, db) -> pd.DataFrame:
    Estimation._meta.set_table_name('est_' + date_time.strftime("%Y%m%d"))

    try:
        db.connect(reuse_if_open=True)
        query = Estimation.select().where(
            Estimation.date_time.between(date_time - timedelta(minutes=1), date_time))

         estimations = pd.DataFrame(list(query.dicts()))
         return estimations

    except Exception as ce:
        logger.error('load of estimations failed for {}, {}', date_time, str(ce))
        return
    finally:
        db.close()
        return

but the performance is worse than sequential running. anyone have any ideas?

[coleifer]

I made a little script that inserts 100K rows and queries it single-threaded, using 4 threads and 4 processes. The single-thread and 4-thread run about the same, but the multiprocess version runs in about half the time.

The psycopg2-only version is faster, but this is somewhat expected since it carries none of the python-side overhead. It is also marginally faster (10%) when run multi-threaded.

I profiled the code and it looks to me like the bulk of the time in peewee is spent generating sql and serializing result rows to model instances, which is to be expected.

[mohamin1995]

No, single-threaded version of psycopg2 almost takes time equal to single-threaded peewee.