Preventing Alembic entering undesired log entries?

Our log files have been flooded with undesired log entries that occur with each Pangres upsert that otherwise went perfectly:

2022-05-25 16:54:30 - Context impl PostgresqlImpl.
2022-05-25 16:54:30 - Will assume transactional DDL.

Was working fine for months but suddenly they’re everywhere. Think it might be caused by updating packages but not sure.

Had a search around and couldn’t find anything, but eventually found the culprit in /venv/lib64/python3.8/site-packages/alembic/runtime/migration.py on lines 201 onwards:

        log.info("Context impl %s.", self.impl.__class__.__name__)
        if self.as_sql:
            log.info("Generating static SQL")
        log.info(
            "Will assume %s DDL.",
            "transactional"
            if self.impl.transactional_ddl
            else "non-transactional",
        )

Our pangres code:

engine_string = "postgresql://username:pw@ip:port/dbname"
engine = sqlalchemy.create_engine(engine_string)
pangres.upsert(engine, df, "tablename", if_row_exists="update", add_new_columns=True)

I’m not a good programmer and don’t want to risk bigger problems by messing with packages’ code… is there anything we can do to disable this?

Also, we’re unsure if it’s most appropriate to ask about this here or over at Alembic. As it’s a Pangres function that ultimately causes it we thought Pangres most responsible but feel free to redirect us elsewhere if appropriate.

[ThibTrip]

Hi @chrisjdixon ! We use alembic for creating new columns or changing data types (this will not happen here because of the default value adapt_dtype_of_empty_db_columns=False).

I’ll see if I can reproduce the issue. Most likely having pangres.upsert add a column to an existing SQL table will do the trick. As for the solution I can already imagine it will be something like setting the logging level for alembic higher than INFO like in this example.

I need to research on that. I’ll keep you updated.

And yes you are right you should not mess with installed code. Here that won’t be necessary but in a similar situation where you’d need to tweak a library you could use a fork for instance.

[ThibTrip]

Sorry @chrisjdixon I forgot to get back to you! I was busy with work and then on holidays.

I tried venv and was able to reproduce the problem. While the log entries you described do not appear on the console they do appear in the log file. I put the code you posted on June 6th in a file ./test_pangres/test.py then ran these commands with python 3.8.13:

$ cd test_pangres
$ python -m venv .
$ source ./bin/activate
(test_pangres) $ python test.py

# console
2022-07-19 14:45:49,793 | INFO     | pangres    | logger:log:73 - Added column example.test (type: BOOLEAN) in table example (schema="None")

# alembic_logging.log
2022-07-19 14:45:36 - Context impl SQLiteImpl.
2022-07-19 14:45:36 - Will assume non-transactional DDL.
2022-07-19 14:45:36 - Added column example.test (type: BOOLEAN) in table example (schema="None")

I was able to remove the two first lines in alembic_logging.log by changing the logging level in the basic configuration to only WARNING. I suppose that alembic detects that there is a logging configuration and hooks itself to it.

Before

logging.basicConfig(filename=‘alembic_logging.log’, level=logging.INFO, format = ‘%(asctime)s - %(message)s’, datefmt=“%Y-%m-%d %H:%M:%S”)

After

logging.basicConfig(filename=‘alembic_logging.log’, level=logging.WARNING, format = ‘%(asctime)s - %(message)s’, datefmt=“%Y-%m-%d %H:%M:%S”)