Weird DB failure after upgrade to django-tenants 3.4.4 & Django 4.1

I am using containerized Postgres as my database:

centos/postgresql-12-centos7 latest ead3e66c2b54 12 months ago 372MB

this hasn’t changed recently and was working fine in previous commits.

After upgrade from django-tenants 3.4.3 to 3.4.4 (and other updates to the app which gets split into tenants, including upgrade to Django 4.1) I am seeing a failure in my automated tests:

psycopg2.errors.RaiseException: Action: PRIVS: Sequences  Diagnostics: line=SQL statement "GRANT SELECT ON subscriber.management_build_build_id_seq TO "kiwi";"
PL/pgSQL function clone_schema(text,text,boolean,boolean) line 606 at EXECUTE. 42P01. relation "subscriber.management_build_build_id_seq" does not exist
CONTEXT:  PL/pgSQL function clone_schema(text,text,boolean,boolean) line 669 at RAISE

Full logs: https://github.com/kiwitcms/tenants/runs/7725825769?check_suite_focus=true

Test which triggers this failure: https://github.com/kiwitcms/tenants/blob/master/tcms_tenants/tests/test_views.py#L117

which fails here https://github.com/kiwitcms/tenants/blob/master/tcms_tenants/utils.py#L93 - executing clone_tenant command!

[SrNetoChan]

Sequences are used to create primary keys. In most recent versions of PostgreSQL, there’s a new thing called identifiers thwt no longer make use of sequences, so maybe django started to use them instead. But it’s hard to tell without having a look into the database.

[atodorov]

More info:

When using Django 4.1 to initialize the database this snippet returns 0 sequences

  -- Create sequences
  action := 'Sequences';
  cnt := 0;
  -- TODO: Find a way to make this sequence's owner is the correct table.
  FOR object IN
    SELECT sequence_name::text
      FROM information_schema.sequences
     WHERE sequence_schema = quote_ident(source_schema)
  LOOP

as a result 0 new sequences are created when cloning. While this next snippet returns quite a lot of records

  -- MV: PRIVS: sequences
  action := 'PRIVS: Sequences';
  cnt := 0;
  FOR arec IN
    SELECT 'GRANT ' || p.perm::perm_type || ' ON ' || quote_ident(dest_schema) || '.' || t.relname::text || ' TO "' || r.rolname || '";' as seq_ddl
    FROM pg_catalog.pg_class AS t CROSS JOIN pg_catalog.pg_roles AS r CROSS JOIN (VALUES ('SELECT'), ('USAGE'), ('UPDATE')) AS p(perm)
    WHERE t.relnamespace::regnamespace::name = quote_ident(source_schema) AND t.relkind = 'S'  AND NOT r.rolsuper AND has_sequence_privilege(r.oid, t.oid, p.perm)
  LOOP

and tries to grant permissions on sequences which it thinks should exist.

If I comment out the snippet above I don’t get the error and new tenants are created successfully. I was able to add new records into the newly created tenant. Primary keys for the newly added records seem to be automatically updated.

What bothers me is that half of the queries select from information_schema and the other half from pg_catalog. Postgres documentation says

The information schema is defined in the SQL standard and can therefore be expected to be portable and remain stable — unlike the system catalogs, which are specific to PostgreSQL and are modeled after implementation concerns.

which IMO means we should try using only one of these sources of information, not both.

@tomturner, @SrNetoChan can you please comment a bit more on my findings? What do you make of it?

Note: I can easily disable the conflicting snippet of SQL code by introducing a condition around the number of sequences copied which should be an easy fix.