Bug in request generated

Hi, I have a bug with a request generated when trying to fetch data from a many 2 many relation:

• the request generated reference a user.id field as a primary key, but the primary key in the model is user.registrationnumber
• I also had some errors when using field name containing capitalized letter ou containing ‘_’ You will find all the details in the sample below.

OS: Centos 7.9 (docker) python version: 3.8.3 ormar version: 0.7.3 database backend: postgresql

To reproduce the error i made the script below:

import asyncio
import uuid
from datetime import date, datetime
from os import major
from typing import List, Optional, Union

import databases
import ormar
import sqlalchemy
from fastapi.encoders import jsonable_encoder
from sqlalchemy import func, text
import jsonpickle


DATABASE_URL="postgresql://postgres:postgres@db:5432/test"
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()


class MainMeta(ormar.ModelMeta):
    metadata = metadata
    database = database


class Role(ormar.Model):
    class Meta(MainMeta):
        pass
    name                : str = ormar.Text(primary_key=True)
    order               : int = ormar.Integer(default=0)
    description         : str = ormar.Text()


class Company(ormar.Model):
    class Meta(MainMeta):
        pass
    name                : str = ormar.Text(primary_key=True)


class UserRoleCompany(ormar.Model):
    class Meta(MainMeta):
        pass


class User(ormar.Model):
    class Meta(MainMeta):
        pass
    registrationnumber  : str = ormar.Text(primary_key=True)
    company             : Company = ormar.ForeignKey(Company)
    name                : str = ormar.Text()
    role                : Optional[Role] = ormar.ForeignKey(Role)
    roleforcompanies    : Optional[Union[Company, List[Company]]] = ormar.ManyToMany(Company, through=UserRoleCompany)
    lastupdate          : date = ormar.DateTime(server_default=sqlalchemy.func.now())


async def main():
    if not database.is_connected:
        print("connection to db {}.".format(DATABASE_URL))
        await database.connect()
    ##########################################################################################
    try:
        print("adding role")
        role_0 = await Role.objects.create(name="user", order=0, description = "no administration right")
        role_1 = await Role.objects.create(name="admin", order=1, description = "standard administration right")
        role_2 = await Role.objects.create(name="super_admin", order=2, description = "super administration right")
        assert await Role.objects.count() == 3

        print("adding company")
        company_0 = await Company.objects.create(name="Company")
        company_1 = await Company.objects.create(name="Subsidiary Company 1")
        company_2 = await Company.objects.create(name="Subsidiary Company 2")
        company_3 = await Company.objects.create(name="Subsidiary Company 3")
        assert await Company.objects.count() == 4

        print("adding user")
        user = await User.objects.create(registrationnumber="00-00000", company=company_0, name="admin", role=role_1)
        assert await User.objects.count() == 1

        print("removing user")
        await user.delete()
        assert await User.objects.count() == 0

        print("adding user with company-role")
        companies: List[Company] = [company_1, company_2]
        # user = await User.objects.create(registrationnumber="00-00000", company=company_0, name="admin", role=role_1, roleforcompanies=companies)
        user = await User.objects.create(registrationnumber="00-00000", company=company_0, name="admin", role=role_1)
        # print(User.__fields__)
        await user.roleforcompanies.add(company_1)
        await user.roleforcompanies.add(company_2)

        users = await User.objects.select_related("roleforcompanies").all()
        print(jsonpickle.encode(jsonable_encoder(users), unpicklable=False, keys=True ))

    except Exception as error:
        print(error)


    """

    This is the request generated:
    'SELECT
    users.registrationnumber as registrationnumber,
    users.company as company,
    users.name as name, users.role as role,
    users.lastupdate as lastupdate,
    cy24b4_userrolecompanys.id as cy24b4_id,
    cy24b4_userrolecompanys.company as cy24b4_company,
    cy24b4_userrolecompanys.user as cy24b4_user,
    jn50a4_companys.name as jn50a4_name \n
    FROM users
    LEFT OUTER JOIN userrolecompanys cy24b4_userrolecompanys ON cy24b4_userrolecompanys.user=users.id
    LEFT OUTER JOIN companys jn50a4_companys ON jn50a4_companys.name=cy24b4_userrolecompanys.company
    ORDER BY users.registrationnumber, jn50a4_companys.name'

    There is an error in the First LEFT OUTER JOIN generated:
    ... companys.user=users.id
    should be:
   ... companys.user=users.registrationnumber

    There is also a \n in the midle of the string...

    The execution produce the error: column users.id does not exist
    """

    ##########################################################################################
    if database.is_connected:
        await database.disconnect()
        print("db closed.")




if __name__ == '__main__':
    asyncio.run(main())

I’m new to python, sqlalchemy, fastapi anr ormar… maybe i made some mistakes… Thanks for this great project.

[collerek]

@soderluk Yes there was a bug, added your test and it’s passing.

But note that many_to_many relations are always added as weakrefs.proxy so in your scenario your test might pass or fail depending on when the garbage collector will run and reference will become invalid (it’s cleaned in ormar when becomes invalid).

If you want to use the parent model in same query and access freshly added related model you should keep the other reference for the child model. i.e.:

users = {}
        for i in range(1, 6):
            user = await User.objects.get(pk=i)
            users[f"user_{i}"] = user # add a reference to object in example in dict so it's not instantly gc and replaced by new one
            if i % 2 == 0:
                await s1.students.add(user)
            else:
                await s2.students.add(user)

        assert len(s1.students) == 2 # that way you are sure the students are here and are valid
        assert len(s2.students) == 3 # otherwise you need to refresh again from db

Note: I am thinking about changing many2many to concrete references, but that will create circular references. Python CAN handle this scenario and collect circles when they have no outside reference but it can affect performance, so I need to check it before change like this.

The bug with wrong table prefixes should be fixed in 0.7.4 - please update and try.

[soderluk]

@collerek Confirming that now the relations works again 😃

Thank you again for your quick and prompt responses and fixes!