exportdata / loaddata cycle doesn't work, possibly due to abstract models?

So I did something that seems to be pretty simple to me, but maybe I’m just doing it wrong since its been a few versions since I worked with django, as what I think should be should be simple is failing in a way that suggests ether I have no idea what I am doing, or something is really wrong.

Problem

Once a manage.py cities_light has been done, I cant do a exportdata/loaddata cycle without the load cycle side failing to import anything due to a “Country matching query does not exist.” error.

Software Used

python 2.7 django 1.8 - yesterdays release of the new version django-cities-light - pulled this form pip yesterday

Repro steps

(1). Create new empty dkango project. (2). Add models to the first app of the project. Make sure to add Country, Region, and City models that literally only look like this for each of them:

from cities_light.abstract_models import AbstractCountry
from cities_light.receivers import connect_default_signals


class Country(AbstractCountry):
    pass
connect_default_signals(Country)

(3). Do the 2+ hour import of the city-light data. (4). Do an exportdata to export all the data in the db a city_light.json file:

python manage.py dumpdata --all > city_light.json

(5). Delete the sqlite db to trick django into thinking it needs to be created again.

 C:\Users\Duane\Dropbox\code\github\mouser>del db.sqlite3

(6). I then do a full migration of my app so that it re-initalizes a new empty database with the correct schema that I had before.

C:\Users\Duane\Dropbox\code\github\mouser>python manage.py migrate
C:\Python27\lib\site-packages\south\modelsinspector.py:20: RemovedInDjango19Warning: django.contrib.contenttypes.generic is deprecated and will be removed in Django 1.9. Its contents have been moved to the fields, forms, and admin submodules of django.contrib.contenttypes.
  from django.contrib.contenttypes import generic

Operations to perform:
  Synchronize unmigrated apps: staticfiles, messages
  Apply all migrations: sessions, admin, auth, launcher, contenttypes, cities_light
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
  Installing custom SQL...
Running migrations:
  Rendering model states... DONE
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying cities_light.0001_initial... OK
  Applying cities_light.0002_city... OK
  Applying cities_light.0003_auto_20141120_0342... OK
  Applying cities_light.0004_auto_20150403_2001... OK
  Applying launcher.0001_initial... OK
  Applying launcher.0002_auto_20150403_2001... OK
  Applying sessions.0001_initial... OK

(7). Now at this point the database should be EXACTLY the same schema as it was when I did the data export. Exactly,just with no data. Right?

(8). Now attempt to load the data from the saved city_light.json file into the database with the hope of never having to do another 2 hour long import of just city data again:

C:\Users\Duane\Dropbox\code\github\mouser>python manage.py loaddata -v 3 city_light.json
C:\Python27\lib\site-packages\south\modelsinspector.py:20: RemovedInDjango19Warning: django.contrib.contenttypes.generic is deprecated and will be removed in Django 1.9. Its contents have been moved to the fields, forms, and admin submodules of django.contrib.contenttypes.
  from django.contrib.contenttypes import generic

Loading 'city_light' fixtures...
Checking 'C:\Users\Duane\Dropbox\code\github\mouser\launcher\fixtures' for fixtures...
No fixture 'city_light' in 'C:\Users\Duane\Dropbox\code\github\mouser\launcher\fixtures'.
Checking 'C:\Users\Duane\Dropbox\code\github\mouser' for fixtures...
Installing json fixture 'city_light' from 'C:\Users\Duane\Dropbox\code\github\mouser'.
Traceback (most recent call last):
  File "manage.py", line 12, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 338, in execute_from_command_line
        utility.execute()
      File "C:\Python27\lib\site-packages\django\core\management\__init__.py", line 330, in execute
        self.fetch_command(subcommand).run_from_argv(self.argv)
      File "C:\Python27\lib\site-packages\django\core\management\base.py", line 390, in run_from_argv
        self.execute(*args, **cmd_options)
      File "C:\Python27\lib\site-packages\django\core\management\base.py", line 441, in execute
    output = self.handle(*args, **options)
      File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 60, in handle
        self.loaddata(fixture_labels)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 90, in loaddata
    self.load_label(fixture_label)
  File "C:\Python27\lib\site-packages\django\core\management\commands\loaddata.py", line 147, in load_label
    obj.save(using=self.using)
  File "C:\Python27\lib\site-packages\django\core\serializers\base.py", line 173, in save
    models.Model.save_base(self.object, using=using, raw=True)
  File "C:\Python27\lib\site-packages\django\db\models\base.py", line 734, in save_base
    update_fields=update_fields)
  File "C:\Python27\lib\site-packages\django\dispatch\dispatcher.py", line 201, in send
    response = receiver(signal=self, sender=sender, **named)
  File "C:\Python27\lib\site-packages\cities_light\receivers.py", line 28, in set_display_name
    instance.display_name = instance.get_display_name()
  File "C:\Python27\lib\site-packages\cities_light\abstract_models.py", line 114, in get_display_name    
    return '%s, %s' % (self.name, self.country.name)
  File "C:\Python27\lib\site-packages\django\db\models\fields\related.py", line 602, in __get__
    rel_obj = qs.get()
  File "C:\Python27\lib\site-packages\django\db\models\query.py", line 334, in get
    self.model._meta.object_name
launcher.models.country.DoesNotExist: Problem installing fixture     'C:\Users\Duane\Dropbox\code\github\mouser\city_light.json': Country matching query does not exist.

As you can see it seems to be failing to detect the models after loading the abstract ones.

I’m totally blocked, and I have no idea what to do. I have done everything according to the docs that I am aware of, and I expect to be able to export/import data at will, that does not work in my experience so far.

What I Am trying to do

I want my app to be initialized from migrations, as that is the new django way and I want to embrace the future. I don’t want to have to download a large file form the internet and suffer though a huge deployment every time I deploy; I want to do it at most once, then have all the data in a django migration i just apply and am done with, that I can also check into our source control (git, in this case). That is what I am trying to do

[st4lk]

Ok, i have reproduced the problem. It is linked to usage of abstract models, but looks like it is not a problem of abstract models itself. It is caused by the order of custom models. So, here what i tried. My custom main/models.py:

from cities_light.abstract_models import AbstractCountry, AbstractRegion, AbstractCity
from cities_light.receivers import connect_default_signals

class City(AbstractCity):
    pass
connect_default_signals(City)


class Country(AbstractCountry):
    pass
connect_default_signals(Country)


class Region(AbstractRegion):
    pass
connect_default_signals(Region)

Pay attention, models are declared in this order: City, Country, Region. Now, fill database:

python manage.py cities_light --force-import-all

Create fixture (--indent 4 just for readability):

python manage.py dumpdata --all --indent 4 > city_light.json

Everything is ok for now. Lets look at the beginning of city_light.json. It will contain something like this:

[
// some contenttypes
{
    "fields": {
        "model": "city",
        "app_label": "main"
    },
    "model": "contenttypes.contenttype",
    "pk": 7
},
{
    "fields": {
        "model": "country",
        "app_label": "main"
    },
    "model": "contenttypes.contenttype",
    "pk": 8
},
{
    "fields": {
        "model": "region",
        "app_label": "main"
    },
    "model": "contenttypes.contenttype",
    "pk": 9
},
{
    "fields": {
        "display_name": "Yerres, \u00cele-de-France, France",
        "name": "Yerres",
        "country": 1,
        "region": 11,
        "alternate_names": "",
        "search_names": "yerresfrance yerresiledefrancefrance",
        "longitude": "2.49338",
        "geoname_id": 2967245,
        "feature_code": "PPL",
        "name_ascii": "Yerres",
        "latitude": "48.71785",
        "slug": "yerres",
        "population": 28897
    },
    "model": "main.city",
    "pk": 1
},
// other cities ...
{
    "fields": {
        "code2": "FR",
        "code3": "FRA",
        "name": "France",
        "tld": "fr",
        "alternate_names": "",
        "continent": "EU",
        "geoname_id": 3017382,
        "phone": "33",
        "name_ascii": "France",
        "slug": "france"
    },
    "model": "main.country",
    "pk": 1
},
// other countries ...
{
    "fields": {
        "display_name": "Rh\u00f4ne-Alpes, France",
        "name": "Rh\u00f4ne-Alpes",
        "country": 1,
        "alternate_names": "Rh\u00f4ne-Alpes Region",
        "geoname_id": 2983751,
        "geoname_code": "B9",
        "name_ascii": "Rhone-Alpes",
        "slug": "rhone-alpes"
    },
    "model": "main.region",
    "pk": 1
},
// other regions ...
// and some other django models, not interesting here ...
]

As we can see, fixture reflect the order of our declared models: first goes City then Country and finally Region.

Now, delete db.sqlite3, create new by applying migrations, and try to loaddata. We’ll get the same error, as in description of this issue.

Why?

Because some pre_save signals are connected to your models. And one of them, (exactly set_display_name) will cause access to model by ForeignKey. Before saving the city models this signal receiver will try to get city.country by foreignkey. But, as we can see for fixture, City will be loaded first, whereas Country is not in the database yet. And we get DoesNotExist error.

How to fix?

1. Simple but not beautiful solution. Just comment the signals connection to your models:

   # connect_default_signals(Country)
   # connect_default_signals(City)
   # connect_default_signals(Region)
   

   when you do loaddata and uncomment after (all fields already was populated and saved in fixtures).

2. Good, but complicated solution. Declare you custom cities_light models in the same order, as cities_light package does:

   class Country(AbstractCountry): pass
   class Region(AbstractRegion): pass
   class City(AbstractCity): pass
   

   Now, it is needed to create fixture again (so models will be saved in correct order). Keep in mind, that your database is not changed, you just change the order of models declaration and create new fixture. In that case, when you’ll delete the database and create new one (empty), database will be filled with contenttypes objects automatically (matters, because flag --all was used). But, as order of our models is changed, those contenttypes objects will also have different order and consequently other ids (different from our fixture, that was created from another database).

   If we will try to loaddata now, following error probably gonna to happen:

   django.db.utils.IntegrityError:
       Problem installing fixture '.../city_light.json':
       Could not load contenttypes.ContentType(pk=7):
       UNIQUE constraint failed: django_content_type.app_label, django_content_type.model
   

   To avoid it, we must manually set correct ids in our city_light.json for contenttypes. Or, after changing order of custom cities_light models, delete database, do python manage.py cities_light --force-import-all again and make fixture from this database. In that case no manual work with contenttypes is needed.

So, as i see, nothing to do with cities_light package according to this issue. Maybe we just need to mention in the docs, that order of custom models matters in case of fixtures and signals.

[rubenanapu]

Today (24th, April, 2020) I had a similar problem with Django 2.2 (nothing related to django-cities-light)

My fixtures file was as simple as this:

[
{
    "model": "contenttypes.contenttype",
    "pk": 1,
    "fields": {
        "app_label": "admin",
        "model": "logentry"
    }
},
{
    "model": "contenttypes.contenttype",
    "pk": 2,
    "fields": {
        "app_label": "auth",
        "model": "permission"
    }
}]

When I ran ./manage.py loaddata initial_data.json, I got:

django.db.utils.IntegrityError: Problem installing fixture '/home/user/reponame/projectname/initial_data.json': Could not load contenttypes.ContentType(pk=2): UNIQUE constraint failed: django_content_type.app_label, django_content_type.model

What I did to make it work was just rename the pk to id for the contenttypes.contenttype model. After that, the migration worked as expected.

./manage.py loaddata initial_data.json 
Installed 2 object(s) from 1 fixture(s)

After the change, my initial_data.json file was:

[
{
    "model": "contenttypes.contenttype",
    "id": 1,
    "fields": {
        "app_label": "admin",
        "model": "logentry"
    }
},
{
    "model": "contenttypes.contenttype",
    "id": 2,
    "fields": {
        "app_label": "auth",
        "model": "permission"
    }
}]

It is worth mentioning that my original initial_dataj.json has many other models, but renaming pk to id only for the contenttypes.contenttype solved my problem.