Lightrun
question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

How to dumpdata + loaddata

See original GitHub issue

Going a bit mad trying to work out the best incantation to get dumpdata/loaddata to work in harmony -

staging computer 1 with pages on a real db

$ python manage.py dumpdata --natural-foreign --indent=4 -e contenttypes -e auth.Permission -e sessions -e wagtailcore.pagerevision > data.json

dev computer 2 with sqlite db deleted

$ python manage.py migrate --run-syncdb --no-input && python manage.py loaddata data.json 
Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
Operations to perform:
  Synchronize unmigrated apps: captcha, clear_cache, crispy_forms, modelcluster, runserver_nostatic, staticfiles, storages, wagtail_modeltranslation, wagtail_modeltranslation_makemigrations, wagtail_modeltranslation_migrate, wagtailfrontendcache, wagtailmarkdown, wagtailsettings, wagtailsites, wagtailsnippets
  Apply all migrations: admin, auth, contact, contenttypes, home, info, invest, sector, sessions, setup_guide, taggit, wagtailadmin, wagtailcore, wagtaildocs, wagtailembeds, wagtailforms, wagtailimages, wagtailredirects, wagtailsearch, wagtailusers
Synchronizing apps without migrations:
  Creating tables...
    Running deferred SQL...
Running migrations:
  Applying contenttypes.0001_initial... OK
  Applying auth.0001_initial... OK
  Applying admin.0001_initial... OK
  Applying admin.0002_logentry_remove_auto_add... OK
  Applying contenttypes.0002_remove_content_type_name... OK
  Applying auth.0002_alter_permission_name_max_length... OK
  Applying auth.0003_alter_user_email_max_length... OK
  Applying auth.0004_alter_user_username_opts... OK
  Applying auth.0005_alter_user_last_login_null... OK
  Applying auth.0006_require_contenttypes_0002... OK
  Applying auth.0007_alter_validators_add_error_messages... OK
  Applying auth.0008_alter_user_username_max_length... OK
  Applying wagtailcore.0001_squashed_0016_change_page_url_path_to_text_field... OK
  Applying wagtailcore.0017_change_edit_page_permission_description... OK
  Applying wagtailcore.0018_pagerevision_submitted_for_moderation_index... OK
  Applying wagtailcore.0019_verbose_names_cleanup... OK
  Applying wagtailcore.0020_add_index_on_page_first_published_at... OK
  Applying wagtailcore.0021_capitalizeverbose... OK
  Applying wagtailcore.0022_add_site_name... OK
  Applying wagtailcore.0023_alter_page_revision_on_delete_behaviour... OK
  Applying wagtailcore.0024_collection... OK
  Applying wagtailcore.0025_collection_initial_data... OK
  Applying wagtailcore.0026_group_collection_permission... OK
  Applying wagtailcore.0027_fix_collection_path_collation... OK
  Applying wagtailcore.0024_alter_page_content_type_on_delete_behaviour... OK
  Applying wagtailcore.0028_merge... OK
  Applying wagtailcore.0029_unicode_slugfield_dj19... OK
  Applying wagtailcore.0030_index_on_pagerevision_created_at... OK
  Applying wagtailcore.0031_add_page_view_restriction_types... OK
  Applying wagtailcore.0032_add_bulk_delete_page_permission... OK
  Applying wagtailcore.0033_remove_golive_expiry_help_text... OK
  Applying wagtailcore.0034_page_live_revision... OK
  Applying wagtailcore.0035_page_last_published_at... OK
  Applying wagtailcore.0036_populate_page_last_published_at... OK
  Applying wagtailcore.0037_set_page_owner_editable... OK
  Applying wagtailcore.0038_make_first_published_at_editable... OK
  Applying wagtailcore.0039_collectionviewrestriction... OK
  Applying wagtailcore.0040_page_draft_title... OK
  Applying contact.0001_initial... OK
  Applying contact.0002_auto_20180322_1443... OK
  Applying contact.0003_auto_20180327_1341... OK
  Applying taggit.0001_initial... OK
  Applying wagtailimages.0001_initial... OK
  Applying wagtailimages.0002_initial_data... OK
  Applying wagtailimages.0003_fix_focal_point_fields... OK
  Applying wagtailimages.0004_make_focal_point_key_not_nullable... OK
  Applying wagtailimages.0005_make_filter_spec_unique... OK
  Applying wagtailimages.0006_add_verbose_names... OK
  Applying wagtailimages.0007_image_file_size... OK
  Applying wagtailimages.0008_image_created_at_index... OK
  Applying wagtailimages.0009_capitalizeverbose... OK
  Applying wagtailimages.0010_change_on_delete_behaviour... OK
  Applying wagtailimages.0011_image_collection... OK
  Applying wagtailimages.0012_copy_image_permissions_to_collections... OK
  Applying wagtailimages.0013_make_rendition_upload_callable... OK
  Applying wagtailimages.0014_add_filter_spec_field... OK
  Applying wagtailimages.0015_fill_filter_spec_field... OK
  Applying wagtailimages.0016_deprecate_rendition_filter_relation... OK
  Applying wagtailimages.0017_reduce_focal_point_key_max_length... OK
  Applying wagtailimages.0018_remove_rendition_filter... OK
  Applying wagtailimages.0019_delete_filter... OK
  Applying home.0001_initial... OK
  Applying home.0002_create_homepage... OK
  Applying home.0003_auto_20180129_1650... OK
  Applying home.0004_homepage_how_we_help... OK
  Applying home.0005_homepage_subsections... OK
  Applying home.0006_auto_20180309_1729... OK
  Applying home.0007_auto_20180309_1742... OK
  Applying home.0008_remove_homepage_sector_lead_in... OK
  Applying home.0009_auto_20180309_2042... OK
  Applying home.0010_auto_20180322_1443... OK
  Applying home.0011_auto_20180327_1341... OK
  Applying info.0001_initial... OK
  Applying info.0002_auto_20180322_1443... OK
  Applying info.0003_auto_20180327_1341... OK
  Applying invest.0001_initial... OK
  Applying invest.0002_branding_footer_logo... OK
  Applying invest.0003_auto_20180308_0145... OK
  Applying invest.0004_auto_20180308_1104... OK
  Applying invest.0005_auto_20180308_1137... OK
  Applying invest.0006_auto_20180309_1448... OK
  Applying invest.0007_auto_20180309_1543... OK
  Applying invest.0008_branding_language_choice_icon... OK
  Applying sector.0001_initial... OK
  Applying sector.0002_industrieslandingpage_hero_image... OK
  Applying sector.0003_sectorpage_show_on_frontpage... OK
  Applying sector.0004_auto_20180305_1429... OK
  Applying sector.0005_auto_20180305_1554... OK
  Applying sector.0006_auto_20180322_1443... OK
  Applying sector.0007_auto_20180327_1341... OK
  Applying sessions.0001_initial... OK
  Applying setup_guide.0001_initial... OK
  Applying setup_guide.0002_auto_20180207_1200... OK
  Applying setup_guide.0003_setupguidelandingpage_lead_in... OK
  Applying setup_guide.0004_auto_20180322_1443... OK
  Applying setup_guide.0005_auto_20180327_1341... OK
  Applying taggit.0002_auto_20150616_2121... OK
  Applying wagtailadmin.0001_create_admin_access_permissions... OK
  Applying wagtaildocs.0001_initial... OK
  Applying wagtaildocs.0002_initial_data... OK
  Applying wagtaildocs.0003_add_verbose_names... OK
  Applying wagtaildocs.0004_capitalizeverbose... OK
  Applying wagtaildocs.0005_document_collection... OK
  Applying wagtaildocs.0006_copy_document_permissions_to_collections... OK
  Applying wagtaildocs.0005_alter_uploaded_by_user_on_delete_action... OK
  Applying wagtaildocs.0007_merge... OK
  Applying wagtailembeds.0001_initial... OK
  Applying wagtailembeds.0002_add_verbose_names... OK
  Applying wagtailembeds.0003_capitalizeverbose... OK
  Applying wagtailforms.0001_initial... OK
  Applying wagtailforms.0002_add_verbose_names... OK
  Applying wagtailforms.0003_capitalizeverbose... OK
  Applying wagtailredirects.0001_initial... OK
  Applying wagtailredirects.0002_add_verbose_names... OK
  Applying wagtailredirects.0003_make_site_field_editable... OK
  Applying wagtailredirects.0004_set_unique_on_path_and_site... OK
  Applying wagtailredirects.0005_capitalizeverbose... OK
  Applying wagtailsearch.0001_initial... OK
  Applying wagtailsearch.0002_add_verbose_names... OK
  Applying wagtailsearch.0003_remove_editors_pick... OK
  Applying wagtailusers.0001_initial... OK
  Applying wagtailusers.0002_add_verbose_name_on_userprofile... OK
  Applying wagtailusers.0003_add_verbose_names... OK
  Applying wagtailusers.0004_capitalizeverbose... OK
  Applying wagtailusers.0005_make_related_name_wagtail_specific... OK
  Applying wagtailusers.0006_userprofile_prefered_language... OK
Missing languages in "search_description" field from "wagtailcore.page" model: en, de, es, fr, pt, ar, ja, zh-cn

SQL to synchronize "wagtailcore.page" schema:
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_en" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_de" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_es" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_fr" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_pt" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_ar" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_ja" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "search_description_zh_cn" text;
Executing SQL...
Done
Missing languages in "url_path" field from "wagtailcore.page" model: en, de, es, fr, pt, ar, ja, zh-cn

SQL to synchronize "wagtailcore.page" schema:
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_en" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_de" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_es" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_fr" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_pt" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_ar" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_ja" text;
   ALTER TABLE "wagtailcore_page" ADD COLUMN "url_path_zh_cn" text;
Executing SQL...
Done
Missing languages in "title" field from "wagtailcore.page" model: en, de, es, fr, pt, ar, ja, zh-cn

SQL to synchronize "wagtailcore.page" schema:
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_en" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_de" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_es" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_fr" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_pt" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_ar" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_ja" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "title_zh_cn" varchar(255);
Executing SQL...
Done
Missing languages in "seo_title" field from "wagtailcore.page" model: en, de, es, fr, pt, ar, ja, zh-cn

SQL to synchronize "wagtailcore.page" schema:
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_en" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_de" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_es" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_fr" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_pt" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_ar" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_ja" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "seo_title_zh_cn" varchar(255);
Executing SQL...
Done
Missing languages in "slug" field from "wagtailcore.page" model: en, de, es, fr, pt, ar, ja, zh-cn

SQL to synchronize "wagtailcore.page" schema:
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_en" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_de" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_es" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_fr" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_pt" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_ar" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_ja" varchar(255);
   ALTER TABLE "wagtailcore_page" ADD COLUMN "slug_zh_cn" varchar(255);
Executing SQL...
Done
Converted retries value: 3 -> Retry(total=3, connect=None, read=None, redirect=None, status=None)
Traceback (most recent call last):
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: UNIQUE constraint failed: wagtailcore_groupcollectionpermission.group_id, wagtailcore_groupcollectionpermission.collection_id, wagtailcore_groupcollectionpermission.permission_id

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 364, in execute_from_command_line
    utility.execute()
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/management/__init__.py", line 356, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/management/base.py", line 283, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/management/base.py", line 330, in execute
    output = self.handle(*args, **options)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 69, in handle
    self.loaddata(fixture_labels)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 109, in loaddata
    self.load_label(fixture_label)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/management/commands/loaddata.py", line 175, in load_label
    obj.save(using=self.using)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/core/serializers/base.py", line 205, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/models/base.py", line 838, in save_base
    updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/models/base.py", line 905, in _save_table
    forced_update)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/models/base.py", line 955, in _do_update
    return filtered._update(values) > 0
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/models/query.py", line 664, in _update
    return query.get_compiler(self.db).execute_sql(CURSOR)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 1204, in execute_sql
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 899, in execute_sql
    raise original_exception
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/models/sql/compiler.py", line 889, in execute_sql
    cursor.execute(sql, params)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/utils.py", line 94, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/utils/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/mnt/data/home/stu/.virtualenvs/dit-invest/local/lib/python3.6/site-packages/django/db/backends/sqlite3/base.py", line 328, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: Problem installing fixture '/mnt/data/home/stu/projects/work/dit/invest/data.json': Could not load wagtailcore.GroupCollectionPermission(pk=1): UNIQUE constraint failed: wagtailcore_groupcollectionpermission.group_id, wagtailcore_groupcollectionpermission.collection_id, wagtailcore_groupcollectionpermission.permission_id

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:5

github_iconTop GitHub Comments

3reactions
AdrienLemairecommented, Apr 19, 2019

@stuaxo @gasman I cannot find the related SO thread, can you share it here?

https://stackoverflow.com/search?q=[wagtail]+dumpdata

When I tried to dumpdata my local sqlite db to share it with other devs, they encountered many issues, and I ended up with

$ ./manage.py dumpdata  --format yaml --natural-primary --exclude auth.permission --exclude contenttypes --exclude wagtailcore.GroupCollectionPermission > fixtures/db.yaml

But, although they can now loaddata that file, their local runserver project is completely broken.

EDIT: I through the problem was natural-primary vs natural-foreign, but the loaddata wagtail error persists

$ ./manage.py dumpdata  --format yaml --natural-foreign --exclude auth.permission --exclude contenttypes  > fixture.yaml
$ # in coworker setup
$ rm db.sqlite3
$ ./manage.py migrate && ./manage.py loaddata fixture.yaml
Could not load wagtailcore.GroupCollectionPermission(pk=1): UNIQUE constraint failed: wagtailcore_groupcollectionpermission.group_id, wagtailcore_groupcollectionpermission.collection_id, wagtailcore_groupcollectionpermission.permission_id
1reaction
perepicornellcommented, Aug 10, 2022

In case is helpful for anyone stumbling upon this page, this helped me figure it out: https://www.accordbox.com/blog/how-export-restore-wagtail-site/

I could generate fixtures that then worked fine when using loaddata with this command:

python manage.py dumpdata --natural-foreign --indent 2 \
    -e contenttypes -e auth.permission -e postgres_search.indexentry \
    -e wagtailcore.groupcollectionpermission \
    -e wagtailcore.grouppagepermission -e wagtailimages.rendition \
    -e wagtailcore.pagerevision \
    -e sessions > data.json

In my case I created a management command for it which is basically:

    def handle(self, *args, **options):
        parameters = (
            "--natural-foreign",
            "--indent=2",
            "-e=contenttypes",
            "-e=auth.permission",
            "-e=wagtailcore.groupcollectionpermission",
            "-e=wagtailcore.grouppagepermission",
            "-e=wagtailimages.rendition",
            "-e=wagtailcore.pagerevision",
            "-e=sessions",
            "-o=apps/base/fixtures/full_site.json",
        )
        call_command("dumpdata", *parameters, verbosity=2)

        self.stdout.write(
            "Fixtures updated."
        )

I just implemented it, so maybe all these excludes are resulting in missing some important information and I’m not yet aware of, beware of that if you intend to use it.

The official documentation is this: https://docs.wagtail.org/en/v3.0.1/advanced_topics/testing.html#using-dumpdata Which was insufficient for me to know how to do it, perhaps someone could add a more specific example to it 😃

Read more comments on GitHub >

github_iconTop Results From Across the Web

Django dumpdata and loaddata (Example) - Coderwall
dumpdata command. It is a django management command, which can be use to backup(export) you model instances or whole database ...
Read more >
How to Dump Your Django Database and Load It into a New ...
The manage.py dumpdata command dumps the data from your database either in the command line or in a file that you specify. You...
Read more >
Django dump data for a single model? - Stack Overflow
As of version 1.1 and greater, the Django dumpdata management command allows you to dump data from individual tables:
Read more >
django-admin and manage.py
The output of dumpdata can be used as input for loaddata . Note that dumpdata uses the default manager on the model for...
Read more >
Import and Export Data in Django | Individual Software Process
json , and import the data into the database. See the django-admin docs for more dumpdata and loaddata options. Creating Initial Data for...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found
Previous page

New StreamField & InlinePanel UIs

Next page

RichTextField not visible when used as a field on custom Image model