Bug: NOT NULL constraint failed: core_archiveresult.output when upgrading v0.4.24 archive to v0.6

I upgraded from v0.4.24 to v0.6.0 and ran archivebox init. After the list of migrations, it output:

[*] Checking links from indexes and archive folders (safe to Ctrl+C)...
    √ Added 1054 orphaned links from existing archive directories.
    ! Skipped adding 2236 invalid link data directories.
    (long list of archive dirs here)

    Hint: For more information about the link data directories that were skipped, run:
        archivebox status
        archivebox list --status=invalid

[*] [2021-04-13 05:09:37] Writing 1054 links to main index...
Traceback (most recent call last):
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/query.py", line 589, in update_or_create
    obj = self.select_for_update().get(**kwargs)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/query.py", line 429, in get
    raise self.model.DoesNotExist(
core.models.DoesNotExist: ArchiveResult matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
sqlite3.IntegrityError: NOT NULL constraint failed: core_archiveresult.output

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/pigmonkey/.local/bin//archivebox", line 8, in <module>
    sys.exit(main())
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/cli/__init__.py", line 140, in main
    run_subcommand(
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/cli/__init__.py", line 80, in run_subcommand
    module.main(args=subcommand_args, stdin=stdin, pwd=pwd)    # type: ignore
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/cli/archivebox_init.py", line 43, in main
    init(
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/main.py", line 433, in init
    write_main_index(list(pending_links.values()), out_dir=out_dir)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/index/__init__.py", line 232, in write_main_index
    write_sql_main_index(links, out_dir=out_dir)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/index/sql.py", line 88, in write_sql_main_index
    write_link_to_sql_index(link)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/util.py", line 114, in typechecked_function
    return func(*args, **kwargs)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/index/sql.py", line 66, in write_link_to_sql_index
    result, _ = ArchiveResult.objects.update_or_create(
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/query.py", line 594, in update_or_create
    obj, created = self._create_object_from_params(kwargs, params, lock=True)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/query.py", line 610, in _create_object_from_params
    obj = self.create(**params)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/query.py", line 447, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/base.py", line 753, in save
    self.save_base(using=using, force_insert=force_insert,
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/base.py", line 790, in save_base
    updated = self._save_table(
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/base.py", line 933, in _do_insert
    return manager._insert(
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: NOT NULL constraint failed: core_archiveresult.output
$  archivebox version
ArchiveBox v0.6.0
Cpython Linux Linux-5.11.11-hardened1-1-hardened-x86_64-with-glibc2.33 x86_64
IN_DOCKER=False DEBUG=False IS_TTY=True TZ=UTC SEARCH_BACKEND_ENGINE=ripgrep

[i] Dependency versions:
 √  ARCHIVEBOX_BINARY     v0.6.0          valid     /home/pigmonkey/.local/pipx/venvs/archivebox/bin/archivebox
 √  PYTHON_BINARY         v3.9.2          valid     /usr/bin/python3.9
 √  DJANGO_BINARY         v3.1.8          valid     /home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/django/bin/django-admin.py
 √  CURL_BINARY           v7.76.0         valid     /usr/bin/curl
 √  WGET_BINARY           v1.21.1         valid     /usr/bin/wget
 √  NODE_BINARY           v15.14.0        valid     /usr/bin/node
 √  SINGLEFILE_BINARY     v0.3.16         valid     ./node_modules/single-file/cli/single-file
 √  READABILITY_BINARY    v0.0.2          valid     ./node_modules/readability-extractor/readability-extractor
 -  MERCURY_BINARY        -               disabled  ./node_modules/@postlight/mercury-parser/cli.js
 -  GIT_BINARY            -               disabled  /usr/bin/git
 -  YOUTUBEDL_BINARY      -               disabled  /usr/bin/youtube-dl
 √  CHROME_BINARY         v89.0.4389.114  valid     /usr/bin/chromium
 √  RIPGREP_BINARY        v12.1.1         valid     /usr/bin/rg

[i] Source-code locations:
 √  PACKAGE_DIR           23 files        valid     /home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox
 √  TEMPLATES_DIR         3 files         valid     /home/pigmonkey/.local/pipx/venvs/archivebox/lib/python3.9/site-packages/archivebox/templates
 -  CUSTOM_TEMPLATES_DIR  -               disabled

[i] Secrets locations:
 -  CHROME_USER_DATA_DIR  -               disabled
 -  COOKIES_FILE          -               disabled

[i] Data locations:
 √  OUTPUT_DIR            9 files         valid     /home/pigmonkey/tmp/bookmarks
 √  SOURCES_DIR           0 files         valid     ./sources
 √  LOGS_DIR              0 files         valid     ./logs
 √  ARCHIVE_DIR           2236 files      valid     ./archive
 √  CONFIG_FILE           411.0 Bytes     valid     ./ArchiveBox.conf
 √  SQL_INDEX             920.0 KB        valid     ./index.sqlite3

[pirate]

New instructions here: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives

Also note I’ve added a new DB/filesystem troubleshooting area to the wiki that may help people arriving here from Google: https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives#database-troubleshooting

Contributions/suggestions welcome there.

[milosz]

I have ~20G archive backup from 2019 year. Thanks, I will try this intermediate step. I am thrilled that it is possible.