Unable to stand up nautobot in a HA mode (one write instance & one read-only instance)

Environment

• Nautobot version (Docker tag too if applicable): 1.3.6
• Python version: 3.10
• Database platform, version: postgres
• Middleware(s):

Steps to Reproduce

1. Standup nautobot using docker-compose but change the compose file so the database port is exposed
2. Connect to the postgres database and create a read-only user and set transactions to read-only

CREATE USER nautobot_read WITH PASSWORD 'readonly';
GRANT CONNECT ON DATABASE nautobot TO nautobot_read;
GRANT USAGE ON SCHEMA public TO nautobot_read;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO nautobot_read;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO nautobot_read;
ALTER USER nautobot_read set default_transaction_read_only = on;

1. Copy the docker-compose directory to another folder and change the compose file to use a different port for exposing the web interface and remove the db container and volume
2. Change the local.env to update the DB settings so they use a DB host of the local IP on the machine and the database port that was exposed in step 1 and the DB account/password for the read user setup in Step 2.

Expected Behavior

I’m able to spin up a write instance and a separate read-only instance.

The idea being that nautobot can be hosted in two different sites, one site with write the other for read, the redis cluster is synced cross-site and the database has read-only replicas in the other site.

Observed Behavior

The read-only instance complains it doesn’t have the permissions to perform INSERT/UPDATE statements in different tables.

nauto_read-celery_beat-1      | Traceback (most recent call last):
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
nauto_read-celery_beat-1      |     return self.cursor.execute(sql, params)
nauto_read-celery_beat-1      | psycopg2.errors.ReadOnlySqlTransaction: cannot execute UPDATE in a read-only transaction
nauto_read-celery_beat-1      | 
nauto_read-celery_beat-1      | 
nauto_read-celery_beat-1      | The above exception was the direct cause of the following exception:
nauto_read-celery_beat-1      | 
nauto_read-celery_beat-1      | Traceback (most recent call last):
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 320, in update_from_dict
nauto_read-celery_beat-1      |     entry = self.Entry.from_entry(name,
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 180, in from_entry
nauto_read-celery_beat-1      |     name=name, defaults=cls._unpack_fields(**entry),
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 187, in _unpack_fields
nauto_read-celery_beat-1      |     model_schedule, model_field = cls.to_model_schedule(schedule)
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django_celery_beat/schedulers.py", line 172, in to_model_schedule
nauto_read-celery_beat-1      |     model_schedule.save()
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save
nauto_read-celery_beat-1      |     self.save_base(using=using, force_insert=force_insert,
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
nauto_read-celery_beat-1      |     updated = self._save_table(
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 858, in _save_table
nauto_read-celery_beat-1      |     updated = self._do_update(base_qs, using, pk_val, values, update_fields,
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 912, in _do_update
nauto_read-celery_beat-1      |     return filtered._update(values) > 0
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 802, in _update
nauto_read-celery_beat-1      |     return query.get_compiler(self.db).execute_sql(CURSOR)
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
nauto_read-celery_beat-1      |     cursor = super().execute_sql(result_type)
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
nauto_read-celery_beat-1      |     cursor.execute(sql, params)
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/cacheops/transaction.py", line 97, in execute
nauto_read-celery_beat-1      |     result = self._no_monkey.execute(self, sql, params)
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
nauto_read-celery_beat-1      |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
nauto_read-celery_beat-1      |     return executor(sql, params, many, context)
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
nauto_read-celery_beat-1      |     with self.db.wrap_database_errors:
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
nauto_read-celery_beat-1      |     raise dj_exc_value.with_traceback(traceback) from exc_value
nauto_read-celery_beat-1      |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
nauto_read-celery_beat-1      |     return self.cursor.execute(sql, params)
nauto_read-celery_beat-1      | django.db.utils.InternalError: cannot execute UPDATE in a read-only transaction
nauto_read-celery_beat-1      | 

nauto_read-nautobot-1         | Traceback (most recent call last):
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 581, in get_or_create
nauto_read-nautobot-1         |     return self.get(**kwargs), False
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/cacheops/query.py", line 351, in get
nauto_read-nautobot-1         |     return qs._no_monkey.get(qs, *args, **kwargs)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 435, in get
nauto_read-nautobot-1         |     raise self.model.DoesNotExist(
nauto_read-nautobot-1         | django.contrib.contenttypes.models.ContentType.DoesNotExist: ContentType matching query does not exist.
nauto_read-nautobot-1         | 
nauto_read-nautobot-1         | During handling of the above exception, another exception occurred:
nauto_read-nautobot-1         | 
nauto_read-nautobot-1         | Traceback (most recent call last):
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
nauto_read-nautobot-1         |     return self.cursor.execute(sql, params)
nauto_read-nautobot-1         | psycopg2.errors.ReadOnlySqlTransaction: cannot execute INSERT in a read-only transaction
nauto_read-nautobot-1         | 
nauto_read-nautobot-1         | 
nauto_read-nautobot-1         | The above exception was the direct cause of the following exception:
nauto_read-nautobot-1         | 
nauto_read-nautobot-1         | Traceback (most recent call last):
nauto_read-nautobot-1         |   File "/usr/local/bin/nautobot-server", line 8, in <module>
nauto_read-nautobot-1         |     sys.exit(main())
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/nautobot/core/cli.py", line 54, in main
nauto_read-nautobot-1         |     run_app(
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/nautobot/core/runner/runner.py", line 266, in run_app
nauto_read-nautobot-1         |     management.execute_from_command_line([runner_name, command] + command_args)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
nauto_read-nautobot-1         |     utility.execute()
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 413, in execute
nauto_read-nautobot-1         |     self.fetch_command(subcommand).run_from_argv(self.argv)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 354, in run_from_argv
nauto_read-nautobot-1         |     self.execute(*args, **cmd_options)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
nauto_read-nautobot-1         |     output = self.handle(*args, **options)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/nautobot/core/management/commands/post_upgrade.py", line 78, in handle
nauto_read-nautobot-1         |     call_command(
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/__init__.py", line 181, in call_command
nauto_read-nautobot-1         |     return command.execute(*args, **defaults)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 398, in execute
nauto_read-nautobot-1         |     output = self.handle(*args, **options)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/base.py", line 89, in wrapped
nauto_read-nautobot-1         |     res = handle_func(*args, **kwargs)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/commands/migrate.py", line 268, in handle
nauto_read-nautobot-1         |     emit_post_migrate_signal(
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/core/management/sql.py", line 42, in emit_post_migrate_signal
nauto_read-nautobot-1         |     models.signals.post_migrate.send(
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 180, in send
nauto_read-nautobot-1         |     return [
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/dispatch/dispatcher.py", line 181, in <listcomp>
nauto_read-nautobot-1         |     (receiver, receiver(signal=self, sender=sender, **named))
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/constance/apps.py", line 28, in create_perm
nauto_read-nautobot-1         |     content_type, created = ContentType.objects.using(using).get_or_create(
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 588, in get_or_create
nauto_read-nautobot-1         |     return self.create(**params), True
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 453, in create
nauto_read-nautobot-1         |     obj.save(force_insert=True, using=self.db)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 739, in save
nauto_read-nautobot-1         |     self.save_base(using=using, force_insert=force_insert,
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 776, in save_base
nauto_read-nautobot-1         |     updated = self._save_table(
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 881, in _save_table
nauto_read-nautobot-1         |     results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/base.py", line 919, in _do_insert
nauto_read-nautobot-1         |     return manager._insert(
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/manager.py", line 85, in manager_method
nauto_read-nautobot-1         |     return getattr(self.get_queryset(), name)(*args, **kwargs)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/query.py", line 1270, in _insert
nauto_read-nautobot-1         |     return query.get_compiler(using=using).execute_sql(returning_fields)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
nauto_read-nautobot-1         |     cursor.execute(sql, params)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/cacheops/transaction.py", line 97, in execute
nauto_read-nautobot-1         |     result = self._no_monkey.execute(self, sql, params)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 66, in execute
nauto_read-nautobot-1         |     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
nauto_read-nautobot-1         |     return executor(sql, params, many, context)
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 79, in _execute
nauto_read-nautobot-1         |     with self.db.wrap_database_errors:
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/utils.py", line 90, in __exit__
nauto_read-nautobot-1         |     raise dj_exc_value.with_traceback(traceback) from exc_value
nauto_read-nautobot-1         |   File "/usr/local/lib/python3.10/site-packages/django/db/backends/utils.py", line 84, in _execute
nauto_read-nautobot-1         |     return self.cursor.execute(sql, params)
nauto_read-nautobot-1         | django.db.utils.InternalError: cannot execute INSERT in a read-only transaction

[joaopsys]

When it comes to that authentiction error @gneville-ot and @jathanism, I believe Nautobot still tries to update the last login time under MAINTENANCE_MODE in some cases, for example when it comes to remote user authentication.

The MAINTENANCE_MODE check exists in users/views.py - https://github.com/nautobot/nautobot/blob/21c85f25e6458ad97e62484bc89e5effa1328a80/nautobot/users/views.py#L67

However remote user authentication doesn’t seem to pass that check, it goes straight to core/middleware.py and is passed to Django, right? https://github.com/nautobot/nautobot/blob/21c85f25e6458ad97e62484bc89e5effa1328a80/nautobot/core/middleware.py#L23

I have fixed this by adding the following to core/middleware.py under process_request under class RemoteUserMiddleware(RemoteUserMiddleware_):

+ from django.contrib.auth.signals import user_logged_in
+ from django.contrib.auth.models import update_last_login
...
...
class RemoteUserMiddleware(RemoteUserMiddleware_):
...
    def process_request(self, request):
    ...
+        if settings.MAINTENANCE_MODE:
 +           user_logged_in.disconnect(update_last_login, dispatch_uid="update_last_login")

I have no clue if this is the right approach though, but it did fix it for me.

[glennmatthews]

At a glance, looks like an issue with social_core/social_django get_and_store_nonce(), probably https://python-social-auth.readthedocs.io/en/latest/storage.html. It looks like it might be necessary to find or write an alternative to DjangoAssociationMixin that understands Nautobot’s MAINTENANCE_MODE setting and doesn’t try to write to the DB in that case.