Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

SSL error: decryption failed or bad record mac & SSL SYSCALL error: EOF detected

See original GitHub issue

Environment

Python version: 3.8.5
Nautobot version: 1.0.0b2

Steps to Reproduce

become user nautobot
Download & extract 1.0.0b2
Execute install.ssh (use existing config except unicorn.py; Adapt changes in systemd service files)
Start nautobot
Open Nautobot WebUI and surf a bit

This only happens after a fresh start of nautobot. The error disappears after a while until a nautobot restart.

Expected Behavior

no HTTP500

Observed Behavior

fyi: /opt/nautobot/current is a symlink to /opt/nautobot/1.0.0b2/

Server Error
There was a problem with your request. Please contact an administrator.

The complete exception is provided below:

<class 'django.db.utils.OperationalError'>

SSL error: decryption failed or bad record mac

Python version: 3.8.5
Nautobot version: 1.0.0b2

OperationalError at /plugins/device-onboarding/
SSL error: decryption failed or bad record mac
Request Method:	GET
Request URL:	https://<schnipp>/plugins/device-onboarding/
Django Version:	3.1.7
Exception Type:	OperationalError
Exception Value:	
SSL error: decryption failed or bad record mac
Exception Location:	/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py, line 84, in _execute
Python Executable:	/usr/bin/python3.8
Python Version:	3.8.5
Python Path:	
['.',
 '',
 '/opt/nautobot/1.0.0b2/venv/bin',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/git/ext/gitdb',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/gitdb/ext/smmap']
Server time:	Wed, 10 Mar 2021 12:34:59 +0000

Environment:


Request Method: GET
Request URL: https://<schnipp>/plugins/device-onboarding/

Django Version: 3.1.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'taggit',
 'timezone_field',
 'nautobot.core',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'drf_yasg',
 'graphene_django',
 'nautobot_device_onboarding.OnboardingConfig']
Installed Middleware:
['django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.APIVersionMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The above exception (SSL error: decryption failed or bad record mac
) was the direct cause of the following exception:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/views.py", line 91, in dispatch
    if not self.has_permission():
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/views.py", line 73, in has_permission
    if user.has_perms((permission_required, *self.additional_permissions)):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 376, in _setup
    self._wrapped = self._setupfunc()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
    user_id = _get_user_session_key(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__
    return self._session[key]
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
    self._session_cache = self.load()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
    s = self._get_session_from_db()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
    return self.model.objects.get(
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 273, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

Exception Type: OperationalError at /plugins/device-onboarding/
Exception Value: SSL error: decryption failed or bad record mac

Seconde error:

OperationalError at /
SSL SYSCALL error: EOF detected
Request Method:	GET
Request URL:	https://<schnipp>/
Django Version:	3.1.7
Exception Type:	OperationalError
Exception Value:	
SSL SYSCALL error: EOF detected
Exception Location:	/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py, line 84, in _execute
Python Executable:	/usr/bin/python3.8
Python Version:	3.8.5
Python Path:	
['.',
 '',
 '/opt/nautobot/1.0.0b2/venv/bin',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/git/ext/gitdb',
 '/usr/lib/python38.zip',
 '/usr/lib/python3.8',
 '/usr/lib/python3.8/lib-dynload',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages',
 '/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/gitdb/ext/smmap']
Server time:	Wed, 10 Mar 2021 13:06:33 +0000

Environment:


Request Method: GET
Request URL: https://<schnipp>/

Django Version: 3.1.7
Python Version: 3.8.5
Installed Applications:
['django.contrib.admin',
 'django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',
 'django.contrib.messages',
 'django.contrib.staticfiles',
 'django.contrib.humanize',
 'cacheops',
 'corsheaders',
 'django_filters',
 'django_tables2',
 'django_prometheus',
 'mptt',
 'rest_framework',
 'taggit',
 'timezone_field',
 'nautobot.core',
 'nautobot.circuits',
 'nautobot.dcim',
 'nautobot.ipam',
 'nautobot.extras',
 'nautobot.tenancy',
 'nautobot.users',
 'nautobot.utilities',
 'nautobot.virtualization',
 'django_rq',
 'drf_yasg',
 'graphene_django',
 'nautobot_device_onboarding.OnboardingConfig']
Installed Middleware:
['django_prometheus.middleware.PrometheusBeforeMiddleware',
 'corsheaders.middleware.CorsMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.common.CommonMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware',
 'django.middleware.clickjacking.XFrameOptionsMiddleware',
 'django.middleware.security.SecurityMiddleware',
 'nautobot.core.middleware.ExceptionHandlingMiddleware',
 'nautobot.core.middleware.RemoteUserMiddleware',
 'nautobot.core.middleware.APIVersionMiddleware',
 'nautobot.core.middleware.ObjectChangeMiddleware',
 'django_prometheus.middleware.PrometheusAfterMiddleware']



Traceback (most recent call last):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 233, in _get_session
    return self._session_cache

During handling of the above exception ('SessionStore' object has no attribute '_session_cache'), another exception occurred:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

The above exception (SSL SYSCALL error: EOF detected
) was the direct cause of the following exception:
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 70, in view
    return self.dispatch(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/views/generic/base.py", line 98, in dispatch
    return handler(request, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/core/views/__init__.py", line 45, in get
    ConsolePort.objects.restrict(request.user, "view")
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/nautobot/utilities/querysets.py", line 21, in restrict
    if user.is_superuser or permission_is_exempt(permission_required):
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 240, in inner
    self._setup()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/utils/functional.py", line 376, in _setup
    self._wrapped = self._setupfunc()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 23, in <lambda>
    request.user = SimpleLazyObject(lambda: get_user(request))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/middleware.py", line 11, in get_user
    request._cached_user = auth.get_user(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 174, in get_user
    user_id = _get_user_session_key(request)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/auth/__init__.py", line 58, in _get_user_session_key
    return get_user_model()._meta.pk.to_python(request.session[SESSION_KEY])
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 65, in __getitem__
    return self._session[key]
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/base.py", line 238, in _get_session
    self._session_cache = self.load()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 43, in load
    s = self._get_session_from_db()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/contrib/sessions/backends/db.py", line 32, in _get_session_from_db
    return self.model.objects.get(
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 353, in get
    return qs._no_monkey.get(qs, *args, **kwargs)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 425, in get
    num = len(clone)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 269, in __len__
    self._fetch_all()
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/query.py", line 273, in _fetch_all
    return self._no_monkey._fetch_all(self)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1308, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/query.py", line 53, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1156, in execute_sql
    cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
    return super().execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/cacheops/transaction.py", line 93, in execute
    result = self._no_monkey.execute(self, sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/nautobot/1.0.0b2/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)

Exception Type: OperationalError at /
Exception Value: SSL SYSCALL error: EOF detected

Issue Analytics

State:
Created 3 years ago
Comments:23 (16 by maintainers)

Top GitHub Comments

2reactions

jathanismcommented, Mar 13, 2021

This only affects Ubuntu systems where PostgreSQL was installed via apt, in which SSL is enabled by default. You can confirm this by connecting to the database using psql and running \conninfo:

$ psql nautobot -U nautobot -h localhost --password
Password:
psql (12.6 (Ubuntu 12.6-0ubuntu0.20.04.1))
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)
Type "help" for help.

nautobot=> \conninfo
You are connected to database "nautobot" as user "nautobot" on host "localhost" (address "::1") at port "5432".
SSL connection (protocol: TLSv1.3, cipher: TLS_AES_256_GCM_SHA384, bits: 256, compression: off)

After more debugging, I can 100% get the issue to go away if I do any one of these:

Don’t use processes = in uwsgi.ini or --processes CLI flag when starting uWSGI, which doesn’t result in pre-forking
Disable SSL connection to PGSQL by setting DATABASES['default']['OPTIONS'] = {"sslmode": "disable"} in nautobot_config.py
Disable persistent connections by setting DATABASES['default']['CONN_MAX_AGE'] = 0 in nautobot_config.py, which is the Django default to always close the connection on every request.

Obviously the first two are not desirable for production use. The third option in my opinion should be our new default for the common case if we’re going to continue to recommend uWSGI over Gunicorn.

The root cause is the aggressive pre-forking of uWSGI which performs copy-on-write on subprocesses that often duplicates the database connection objects. This issue did not arise when using Gunicorn because its default sync worker class defers to every worker handling only a single request before cycling and deliberately closing the database connection. This mirrors the default behavior of Django with CONN_MAX_AGE=0, which is why when I set that, uWSGI behaves as expected.

1reaction

chrismargetcommented, Mar 15, 2021

@mkrsn and @jathanism, thanks for having this conversation. I’ve run into the same problem (using the new install instructions, on Ubutntu 20.04, using the distro-supplied PostgreSQL), and have worked around it by setting the database MAX_CONN_AGE to zero.

Top Results From Across the Web

python multiprocessing + peewee + postgresql fails with SSL ...

I am trying to write a Python model which is capable of doing some processing in a PostgreSQL database using the multi-threading module...

psql: ssl syscall error: eof detected azure

no connection to the server; SSL error: decryption failed or bad record mac; could not receive data from server: Connection timed out; These...

OperationalError: SSL SYSCALL error: EOF detected Issue ...

Dear all, I keep getting this error with all deployment scenarios from docker, ... LOG: SSL error: decryption failed or bad record mac...

psycopg2.OperationalError: SSL when running ci_hsc_gen3

I had found result that talked about TCP keepalive so I tried pool_pre_ping=True in the create_engine call, but that didn't seem to make...

SQLAlchemy + multiprocessing problems - Google Groups

OperationalError: SSL error: sslv3 alert bad record mac; sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected.