cryptography.fernet.InvalidToken when attempting to access connections

Apache Airflow version: 2.1.0 (Helm chart) from main branch 5c7d758e24595c485553b0449583ff238114d47d

Kubernetes version (if you are using kubernetes) (use kubectl version):

Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.6+k3s1", GitCommit:"8d0432824a9fd9474b67138b7630c33f285d332f", GitTreeState:"clean", BuildDate:"2021-04-16T19:04:44Z", GoVersion:"go1.15.10", Compiler:"gc", Platform:"linux/amd64"}```

**Environment**:

- **Cloud provider or hardware configuration**:
- **OS** (e.g. from /etc/os-release): macOS Catalina 10.15.7 
- **Kernel** (e.g. `uname -a`): Darwin MT-207576 19.6.0 Darwin Kernel Version 19.6.0: Mon Apr 12 20:57:45 PDT 2021; root:xnu-6153.141.28.1~1/RELEASE_X86_64 x86_64
- **Install tools**: helm version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"dirty", GoVersion:"go1.15.5"}
- **Others**:

**What happened**:

After installaing Airflow and port forwarding from kubectl, when I navigate to the **connections** dropdown menu, I get the following trace

Ooops!

Something bad has happened. Please consider letting us know by creating a bug report using GitHub.

Python version: 3.6.13 Airflow version: 2.1.0 Node: airflow-webserver-67d745bff9-5sm7w

Traceback (most recent call last): File “/home/airflow/.local/lib/python3.6/site-packages/flask/app.py”, line 2447, in wsgi_app response = self.full_dispatch_request() File “/home/airflow/.local/lib/python3.6/site-packages/flask/app.py”, line 1952, in full_dispatch_request rv = self.handle_user_exception(e) File “/home/airflow/.local/lib/python3.6/site-packages/flask/app.py”, line 1821, in handle_user_exception reraise(exc_type, exc_value, tb) File “/home/airflow/.local/lib/python3.6/site-packages/flask/_compat.py”, line 39, in reraise raise value File “/home/airflow/.local/lib/python3.6/site-packages/flask/app.py”, line 1950, in full_dispatch_request rv = self.dispatch_request() File “/home/airflow/.local/lib/python3.6/site-packages/flask/app.py”, line 1936, in dispatch_request return self.view_functionsrule.endpoint File “/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/security/decorators.py”, line 109, in wraps return f(self, *args, **kwargs) File “/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/views.py”, line 551, in list widgets = self._list() File “/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/baseviews.py”, line 1134, in _list page_size=page_size, File “/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/baseviews.py”, line 1033, in get_list_widget page_size=page_size, File “/home/airflow/.local/lib/python3.6/site-packages/flask_appbuilder/models/sqla/interface.py”, line 435, in query query_results = query.all() File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/query.py”, line 3373, in all return list(self) File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py”, line 100, in instances cursor.close() File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py”, line 70, in exit with_traceback=exc_tb, File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/util/compat.py”, line 182, in raise raise exception File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py”, line 80, in instances rows = [proc(row) for row in fetch] File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py”, line 80, in <listcomp> rows = [proc(row) for row in fetch] File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/loading.py”, line 601, in _instance state.manager.dispatch.load(state, context) File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/event/attr.py”, line 322, in call fn(*args, **kw) File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/mapper.py”, line 3397, in _event_on_load instrumenting_mapper._reconstructor(state.obj()) File “/home/airflow/.local/lib/python3.6/site-packages/airflow/models/connection.py”, line 150, in on_db_load if self.password: File “/home/airflow/.local/lib/python3.6/site-packages/sqlalchemy/orm/attributes.py”, line 365, in get retval = self.descriptor.get(instance, owner) File “/home/airflow/.local/lib/python3.6/site-packages/airflow/models/connection.py”, line 235, in get_password return fernet.decrypt(bytes(self._password, ‘utf-8’)).decode() File “/home/airflow/.local/lib/python3.6/site-packages/cryptography/fernet.py”, line 194, in decrypt raise InvalidToken cryptography.fernet.InvalidToken

**What you expected to happen**:

I expected to see the Connections pane.

**How to reproduce it**:

git clone airflow && cd airflow/chart helm install airflow . -n airflow-test kubectl port-forward svc/airflow-webserver 8080:8080 --namespace airflow-test

Navigate to http://localhost:8080/connection/list/ and the trace appears. 

[Gurulhu]

I think I found what the issue might be: Inspecting the code it fails at password decryption so the issue wasn’t about having no token. My environment wasn’t properly clean between releases and the postgres RDS got init’d with a random-generated fernet key A, and the current release went up with another random generated fernet key B without ever rotating the key in the RDS, as it’s out of the helm delete scope. Wiping the RDS before spinning the chart up fixed it. (Test db, no useful data was harmed)

[github-actions[bot]]

This issue has been closed because it has not received response from the issue author.