Don't rely on the database to generate the schema

Describe the bug

To generate the schema for a project I am working on, the django project required a valid database connection. This makes generating the schema not easy.

I was always able to generate the schema without the database on other projects, and though this would be a bug. But I could be wrong and this is actually a requirement of DRF/spectacular ?

While there is probably a bug on our side, I though spectacular might want to handle this error, and not fail hard.

But I’ve done some digging, and maybe this can be fixed:

A trace when trying to generate the schema:

DJANGO_SETTINGS_MODULE=config.settings.testing \
	poetry run python manage.py spectacular --file schema.yml
2022-11-26 18:18:08,046 funkwhale_api.config INFO     Running with the following plugins enabled: funkwhale_api.contrib.scrobbler, funkwhale_api.contrib.listenbrainz, funkwhale_api.contrib.maloja
Warning #0: ActivityViewSet: AutoSerializer: registered extensions CookieTokenRefreshSerializerExtension for "dj_rest_auth.jwt_auth.CookieTokenRefreshSerializer" has an installed app but target class was not found.
Warning #1: AlbumViewSet: AlbumFilter: Unable to guess choice types from values, filter method's type hint or find "tag" in model. Defaulting to string.
Traceback (most recent call last):
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: could not connect to server: Connection refused
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/manage.py", line 29, in <module>
    execute_from_command_line(sys.argv)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/management/commands/spectacular.py", line 58, in handle
    schema = generator.get_schema(request=None, public=True)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/generators.py", line 268, in get_schema
    paths=self.parse(request, public),
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/generators.py", line 239, in parse
    operation = view.schema.get_operation(
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/openapi.py", line 117, in get_operation
    request_body = self._get_request_body()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/openapi.py", line 1330, in _get_request_body
    schema, request_body_required = self._get_request_for_media_type(
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/openapi.py", line 1367, in _get_request_for_media_type
    component = self.resolve_serializer(serializer, direction)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/openapi.py", line 1677, in resolve_serializer
    component.schema = self._map_serializer(
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/openapi.py", line 979, in _map_serializer
    schema = self._map_basic_serializer(serializer, direction)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/openapi.py", line 1074, in _map_basic_serializer
    schema = self._map_serializer_field(field, direction)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/drf_spectacular/openapi.py", line 959, in _map_serializer_field
    warn(f'could not resolve serializer field "{field}". Defaulting to "string"')
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/rest_framework/fields.py", line 707, in __repr__
    return representation.field_repr(self)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/rest_framework/utils/representation.py", line 55, in field_repr
    kwarg_string = ', '.join([
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/rest_framework/utils/representation.py", line 56, in <listcomp>
    '%s=%s' % (key, smart_repr(val))
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/rest_framework/utils/representation.py", line 33, in smart_repr
    value = repr(value)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 256, in __repr__
    data = list(self[:REPR_OUTPUT_SIZE + 1])
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 262, in __len__
    self._fetch_all()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 1324, in _fetch_all
    self._result_cache = list(self._iterable_class(self))
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 51, in __iter__
    results = compiler.execute_sql(chunked_fetch=self.chunked_fetch, chunk_size=self.chunk_size)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1173, in execute_sql
    cursor = self.connection.cursor()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 259, in cursor
    return self._cursor()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 235, in _cursor
    self.ensure_connection()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 219, in ensure_connection
    self.connect()
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/base/base.py", line 200, in connect
    self.connection = self.get_new_connection(conn_params)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/utils/asyncio.py", line 33, in inner
    return func(*args, **kwargs)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 187, in get_new_connection
    connection = Database.connect(**conn_params)
  File "/home/jo/git/dev.funkwhale.audio/funkwhale/funkwhale/api/.venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: could not connect to server: Connection refused
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?

make: *** [Makefile:18: schema] Error 1

From this trace I was able to find: https://github.com/tfranzel/drf-spectacular/blob/master/drf_spectacular/openapi.py#L870 And maybe also https://github.com/tfranzel/drf-spectacular/blob/master/drf_spectacular/openapi.py#L861

Which seemed to trigger a smart_repr for some fields, and somehow required the database (relation name ?).

I changed the line with:

        warn(f'could not resolve serializer field "{field.field_name}". Defaulting to "string"')

And now the schema generation completes, even if I still have a database connection error in the middle:

DJANGO_SETTINGS_MODULE=config.settings.testing \
	poetry run python manage.py spectacular --file schema.yml
2022-11-26 18:36:00,151 funkwhale_api.config INFO     Running with the following plugins enabled: funkwhale_api.contrib.scrobbler, funkwhale_api.contrib.listenbrainz, funkwhale_api.contrib.maloja
Warning #0: ActivityViewSet: AutoSerializer: registered extensions CookieTokenRefreshSerializerExtension for "dj_rest_auth.jwt_auth.CookieTokenRefreshSerializer" has an installed app but target class was not found.
Warning #1: AlbumViewSet: AlbumFilter: Unable to guess choice types from values, filter method's type hint or find "tag" in model. Defaulting to string.
Warning #2: AlbumViewSet: AlbumCreateSerializer: could not resolve serializer field "cover". Defaulting to "string"
Warning #3: AlbumViewSet: AlbumCreateSerializer: could not resolve serializer field "artist". Defaulting to "string"
Warning #4: ArtistViewSet: ArtistFilter: Unable to guess choice types from values, filter method's type hint or find "tag" in model. Defaulting to string.
Error #0: RegisterView: exception raised while getting serializer. Hint: Is get_serializer_class() returning None or is get_queryset() not working without a request? Ignoring the view for now. (Exception: could not connect to server: Connection refused
	Is the server running on host "localhost" (::1) and accepting
	TCP/IP connections on port 5432?
could not connect to server: Connection refused
	Is the server running on host "localhost" (127.0.0.1) and accepting
	TCP/IP connections on port 5432?
)
Warning #5: ChannelViewSet: ChannelFilter: Unable to guess choice types from values, filter method's type hint or find "tag" in model. Defaulting to string.
Warning #6: ChannelViewSet: ChannelCreateSerializer: could not resolve serializer field "cover". Defaulting to "string"
Warning #7: ChannelViewSet: ChannelUpdateSerializer: could not resolve serializer field "cover". Defaulting to "string"
Warning #8: LibraryFollowViewSet: LibraryFollowSerializer: could not resolve serializer field "target". Defaulting to "string"
Warning #9: AdminSettings: GlobalPreferenceSerializer: unable to resolve type hint for function "get_identifier". Consider using a type hint or @extend_schema_field. Defaulting to string.
Warning #10: AdminSettings: GlobalPreferenceSerializer: unable to resolve type hint for function "get_default". Consider using a type hint or @extend_schema_field. Defaulting to string.
Warning #11: AdminSettings: GlobalPreferenceSerializer: could not resolve serializer field "value". Defaulting to "string"
Warning #12: AdminSettings: GlobalPreferenceSerializer: unable to resolve type hint for function "get_verbose_name". Consider using a type hint or @extend_schema_field. Defaulting to string.
Warning #13: AdminSettings: GlobalPreferenceSerializer: unable to resolve type hint for function "get_help_text". Consider using a type hint or @extend_schema_field. Defaulting to string.
Warning #14: AdminSettings: GlobalPreferenceSerializer: unable to resolve type hint for function "get_additional_data". Consider using a type hint or @extend_schema_field. Defaulting to string.
Warning #15: AdminSettings: GlobalPreferenceSerializer: unable to resolve type hint for function "get_field". Consider using a type hint or @extend_schema_field. Defaulting to string.
Warning #16: @extend_schema_view argument "get" was not found on view ListenViewSet. method override for "get" will be ignored.
Warning #17: ManageUserRequestViewSet: ManageUserRequestSerializer: Encountered 2 components with identical names "ManageUserRequest" and different classes <class 'funkwhale_api.manage.serializers.ManageUserRequestSerializer'> and <class 'funkwhale_api.manage.serializers.ManageUserSerializer'>. This will very likely result in an incorrect schema. Try renaming one.
Warning #18: PlaylistViewSet: PlaylistTrackSerializer: unable to resolve type hint for function "get_track". Consider using a type hint or @extend_schema_field. Defaulting to string.
Warning #19: PluginViewSet: Serializer: Component name "" contains illegal characters. Only "A-Z a-z 0-9 - . _" are allowed. Furthermore, "-" and "." are discoursed due to potential tooling issues. This likely leads to an invalid schema.
Warning #20: RadioViewSet: FilterSerializer: Could not derive type for ReadOnlyField "fields" because the serializer class has no associated model (Meta class). Consider using some other field like CharField(read_only=True) instead. defaulting to string.
Warning #21: TrackViewSet: TrackFilter: Unable to guess choice types from values, filter method's type hint or find "tag" in model. Defaulting to string.
Warning #22: TrackViewSet: TrackFilter: Unable to guess choice types from values, filter method's type hint or find "artist" in model. Defaulting to string.
Warning #23: UploadViewSet: UploadForOwnerSerializer: could not resolve serializer field "library". Defaulting to "string"
Warning #24: UploadViewSet: UploadForOwnerSerializer: could not resolve serializer field "channel". Defaulting to "string"
Warning #25: UploadViewSet: TrackMetadataSerializer: could not resolve serializer field "album". Defaulting to "string"
Warning #26: UploadViewSet: TrackMetadataSerializer: could not resolve serializer field "artists". Defaulting to "string"
Warning #27: UploadViewSet: TrackMetadataSerializer: could not resolve serializer field "cover_data". Defaulting to "string"
Warning #28: UserViewSet: UserWriteSerializer: could not resolve serializer field "avatar". Defaulting to "string"
Warning #29: operationId "change_password" has collisions [('/api/v1/auth/password/change/', 'post'), ('/api/v1/auth/registration/change-password/', 'post')]. resolving with numeral suffixes.
Warning #30: operationId "unsubscribe_channel" has collisions [('/api/v1/channels/{composite}/unsubscribe/', 'post'), ('/api/v1/channels/{composite}/unsubscribe/', 'delete')]. resolving with numeral suffixes.
Warning #31: operationId "unfavorite_track" has collisions [('/api/v1/favorites/tracks/remove/', 'post'), ('/api/v1/favorites/tracks/remove/', 'delete')]. resolving with numeral suffixes.
Warning #32: operationId "remove_from_playlist" has collisions [('/api/v1/playlists/{id}/remove/', 'post'), ('/api/v1/playlists/{id}/remove/', 'delete')]. resolving with numeral suffixes.

Schema generation summary:
Warnings: 79 (33 unique)
Errors:   4 (1 unique)

But this final error, I can still fix later on.

To Reproduce I couldn’t extract a snippet to reproduce this, but here is the project repository: https://dev.funkwhale.audio/funkwhale/funkwhale/

Here is the CI job that generates the schema, I would like to remove the database requirement. https://dev.funkwhale.audio/funkwhale/funkwhale/-/blob/ad3674e29e51e80875b2568ab3b87f4739b992bd/.gitlab-ci.yml#L251-L278

Expected behavior I would like to generate the schema without the any database connection required, and if a bug occur our side, spectacular should throw a usefull warning.

[jooola]

Waouw @tfranzel Thanks a lot for you insight, didn’t expect that much. I was not even sure if I should keep the issue open as it was a problem on our side.

Yesterday after writing this issue, I had another look at the code, and we have something weird in RegisterSerializer: https://dev.funkwhale.audio/funkwhale/funkwhale/-/blob/develop/api/funkwhale_api/users/serializers.py#L45-L59

The preferences.get() calls are indeed trying to access the database. We should change that. I am unsure if this is the cause of the bug, but looks pretty much like it is, haven’t played around yet.

I have noted all your ideas and notes, I’ll have to take some time to understand and try them out.

[tfranzel]

closing this issue for now. feel free to comment if anything is missing or not working and we will follow-up.