kubernetes_asyncio regression: AttributeError("'ClientResponse' object has no attribute 'getheader'") in ResourceReflector

Bug description

AttributeError("'ClientResponse' object has no attribute 'getheader'") is caused and spawning status of user server is not updated, while the corresponding pod is spawned correctly and is ready on k8s.

The following error log is found in the jupyterhub container log:

Traceback (most recent call last):
  File ""/usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py"", line 392, in start
    await self._list_and_update()
  File ""/usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py"", line 236, in _list_and_update
    initial_resources_raw = await list_method(**kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/kubernetes_asyncio/client/api_client.py"", line 191, in __call_api
    content_type = response_data.getheader('content-type')
AttributeError: 'ClientResponse' object has no attribute 'getheader'
future: <Task finished name='Task-5' coro=<ResourceReflector.start() done, defined at /usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py:379> exception=AttributeError("'ClientResponse' object has no attribute 'getheader'")>
ERROR:asyncio:Task exception was never retrieved
Traceback (most recent call last):
  File ""/usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py"", line 392, in start
    await self._list_and_update()
  File ""/usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py"", line 236, in _list_and_update
    initial_resources_raw = await list_method(**kwargs)
  File ""/usr/local/lib/python3.8/dist-packages/kubernetes_asyncio/client/api_client.py"", line 191, in __call_api
    content_type = response_data.getheader('content-type')
AttributeError: 'ClientResponse' object has no attribute 'getheader'"
future: <Task finished name='Task-4' coro=<ResourceReflector.start() done, defined at /usr/local/lib/python3.8/dist-packages/kubespawner/reflector.py:379> exception=AttributeError("'ClientResponse' object has no attribute 'getheader'")>
ERROR:asyncio:Task exception was never retrieved

It seems that this error is caused by specification of _preload_content=False in kubespawner/reflector.py, as kubenetes_asyncio returns raw aiohttp.ClientResponse object (which does not have getheader method) instead of RESTResponse object (eqipped with getheader method) that wraps ClientResponse, when _preload_content is set False. Evidence of this behavior in kubernetes_asyncio can be found at: https://github.com/tomplus/kubernetes_asyncio/blob/da1e3172acd098e5feb47193bd9d73428078b61a/kubernetes_asyncio/client/rest.py#L178-L189

Your personal set up

• OS: ubuntu 20.04
• Version(s): jupyterhub 2.2.0, python 3.8.10
• kubespawner : main:HEAD @ 0b6ad95

[takeshi-yashiro]

Thank you! I confirmed that the problem is solved by updating kubernetes_asyncio to 21.7.1, just released by tomplus/kubernetes_asyncio#183.

[consideRatio]

Hmmm… But this section, has been removed from the code base of kubernetes_asyncio I thought…

  File ""/usr/local/lib/python3.8/dist-packages/kubernetes_asyncio/client/api_client.py"", line 191, in __call_api
    content_type = response_data.getheader('content-type')

Well, looking at the code instead of using github search, I find it: https://github.com/tomplus/kubernetes_asyncio/blob/da1e3172acd098e5feb47193bd9d73428078b61a/kubernetes_asyncio/client/api_client.py#L191

Making some git blame inspection, I conclude it is an upstream regression recently introduced. I verified that by re-running our test suite against the recently released kubernetes_asyncio package. I opened https://github.com/tomplus/kubernetes_asyncio/issues/183 about this.