CLI 'kubernetes cleanup-pods' fails on invalid label key

Apache Airflow version: 2.0.2 Helm chart version: 1.0.0 Kubernetes version: 1.20

What happened: Airflow airflow-cleanup cronjob is failing with the error below. When I run the same command form the webserver or scheduler pod I got the same error.

> airflow@airflow-webserver-7f9f7954c-p9vv9:/opt/airflow$ airflow kubernetes  cleanup-pods --namespace airflow

Loading Kubernetes configuration
Listing pods in namespace airflow
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 8, in <module>
    sys.exit(main())
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/__main__.py", line 40, in main
    args.func(args)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/cli_parser.py", line 48, in command
    return func(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/utils/cli.py", line 89, in wrapper
    return f(*args, **kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/airflow/cli/commands/kubernetes_command.py", line 111, in cleanup_pods
    pod_list = kube_client.list_namespaced_pod(**list_kwargs)
  File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py", line 12803, in list_namespaced_pod
    (data) = self.list_namespaced_pod_with_http_info(namespace, **kwargs)  # noqa: E501
  File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api/core_v1_api.py", line 12905, in list_namespaced_pod_with_http_info
    collection_formats=collection_formats)
  File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 345, in call_api
    _preload_content, _request_timeout)
  File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 176, in __call_api
    _request_timeout=_request_timeout)
  File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/api_client.py", line 366, in request
    headers=headers)
  File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 241, in GET
    query_params=query_params)
  File "/home/airflow/.local/lib/python3.6/site-packages/kubernetes/client/rest.py", line 231, in request
    raise ApiException(http_resp=r)
kubernetes.client.rest.ApiException: (400)
Reason: Bad Request
HTTP response headers: HTTPHeaderDict({'Audit-Id': '53ee7655-f595-42a5-bdfb-689067a7fe02', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'e14ece85-9601-4034-9a43-7872ebabcbc5', 'X-Kubernetes-Pf-Prioritylevel-Uid': '72601873-fd48-4405-99dc-b7c4cac03b5c', 'Date': 'Sun, 23 May 2021 16:07:37 GMT', 'Content-Length': '428'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"unable to parse requirement: invalid label key \"{'matchExpressions':\": name part must consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyName',  or 'my.name',  or '123-abc', regex used for validation is '([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9]')","reason":"BadRequest","code":400}

How to reproduce it: Create and airflow deployment with Helm chart Enable automatic cleanup

cleanup:
  enabled: true

Run command airflow kubernetes cleanup-pods --namespace airflow

[XD-DENG]

Hi all, thanks to @dlampa , we have a fix for this issue now (https://github.com/apache/airflow/pull/17298).

The fix is merged into main branch now. I have marked it for milestone 2.1.3, so you should be able to expect having this fix in the next release 2.1.3.

[XD-DENG]

Hi @jean-malo , let’s continue the discussion at @dlampa 's PR #17298 ?

I will add my comments there later.

Thanks!