Error: object of type 'NoneType' has no len()

Hi guys!

We’ve been using Procrastinate with great results in production for a while.

Recently I went to add another worker, nearly identical in its configuration to a worker we are running today, just with different name, queues, and import paths.

However, I’m getting a cryptic error and the worker is unable to start/encounters the following exception:

(⎈ minikube:cloud)➜  dev git:(mc/sift-integration) ✗ k logs -n cloud deployments/cloud-app-worker-sift -f
PROCRASTINATE_IMPORT_PATHS ['app.gql_helpers']
INFO:procrastinate.periodic:Registering task app.procrastinate.retry_stalled_k8s_jobs to run periodically with cron */1 * * * *
Launching a worker on sift
Error: object of type 'NoneType' has no len()

The only place in the source code where I even see len() being used is here: https://github.com/procrastinate-org/procrastinate/blob/main/procrastinate/utils.py#L151

Here is the configuration for both the erroring sift worker and the k8s worker from which I mostly copied the worker process configuration:

queueWorkers:
  sift:
    resources: {}
    tolerations: {}
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: app
                  operator: In
                  values: [ 'queue-worker-sift' ]
            topologyKey: kubernetes.io/hostname
    nodeSelector: ""
    replicaCount: 1
    args:
      - "--app=app.procrastinate.async_app"
      - "worker"
      - "--name=sift-worker"
      - "--queues=sift"
      - "--concurrency=5"
      - "--delete-jobs=successful"
    databaseMinConnections: 1
    databaseMaxConnections: 2
    procrastinateImportPaths: [ 'app.gql_helpers' ]
    procrastinateMaxConn: 2
  k8s:
    resources: {}
    tolerations: {}
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
            - key: app
              operator: In
              values: ['queue-worker-k8s']
          topologyKey: kubernetes.io/hostname
    nodeSelector: ""
    replicaCount: 1
    args:
      - "--app=app.procrastinate.async_app"
      - "worker"
      - "--name=k8s-worker"
      - "--queues=k8s"
      - "--concurrency=10"
      - "--delete-jobs=successful"
    databaseMinConnections: 1
    databaseMaxConnections: 2
    procrastinateImportPaths: [ 'app.reconciler', 'app.gql_mutations' ]
    procrastinateMaxConn: 2

And here is our app/procrastinate.py code:

import asyncio
from functools import wraps

from procrastinate import AiopgConnector, App, Psycopg2Connector
from procrastinate.exceptions import UniqueViolation

from app.sentry import init_sentry
from app.settings import (
    DATABASE_URL,
    PROCRASTINATE_IMPORT_PATHS,
    PROCRASTINATE_MAX_CONN,
    TEST_ENV,
    database,
    env,
    get_db_conn_app_name,
)

init_sentry()

async_dsn = f'{DATABASE_URL}?application_name={get_db_conn_app_name(["async", "app", "connector"])}'
sync_dsn = f'{DATABASE_URL}?application_name={get_db_conn_app_name(["sync", "app", "connector"])}'

async_app_connector = AiopgConnector(
    dsn=async_dsn, maxsize=PROCRASTINATE_MAX_CONN, pool_recycle=120  # two minutes
)

async_app = App(
    connector=async_app_connector,
    import_paths=PROCRASTINATE_IMPORT_PATHS,
)

# Periodic tasks should not run in the test env to avoid unexpected behavior
if env != TEST_ENV:
    RUNNING_JOBS_MAX_TIME = (
        60  # time in seconds for running jobs to be deemed as stalled
    )

    @async_app.periodic(cron="*/1 * * * *")
    @async_app.task(
        queue="k8s", queueing_lock="retry_stalled_k8s_jobs", pass_context=True
    )
    async def retry_stalled_k8s_jobs(context, timestamp):
        stalled_jobs = await async_app.job_manager.get_stalled_jobs(
            queue="k8s", nb_seconds=RUNNING_JOBS_MAX_TIME
        )
        for job in stalled_jobs:
            try:
                await async_app.job_manager.retry_job(job)
            except UniqueViolation:
                # raised when a job with the same queuing_lock is already
                # scheduled in the 'todo' state
                pass


database_lock = asyncio.Lock()


def task_with_database(*args, **kwargs):
    def wrap(coro):
        @wraps(coro)
        async def new_coro(*job_args, **job_kwargs):
            if not database.is_connected:
                async with database_lock:
                    # while waiting for the lock, another job may have
                    # connected the database already
                    if not database.is_connected:
                        print("Connecting to the database...")
                        await database.connect()
            return await coro(*job_args, **job_kwargs)

        return async_app.task(*args, **kwargs)(new_coro)

    return wrap

Do you have any leads as to why we may be seeing this error? Thank you 🙏

[ewjoachim]

Ah so you closing the issue tells me that the exception was a side effect of your code being imported by Procrastinate, and not a problem in Procrastinate itself, right ? In that case, glad to be useful 😃 (And glad I added this feature to procrastinate some time ago 😄 )

[ewjoachim]

(BTW, if you’re curious here’s the line that logs any exception as DEBUG, making the rest automatically linked to the log level: https://github.com/procrastinate-org/procrastinate/blob/064c5d10de296014062c9ad80befbfba4c09aa01/procrastinate/cli.py#L52 and here’s the line that selects a log level based on the amount of -v args: https://github.com/procrastinate-org/procrastinate/blob/064c5d10de296014062c9ad80befbfba4c09aa01/procrastinate/cli.py#L26-L31

ok, that’s slightly over-engineered given that level 0 is “INFO”, level 1 is “DEBUG”, and all levels above one are also “DEBUG” 😄 )