Unable to use msgpack as result serializer (datetime object)

Checklist

• I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).
• I have included all related issues and possible duplicate issues in this issue.
• I have included the contents of pip freeze in the issue.
• I have verified that the issue exists against the master branch of Celery.
• I have tried reproducing the issue on more than one message broker and/or result backend.
• I have tried reproducing the issue on more than one workers pool.
• I have tried reproducing the issue with retries, ETA/Countdown & rate limits disabled.

Related Issues and Possible Duplicates

Related Issues

• msgpack repo: Support serializing datetime objects

Environment & Settings

Celery version: 4.3.0rc1 (rhubarb)

celery report Output:

software -> celery:4.3.0rc1 (rhubarb) kombu:4.3.0 py:3.5.6
            billiard:3.6.0.0 py-amqp:2.4.1
platform -> system:Linux arch:64bit, ELF
            kernel version:4.20.12-arch1-1-ARCH imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://localhost:6379/0

accept_content: ['json', 'msgpack']
result_backend: 'redis://localhost:6379/0'
broker_url: 'amqp://user:********@localhost:5672/test'
result_serializer: 'msgpack'

Steps to Reproduce

Python Packages

pip freeze Output:

amqp==2.4.1
billiard==3.6.0.0
celery==4.3.0rc1
kombu==4.3.0
msgpack==0.6.1
pytz==2018.9
redis==3.2.0
vine==1.2.0

Minimally Reproducible Test Case

celeryconfig.py

broker_url = 'amqp://user:password@localhost/test'
result_backend = 'redis://localhost:6379/0'

result_serializer = 'msgpack'
accept_content = ['json', 'msgpack']

tasks.py

from celery import Celery


app = Celery('tasks')
app.config_from_object('celeryconfig')


@app.task
def add(x, y): 
    return x + y 

For convenience, here is a docker-compose.yml file

version: '3'

networks:
  celery_msgpack:

services:
  redis:
    image: redis:latest
    ports:
      - "6379:6379"
    networks:
      - celery_msgpack
  rabbitmq:
    image: rabbitmq:latest
    environment:
      - RABBITMQ_DEFAULT_USER=user
      - RABBITMQ_DEFAULT_PASS=password
      - RABBITMQ_DEFAULT_VHOST=test
    ports:
      - "5672:5672"
      - "15672:15672"
    networks:
      - celery_msgpack

Then, you just need to run the task to reproduce the error.

Expected Behavior

Serializing results with msgpack should still be possible. However, as far as I know, datetime objects are not supported yet. Maybe this date should be represented as a string (isoformat ?).

Actual Behavior

It seems that, when storing the result, the output data is serialized, and a datetime object is added, which can not be serialized by msgpack. The commit that introduced the bug may be this commit. Here is the traceback:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 370, in pack
    return packb(s, use_bin_type=True)
  File "/usr/local/lib/python3.5/site-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 284, in msgpack._packer.Packer.pack
  File "msgpack/_packer.pyx", line 290, in msgpack._packer.Packer.pack
  File "msgpack/_packer.pyx", line 287, in msgpack._packer.Packer.pack
  File "msgpack/_packer.pyx", line 234, in msgpack._packer.Packer._pack
  File "msgpack/_packer.pyx", line 281, in msgpack._packer.Packer._pack
TypeError: can not serialize 'datetime.datetime' object

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 452, in trace_task
    I, R, state, retval = on_error(task_request, exc, uuid)
  File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 339, in on_error
    task, request, eager=eager, call_errbacks=call_errbacks,
  File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 172, in handle_error_state
    call_errbacks=call_errbacks)
  File "/usr/local/lib/python3.5/site-packages/celery/app/trace.py", line 217, in handle_failure
    call_errbacks=call_errbacks,
  File "/usr/local/lib/python3.5/site-packages/celery/backends/base.py", line 160, in mark_as_failure
    traceback=traceback, request=request)
  File "/usr/local/lib/python3.5/site-packages/celery/backends/base.py", line 331, in store_result
    request=request, **kwargs)
  File "/usr/local/lib/python3.5/site-packages/celery/backends/base.py", line 703, in _store_result
    self.set(self.get_key_for_task(task_id), self.encode(meta))
  File "/usr/local/lib/python3.5/site-packages/celery/backends/base.py", line 282, in encode
    _, _, payload = self._encode(data)
  File "/usr/local/lib/python3.5/site-packages/celery/backends/base.py", line 286, in _encode
    return dumps(data, serializer=self.serializer)
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.5/contextlib.py", line 77, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/usr/local/lib/python3.5/site-packages/vine/five.py", line 194, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 221, in dumps
    payload = encoder(data)
  File "/usr/local/lib/python3.5/site-packages/kombu/serialization.py", line 370, in pack
    return packb(s, use_bin_type=True)
  File "/usr/local/lib/python3.5/site-packages/msgpack/__init__.py", line 47, in packb
    return Packer(**kwargs).pack(o)
  File "msgpack/_packer.pyx", line 284, in msgpack._packer.Packer.pack
  File "msgpack/_packer.pyx", line 290, in msgpack._packer.Packer.pack
  File "msgpack/_packer.pyx", line 287, in msgpack._packer.Packer.pack
  File "msgpack/_packer.pyx", line 234, in msgpack._packer.Packer._pack
  File "msgpack/_packer.pyx", line 281, in msgpack._packer.Packer._pack
kombu.exceptions.EncodeError: can not serialize 'datetime.datetime' object

[dejlek]

As far as I know serialization of datetime never worked… We use msgpack for everything and we avoid datetime in arguments.

[trideceth12]

This is also affecting us. For now I will not go past 4.2. Are there any plans to address this issue?