Malformed value for primitive Edm.DateTimeOffset type

We are using pyodata v1.7.1 the whole time and I’m tried to update to the latest v1.10.0 version. But now I get this error when querying data from the odata service:

Traceback (most recent call last):
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/model.py", line 534, in from_json
    milliseconds_since_epoch = matches.group('milliseconds_since_epoch')
AttributeError: 'NoneType' object has no attribute 'group'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/prefect/engine.py", line 1192, in orchestrate_task_run
    result = await run_sync(task.fn, *args, **kwargs)
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 57, in run_sync_in_worker_thread
    return await anyio.to_thread.run_sync(call, cancellable=True)
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/workspace/odatavenv/flow_utilities/prefect/tasks/odata.py", line 60, in fetch_odata
    for cnt, batch in enumerate(batches, start=1):
  File "/workspace/odatavenv/flow_utilities/api/odata.py", line 114, in query_all
    yield list(self.query(skip, top))
  File "/workspace/odatavenv/flow_utilities/api/odata.py", line 98, in query
    response = self._entity_service().skip(skip).top(top).execute()
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 349, in execute
    return self._call_handler(response)
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 362, in _call_handler
    return self._handler(response)
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 1509, in get_entities_handler
    entity = EntityProxy(self._service, self._entity_set, self._entity_set.entity_type, props)
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/service.py", line 814, in __init__
    self._cache[type_proprty.name] = type_proprty.from_json(proprties[type_proprty.name])
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/model.py", line 872, in from_json
    return self.typ.traits.from_json(value)
  File "/workspace/tmp/venv/odatavenv/lib/python3.10/site-packages/pyodata/v2/model.py", line 537, in from_json
    raise PyODataModelError(
pyodata.exceptions.PyODataModelError: Malformed value /Date(1599659339088)/ for primitive Edm.DateTimeOffset type. Expected format is /Date(<ticks>±<offset>)/

Maybe this is related to our instance because they are uncompatible dates (see https://github.com/SAP/python-pyodata/issues/143).

[phanak-sap]

OK, it seems like something to change for Edm.DateTimeOffset parsing of JSON values.

We are currently expecting that the offset part is mandatory and that your service should return /Date(1659430517461+0000)/ for offset set to UTC. I am not able to find in the odata.org specification exactly if the offset part is mandatory, nor in the W3C XML schema. But it seems that at least SAP thinks this should be correct - “As of Release Q1 2019, DateTimeOffset values without timezone information are also accepted in query URLs and edit request payloads. The value will be processed by the server as UTC time.”

So the fix should be to apply this logic as well - if the offset part is missing, work as if +0000 was provided. But IMHO this should be just for the from_json logic, the to_json should always contain offset information.

Relevant code: https://github.com/SAP/python-pyodata/blob/5d1490871f4b824a82dd6eaa148444a99ea4f47d/pyodata/v2/model.py#L531 https://github.com/SAP/python-pyodata/blob/5d1490871f4b824a82dd6eaa148444a99ea4f47d/tests/test_model_v2.py#L749

Possible quick/ugly workaround:

Seems to me that the literal parsing is correct. Could you try with the call with $value for literal form of the specific entity? IMHO this modification for your script should work (if not, maybe secondary bug 😃 ):

for d in data.execute().get_value():

Longer backgroung explanation:

IMHO it is working on old version, well, just because in 1.7.1 the Edm.DateTimeOffset was effectively “not implemented”, with no Traits and in both literal and json format it was just passing the value up trough base class…

links to 1.7.1 tag:

https://github.com/SAP/python-pyodata/blob/5731a4cac63fd2fdceb28f061a71fa078a7f454b/pyodata/v2/model.py#L213

https://github.com/SAP/python-pyodata/blob/5731a4cac63fd2fdceb28f061a71fa078a7f454b/pyodata/v2/model.py#L343

Now, it is worth going trough the commit ec56995551779cb9 that implements the Edm.DateTimeOffset as starting point.

https://blogs.sap.com/2017/01/05/date-and-time-in-sap-gateway-foundation/

[phanak-sap]

Hi @sti0, check the PR #232 - should this work for you? Also, were you able to use the workaround with the get_value()?