JSONDecodeError: sending mail broken since a few days

Hello,

since a few days I cannot send emails anymore, I’m afraid Microsoft may send some additional garbage in the JSON:

/usr/lib/python3/dist-packages/apport/report.py:13: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
  import fnmatch, glob, traceback, errno, sys, atexit, locale, imp, stat
Traceback (most recent call last):
  File "/home/nagios/mailo365-debug2", line 38, in <module>
    m.send()
  File "/usr/local/lib/python3.6/dist-packages/O365/message.py", line 695, in send
    response = self.con.post(url, data=data)
  File "/usr/local/lib/python3.6/dist-packages/O365/connection.py", line 819, in post
    return self.oauth_request(url, 'post', data=data, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/O365/connection.py", line 795, in oauth_request
    self.session = self.get_session(load_token=True)
  File "/usr/local/lib/python3.6/dist-packages/O365/connection.py", line 561, in get_session
    token = self.token_backend.get_token()
  File "/usr/local/lib/python3.6/dist-packages/O365/utils/token.py", line 94, in get_token
    self.token = self.load_token()  # store the token in the 'token' property
  File "/usr/local/lib/python3.6/dist-packages/O365/utils/token.py", line 187, in load_token
    token = self.token_constructor(self.serializer.load(token_file))
  File "/usr/lib/python3.6/json/__init__.py", line 299, in load
    parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 342, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 16 column 2 (char 3574)

Example code (generating the above Traceback):

#!/usr/bin/env python3
# coding: utf8

import sys
import argparse
from O365 import Account,FileSystemTokenBackend

SITELOCAL_APP_ID = 'ads'
SITELOCAL_APP_SECRET = 'dsa'
SITELOCAL_TKM_PATH = '/home/nagios/'
SITELOCAL_TKM_FILE = 'o365_token'

credentials = (SITELOCAL_APP_ID, SITELOCAL_APP_SECRET)
token_backend = FileSystemTokenBackend(token_path=SITELOCAL_TKM_PATH, token_filename=SITELOCAL_TKM_FILE)

account = Account(credentials, token_backend=token_backend)

# just for interactive initialization
#if account.authenticate(scopes=['basic', 'message_all']):
#  print('Authenticated!')

m = account.new_message()

to="[e-mail]"
mailfrom="[e-mail]"
subject="Testsubject"
body="""
Hey

you
"""


m.to.add(to)
if mailfrom: m.sender.address = mailfrom
if subject: m.subject = subject
m.body = body

m.send()

How can I troubleshoot this? Can you reproduce this?

[janscas]

I don’t use it so …no.

But I think it’s pretty well explain in comments.

Also check token.py in utils so you can read the should_refresh_token doctsrings

[lukastribus]

I added a print statement in /usr/lib/python3.6/json/decoder.py and saw the following - invalid - JSON:

{
 "token_type": "Bearer",
 "scope": [
  "profile",
  "openid",
  "email",
  "https://graph.microsoft.com/Mail.ReadWrite",
  "https://graph.microsoft.com/Mail.Send",
  "https://graph.microsoft.com/User.Read"
 ],
 "expires_in": 3599,
 "ext_expires_in": 3599,
 "access_token": "<omitted>",
 "refresh_token": "<omitted>",
 "expires_at": 1621261486.396715
}547
}

Don’t know where the 547 comes from, but it’s not JSON. Of course if the script keeps crashing due to invalid JSON at some point the token will expire.

I uncommented the account.authenticate call and authenticated again, now it works.

Looking at the call-trace again I can now see that the invalid JSON actually came from FileSystemTokenBackend on the disk, not a response from Microsoft. So this looks more like a local fat finger root cause.