twarc2 search without configure on Windows throws JSON parse error

I ran the request below: twarc2 search ‘#ENDSARS-is:retweet’ --start-time 2017-12-01 --end-time 2020-11-30 --flatten --archive C:\Users\USER\Desktop\MyTwarcResults.json

and I got this error message below:

Traceback (most recent call last):
  File "C:\Users\USER\PycharmProjects\workspace\venv\Scripts\twarc2-script.py", line 33, in <module>
    sys.exit(load_entry_point('twarc==2.0.6', 'console_scripts', 'twarc2')())
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\click\core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\click\core.py", line 782, in main
    rv = self.invoke(ctx)
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\click\core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\click\core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\click\core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\click\decorators.py", line 33, in new_func
    return f(get_current_context().obj, *args, **kwargs)
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\twarc\decorators.py", line 172, in __call__
    result = e.response.json()
  File "c:\users\user\pycharmprojects\workspace\venv\lib\site-packages\requests\models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\json\__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\json\decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "C:\Users\USER\AppData\Local\Programs\Python\Python36\lib\json\decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

What exactly be the cause/source of this error, and how can i get help?

[osemele]

This is a special appreciation to you both for your constant support and perseverance. I am so glad to inform you that my twarc2 now works and generates data. I can’t thank you enough sirs, as all your responses were useful. I sincerely appreciate your patience. For the records, I think the improper configuration of the twarc2 (in addition to the twarc configuration) contributed to the reasons for getting those errors.

I will begin to work on twarc2 for my archival data collection now. I will be so glad if you would come to my rescue if problems arise.

Thank you once again for the support.

On Wed, May 5, 2021 at 6:32 PM Ed Summers @.***> wrote:

Yeah, a 400 error from the API is documented https://developer.twitter.com/en/support/twitter-api/error-troubleshooting#http-status-codes as:

The request was invalid or cannot be otherwise served. An accompanying error message will explain further. Requests without authentication or with invalid query parameters are considered invalid and will yield this response.

and then suggest:

Double check the format of your JSON query. For example, if your rule contains double-quote characters associated with an exact-match or other operator, you may need to escape them using a backslash to distinguish them from the structure of the JSON format.

But we’re not actually sending any JSON as part of the search/recent API call it’s just a GET. I guess if we set logging.level to DEBUG we might get some underlying information https://stackoverflow.com/questions/16337511/log-all-requests-from-the-python-requests-module from requests/urllib3? --log-level might actually be a nice option to have…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/DocNow/twarc/issues/441#issuecomment-832876616, or unsubscribe https://github.com/notifications/unsubscribe-auth/ATRWO6HBBPD7HMHPABYHBIDTMF6KJANCNFSM43M324CA .

– Kingsley Oladayo Ogunne

Department of Corporate Services Obafemi Awolowo University Teaching Hospitals Complex P.M.B. 5538 Ile-Ife, Nigeria

Telephone: [phone]

[edsu]

With @AbirRes’ help we were able to figure out that the bearer token was not persisted to the configuration file correctly. It was a ctrl-v character, which seemed to really confuse the Twitter API. I think the ctrl-v ended up in the configuration file because we were previously hiding the input of the token (for screen recording). It could be that some Windows terminals aren’t set up to do ctrl-v properly, and users could not see that it wasn’t working since it was hidden. Tokens should now appear in the console to help catch this in the future.

So if you have this problem, please make sure you are using twarc v2.1.5 or higher:

pip install --upgrade twarc

and then reconfigure twarc2:

twarc2 configure

Hopefully that will allow you to use twarc2 subcommands going forwards. Thanks for everyone’s patience on this!