Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Unhandled `UnicodeDecodeError` exception if response with status 400 and request contains binary payload (for FastHttpUser)

See original GitHub issue

Bug

I get an unhandled UnicodeDecodeError exception if the response has an invalid status code and the request contains binary payload (for FastHttpUser)

Expected behavior

UnicodeDecodeError shouldn’t be raised in case of non-utf-8 characters in the payload

Actual behavior

That is a stack trace:

[2020-06-23 09:14:15,612] 7ab43366d074/ERROR/locust.user.task: 'utf-8' codec can't decode byte 0x95 in position 41: invalid start byte
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 360, in urlopen
    self._verify_status(resp.status_code, url=req.url)
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 302, in _verify_status
    raise BadStatusCode(url, code=status_code)
geventhttpclient.useragent.BadStatusCode: URL http://url/endpoint: code=400

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 284, in run
    self.execute_next_task()
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 309, in execute_next_task
    self.execute_task(self._task_queue.pop(0))
  File "/usr/local/lib/python3.8/site-packages/locust/user/task.py", line 416, in execute_task
    task(self.user)
  File "/app/stress_tests/locustfile.py", line 95, in test_endpoint
    result = self.client.make_request()
  File "/app/tests/base_client.py", line 77, in make_request
    return self.client.request(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 227, in request
    response = self._send_request_safe_mode(method, url, payload=data, headers=headers, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/locust/contrib/fasthttp.py", line 159, in _send_request_safe_mode
    return self.client.urlopen(url, method=method, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 366, in urlopen
    e.http_log = self._conversation_str(req.url, resp, payload=req.payload)
  File "/usr/local/lib/python3.8/site-packages/geventhttpclient/useragent.py", line 424, in _conversation_str
    ret += payload.decode('utf-8') + '\n\n'
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x95 in position 41: invalid start byte

Steps to reproduce

Basically that’s a known issue: an attempt to decode the fully binary string fails with UnicodeDecodeError. It might be reproduced even without locust, just try to decode any binary string:

In [1]: import secrets                                                                                                                                                                                        
In [2]: binary_str = secrets.token_bytes(32)                                                                                                                                                                  
In [3]: binary_str.decode('utf-8')                                                                                                                                                                            
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-3-11a2204405d4> in <module>
----> 1 binary_str.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 2: invalid start byte

Environment

OS: Ubuntu 20.04
Python version: 3.8
Locust version: 1.0.2
Locust command line that you ran: locust

Issue Analytics

State:
Created 3 years ago
Comments:9 (4 by maintainers)

Top GitHub Comments

1reaction

cyberwcommented, Jul 1, 2020

Nice! Maybe the second change makes the first change unnecessary (I know I said I thought it was a bad idea to try and decode a broken response, but maybe there are cases where this is needed for debugging, so I kind of changed my mind 😃

Maybe instead of just silently ignoring the error (pass) do something like: ret += "UnicodeDecodeError"

1reaction

magupovcommented, Jun 25, 2020

Agree, especially because request and response are passed to the exception and can be handled directly if necessary I’ll make a PR soon

Top Results From Across the Web

How to fix: "UnicodeDecodeError: 'ascii' codec can't decode ...

In brief, Unicode strings are an entirely separate type of Python string that does not contain any encoding. They only hold Unicode point...

https://raw.githubusercontent.com/locustio/locust/...

... Unhandled `UnicodeDecodeError` exception if response with status 400 and request contains binary payload \(for FastHttpUser\) ...

Handling Exceptions Returned from the Web API

A 400 error, BadRequest, is used when you have validation errors from data posted back by the user. You pass a ModelStateDictionary object...

400 Bad Request Error: What It Is and How to Fix It

All HTTP response status codes that are in the 4xx category are considered client error responses. These types of messages contrast with errors ......

Understanding 400 Bad Request Exception

Any individual path segment (the portion of the URL that does not include protocol, server name, and query string, for example, http://a/b/c?d=e ...