gRPC client + botocore + datadog exporter instrumentation error

This one is a bit complex - when using the gRPC client instrumentation to make a call to an instrumented gRPC service that then makes a call via boto to the DynamoDB API, the combination of instrumentation leads to this error:

2020-12-17 14:19:39,666 DEBUG    [botocore.httpsession:298] Exception received when sending urllib3 HTTP request
Traceback (most recent call last):
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/botocore/httpsession.py", line 254, in send
    urllib_response = conn.urlopen(
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connection.py", line 234, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/lib64/python3.8/http/client.py", line 1255, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/botocore/awsrequest.py", line 91, in _send_request
    rval = super(AWSConnection, self)._send_request(
  File "/usr/lib64/python3.8/http/client.py", line 1296, in _send_request
    self.putheader(hdr, value)
  File "/home/michael/work/myapp/.venv/lib/python3.8/site-packages/urllib3/connection.py", line 219, in putheader
    _HTTPConnection.putheader(self, header, *values)
  File "/usr/lib64/python3.8/http/client.py", line 1232, in putheader
    if _is_illegal_header_value(values[i]):
TypeError: expected string or bytes-like object

Without either client instrumentation, this call works perfectly.

Steps to reproduce As described above - I don’t have a simplified sample, but I’ll try to produce one.

What is the expected behavior? No exception 😃

What is the actual behavior? The above exception.

Additional context

I’m definitely working on this, but any suggestions would certainly be welcome, I’m not quite sure what the invalid header is or where it might get set.

[toumorokoshi]

I’m running the two services locally, so I can hack whatever I need to add debugging. I did try a bunch of printing, but couldn’t get anything useful before the exception was raised.

Have you tried adding a pdb statement to this snippet, within a try except?

  File "/usr/lib64/python3.8/http/client.py", line 1232, in putheader
    if _is_illegal_header_value(values[i])

try: 
   if is_illegal_header_value(...)
except:
   import pdb;
   pdb.set_trace()

You should be able to extract all the information you need from that pdb session. prints work too, just a bit more time consuming.

[alertedsnake]

That looks a lot like the PR for this which I closed because it didn’t fix the problem at the source, it just prevented it from propagating:

https://github.com/open-telemetry/opentelemetry-python-contrib/pull/272

If you guys think this is the best way to fix it, I guess that works for me.