boto3 integration broken

Hey, I’ve recently started seeing boto3 related failures in my unit tests. The recordings had been working previously, but it seems that something has changed in boto, and the vcrpy support is no longer working.

------------------------------------------------------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------------------------------------------------------
/Users/analogj/repos/lexicon/venv/lib/python2.7/site-packages/urllib3/connectionpool.py:857: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)

[1oglop1]

@jeshan I have some debug information for the issue The biggest obstacle for the issue is that vcrpy does not have much logging

1. botocore does not use vendored requests for all calls it uses botocore.awsrequest which uses urllib3 https://github.com/boto/botocore/blob/develop/botocore/awsrequest.py#L254

2. Any call from botocore or botocore.vendored.requests does not invoke vcr.request

3. vcr.request.Request is never called when using boto3

@kevin1024 I’d need more info how does the lib work internally because I could not figure it out what is called when.

It might be something with vcr.patch which imports botocore.verndored.requests ~3 times https://github.com/kevin1024/vcrpy/blob/master/vcr/patch.py#L410 because botocore.awsrequest has bit different implementation

and then _urllib3_patchers https://github.com/kevin1024/vcrpy/blob/master/vcr/patch.py#L306 is looking for HTTPSConnection but botocore has prefix AWS -> AWSHTTPSConnection

after couple modification I was able to get vcr.request.Request invoked and request recorded.

So in the end it was enough to patch this part https://github.com/kevin1024/vcrpy/blob/master/vcr/patch.py#L315 as:

        if cpool.__name__ == 'botocore.awsrequest':
            log.debug(f"mock {mock_triples}")
            mock_triples = (
                (cpool.AWSHTTPConnectionPool, 'ConnectionCls', stubs.VCRRequestsHTTPConnection),
                (cpool.AWSHTTPSConnectionPool, 'ConnectionCls', stubs.VCRRequestsHTTPSConnection),
            )

It is not very elegant and it’s tested now only on my dev env and boto 1 call. I will do couple more tests and then run the CI to see if tests pass.

[svdgraaf]

Same here! Thanks for everyone involved! 👏