urllib3 should use URLFetch to create HTTPS connections on AppEngine

AppEngine is listed as a supported platform for urllib3.

However, it appears to use the Sockets API in order to make HTTPS connections, as a result of it using socket from the standard library to make the connection.

As a result, this will exhaust quota much faster than if it wrapped the URLFetch service, and is only available to billed (non-free) applications.

Attempting to use it on a free application results in an error.

Additionally, the Sockets API is beta service, and it is impossible to connect to Google’s HTTPS services using the Sockets API.

Instead, urllib3 should use URLFetch by default in order to send HTTPS requests on AppEngine, and use a similar fallback mechanism to httplib in order to fall back to using socket.

[micolous]

Note: I don’t presently work on Google Cloud.

URLFetch is not required for the Python 3.7 Standard Runtime (which uses a different sandboxing system) or any of the Flexible Environment runtimes (which use Compute Engine).

The weirdness is because under the hood, App Engine (on Python 2.7 Standard Runtime) is sending your HTTP(S) requests as an RPC to a service that makes HTTP(S) requests or your behalf. This service is also used for stuff like Google’s web crawler, which doesn’t care too much about implementing things like TLS client certificates or certificate pinning.

I’ve seen plenty of App Engine users shoot themselves in the foot because urllib3 (or requests) used Sockets API without them realising and then blow through the (at the time) very small quota, or otherwise try to access Google API services and not figure out why it doesn’t work. The overwhelming majority of the time, URLFetch provided all the features needed (with the possible exception of being harder to use requests).

But, it’s now 2019, so you should absolutely be using one of the Python 3.7 runtimes, and the clock is literally ticking to migrate. That doesn’t represent the status of Python 2.7 support on App Engine, but you can expect more Python libraries to stop working from 2020, as they start ripping out compatibility code. In some cases, that decision may be made by another library that a library you depend on uses (so you may have a surprise).

With my reporter hat on, I’m going to close the original issue as I’m satisfied that urllib3 now has a way to handle this and I don’t think there is a lot of value in doing more than this given that Python 3.7 Standard Runtime doesn’t need it, and Python 2.7 should be left to sail quietly into the night. 😃

Thank you @theacodes and @shazow for getting in a work-around!

[davidparsson]

If this were to be implemented, I think it still should be possible to use the current approach since AppEngine’s URLFetch API does not support all use-cases currently supported. One example is client-side certificates.