403 error while using github API

Summary

TL;DR: requests raises a 403 while requesting an authenticated Github API route, which otherwise succeeds while using curl/another python library like httpx

Was initially discovered in the ‘ghexport’ project; I did a reasonable amount of debugging and created this repo before submitting this issue to PyGithub, but thats a lot to look through, just leaving it here as context.

It’s been hard to reproduce, the creator of ghexport (where this was initially discovered) didn’t have the same issue, so I’m unsure of the exact reason

Expected Result

requests succeeds for the authenticated request

Actual Result

Request fails, with:

{'message': 'Must have push access to repository', 'documentation_url': 'https://docs.github.com/rest/reference/repos#get-repository-clones'}
failed
Traceback (most recent call last):
  File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 47, in <module>
    main()
  File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 44, in main
    make_request(requests.get, url, headers)
  File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 31, in make_request
    resp.raise_for_status()
  File "/home/sean/.local/lib/python3.9/site-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.github.com/repos/seanbreckenridge/albums/traffic/clones

Reproduction Steps

Apologies if this is a bit too specific, but otherwise requests works great on my system and I can’t find any other way to reproduce this – Is a bit long as it requires an auth token

Go here and create a token with scopes like:

I’ve compared this to httpx, where it doesn’t fail:

#!/usr/bin/env python3

from typing import Callable, Any

import requests
import httpx


# extract status/status_code from the requests/httpx item
def extract_status(obj: Any) -> int:
    if hasattr(obj, "status"):
        return obj.status
    if hasattr(obj, "status_code"):
        return obj.status_code
    raise TypeError("unsupported request object")


def make_request(using_verb: Callable[..., Any], url: str, headers: Any) -> None:

    print("using", using_verb.__module__, using_verb.__qualname__, url)

    resp = using_verb(url, headers=headers)
    status = extract_status(resp)

    print(str(resp.json()))

    if status == 200:
        print("succeeded")
    else:
        print("failed")
        resp.raise_for_status()


def main():
    # see https://github.com/seanbreckenridge/pygithub_requests_error for token scopes
    auth_token = "put your auth token here"

    headers = {
        "Authorization": "token {}".format(auth_token),
        "User-Agent": "requests_error",
        "Accept": "application/vnd.github.v3+json",
    }

    # replace this with a URL you have access to
    url = "https://api.github.com/repos/seanbreckenridge/albums/traffic/clones"

    make_request(httpx.get, url, headers)
    make_request(requests.get, url, headers)


if __name__ == "__main__":
    main()

That outputs:

using httpx get https://api.github.com/repos/seanbreckenridge/albums/traffic/clones
{'count': 15, 'uniques': 10, 'clones': [{'timestamp': '2021-04-12T00:00:00Z', 'count': 1, 'uniques': 1}, {'timestamp': '2021-04-14T00:00:00Z', 'count': 1, 'uniques': 1}, {'timestamp': '2021-04-17T00:00:00Z', 'count': 1, 'uniques': 1}, {'timestamp': '2021-04-18T00:00:00Z', 'count': 2, 'uniques': 2}, {'timestamp': '2021-04-23T00:00:00Z', 'count': 9, 'uniques': 5}, {'timestamp': '2021-04-25T00:00:00Z', 'count': 1, 'uniques': 1}]}
succeeded
using requests.api get https://api.github.com/repos/seanbreckenridge/albums/traffic/clones
{'message': 'Must have push access to repository', 'documentation_url': 'https://docs.github.com/rest/reference/repos#get-repository-clones'}
failed
Traceback (most recent call last):
  File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 48, in <module>
    main()
  File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 45, in main
    make_request(requests.get, url, headers)
  File "/home/sean/Repos/pygithub_requests_error/minimal.py", line 32, in make_request
    resp.raise_for_status()
  File "/home/sean/.local/lib/python3.9/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.github.com/repos/seanbreckenridge/albums/traffic/clones

Another thing that may be useful as context is the pdb trace I did here, which was me stepping into where the request was made in PyGithub, and making all the requests manually using the computed url/headers. Fails when I use requests.get but httpx.get works fine:

> /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(484)__requestEncode()
-> self.NEW_DEBUG_FRAME(requestHeaders)
(Pdb) n
> /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(486)__requestEncode()
-> status, responseHeaders, output = self.__requestRaw(
(Pdb) w
  /home/sean/Repos/ghexport/export.py(109)<module>()
-> main()
  /home/sean/Repos/ghexport/export.py(84)main()
-> j = get_json(**params)
  /home/sean/Repos/ghexport/export.py(74)get_json()
-> return Exporter(**params).export_json()
  /home/sean/Repos/ghexport/export.py(60)export_json()
-> repo._requester.requestJsonAndCheck('GET', repo.url + '/traffic/' + f)
  /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(318)requestJsonAndCheck()
-> *self.requestJson(
  /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(410)requestJson()
-> return self.__requestEncode(cnx, verb, url, parameters, headers, input, encode)
> /home/sean/.local/lib/python3.8/site-packages/github/Requester.py(486)__requestEncode()
-> status, responseHeaders, output = self.__requestRaw(
(Pdb) url
'/repos/seanbreckenridge/advent-of-code-2019/traffic/views'
(Pdb) requestHeaders
{'Authorization': 'token <MY TOKEN HERE>', 'User-Agent': 'PyGithub/Python'}
(Pdb) import requests
(Pdb) requests.get("https://api.github.com" + url, headers=requestHeaders).json()
{'message': 'Must have push access to repository', 'documentation_url': 'https://docs.github.com/rest/reference/repos#get-page-views'}
(Pdb) httpx.get("https://api.github.com" + url, headers=requestHeaders).json()
{'count': 0, 'uniques': 0, 'views': []}
(Pdb) httpx succeeded??

System Information

$ python -m requests.help

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": "3.4.7"
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.3"
  },
  "platform": {
    "release": "5.11.16-arch1-1",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "101010bf",
    "version": "20.0.1"
  },
  "requests": {
    "version": "2.25.1"
  },
  "system_ssl": {
    "version": "101010bf"
  },
  "urllib3": {
    "version": "1.25.9"
  },
  "using_pyopenssl": true
}

$ pip freeze | grep -E 'requests|httpx'
httpx==0.16.1
requests==2.25.1

[seanbreckenridge]

Will see if I can try and find similarities/switch to python2.7 and see if that makes any difference when I debug this further, probably this weekend

[sigmavirus24]

You could look at what’s present in your response.request object. I printed the headers as there shouldn’t be anything else differing. You may also look into whether you have any proxies or other intermediaries that Requests might be detecting and using for your traffic - which httpx might not be using.