Use a github token for checksums?

What is missing?

I think the role should support the use of a Github Token to avoid rate limiting.

Currently, I keep running into a 400 (Bad Request) errors when I deploy lots of nodes in parallel. That seems to rate-limiting from Github’s end. In my environment, I cd to about 20 customer setups, each of them may have multiple nodes. The part we are continuously deploying involves a basic monitoring setup on each node (including the node-exporter).

Why do we need it?

We run Ansible in a container, shared-nothing so to speak. The container gets invoked during CI (merge to main branch).

There is no caching between builds, it seems to work well. There are no side effects, except for this part where each run makes requests against Github (API) resources and seems to run into the rate-limit eventually.

I was digging around, it seems that both of these blocks happen every time (unless I download the binary myself):

Checksum gathering: https://github.com/cloudalchemy/ansible-node-exporter/blob/b9cb0eeccfaa104af5ba23e80c4104fef6814d93/tasks/preflight.yml#L98-L110

Downloading: https://github.com/cloudalchemy/ansible-node-exporter/blob/c6ffcfdef305c39273e7ab13bea69374c92eb171/tasks/install.yml#L20-L52

I would think I wouldn’t download anything — unless I really needed? Do you have any thoughts on changing that?

Environment

• Role version:

  0.21.3

• Ansible version information:

root@5c649b77871a:/ansible-all-the-things# ansible --version
ansible 2.9.8
  config file = /ansible-all-the-things/ansible.cfg
  configured module search path = ['/ansible-all-the-things/library']
  ansible python module location = /usr/local/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.8 (default, Jun 30 2020, 18:36:05) [GCC 8.3.0]

Anything else we need to know?:

[till]

I’ll dig around. Can you re-open this, or do I need to make another ticket?

[till]

Replacing lookup() with uri:

    - name: Get checksum list from github
      uri:
        url: "https://github.com/prometheus/node_exporter/releases/download/v{{ node_exporter_version }}/sha256sums.txt"
        method: GET
        return_content: true
      register: _checksum_result
      until: _checksum_result.status == 200
      retries: 5

    - name: Set _checksums
      set_fact:
        _checksums: "{{ _checksum_result.stdout_lines }}"
      run_once: true

Yields the 400 — but not sure where Authorization header is introduced?

<localhost> EXEC /bin/sh -c 'echo ~root && sleep 0'
--
2888 | <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir /root/.ansible/tmp/ansible-tmp-1595343177.2519362-233-56170830947126 && echo ansible-tmp-1595343177.2519362-233-56170830947126="` echo /root/.ansible/tmp/ansible-tmp-1595343177.2519362-233-56170830947126 `" ) && sleep 0'
2889 | Using module file /usr/local/lib/python3.7/site-packages/ansible/modules/net_tools/basics/uri.py
2890 | Pipelining is enabled.
2891 | <localhost> EXEC /bin/sh -c '/usr/local/bin/python && sleep 0'
2892 | <localhost> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1595343177.2519362-233-56170830947126/ > /dev/null 2>&1 && sleep 0'
2893 | FAILED - RETRYING: Get checksum list from github (1 retries left).Result was: {
2894 | "attempts": 5,
2895 | "changed": false,
2896 | "connection": "close",
2897 | "content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Basic **redacted**</ArgumentValue><RequestId>C27DCB3881334C01</RequestId><HostId>FIqOQQeYf2vmrxT1CEeevSvccLgS7KYSLaduFK8FjCsbkHoUDMtcQfE5RLzPE8HWKvjJIj9ozzA=</HostId></Error>",
2898 | "content_type": "application/xml",
2899 | "date": "Tue, 21 Jul 2020 14:52:58 GMT",
2900 | "elapsed": 0,
2901 | "invocation": {
2902 | "module_args": {
2903 | "attributes": null,
2904 | "backup": null,
2905 | "body": null,
2906 | "body_format": "raw",
2907 | "client_cert": null,
2908 | "client_key": null,
2909 | "content": null,
2910 | "creates": null,
2911 | "delimiter": null,
2912 | "dest": null,
2913 | "directory_mode": null,
2914 | "follow": false,
2915 | "follow_redirects": "safe",
2916 | "force": false,
2917 | "force_basic_auth": false,
2918 | "group": null,
2919 | "headers": {},
2920 | "http_agent": "ansible-httpget",
2921 | "method": "GET",
2922 | "mode": null,
2923 | "owner": null,
2924 | "regexp": null,
2925 | "remote_src": null,
2926 | "removes": null,
2927 | "return_content": true,
2928 | "selevel": null,
2929 | "serole": null,
2930 | "setype": null,
2931 | "seuser": null,
2932 | "src": null,
2933 | "status_code": [
2934 | 200
2935 | ],
2936 | "timeout": 30,
2937 | "unix_socket": null,
2938 | "unsafe_writes": null,
2939 | "url": "https://github.com/prometheus/node_exporter/releases/download/v1.0.1/sha256sums.txt",
2940 | "url_password": null,
2941 | "url_username": null,
2942 | "use_proxy": true,
2943 | "validate_certs": true
2944 | }
2945 | },
2946 | "msg": "Status code was 400 and not [200]: HTTP Error 400: Bad Request",
2947 | "redirected": false,
2948 | "retries": 6,
2949 | "server": "AmazonS3",
2950 | "status": 400,
2951 | "transfer_encoding": "chunked",
2952 | "url": "https://github.com/prometheus/node_exporter/releases/download/v1.0.1/sha256sums.txt",
2953 | "x_amz_id_2": "FIqOQQeYf2vmrxT1CEeevSvccLgS7KYSLaduFK8FjCsbkHoUDMtcQfE5RLzPE8HWKvjJIj9ozzA=",
2954 | "x_amz_request_id": "C27DCB3881334C01"
2955 | }