obtain a dedicated API key for testing NREL psm3

Is your feature request related to a problem? Please describe. came up in #866 see this comment but related to most PR and testing in general - due to NREL PSM3 throttle limits for the DEMO key the CI’s frequently fail

Describe the solution you’d like obtain a dedicated API key from NREL developer network and upload it to Azure pipelines as a secret variable, ditto for Travis, then for testing, get this from os.environ[<PVLIB TEST NREL API KEY>] and use it instead of the DEMO key which has severe throttle limits and is shared by everyone

Describe alternatives you’ve considered just live with the test failures, or skip theses tests from the CI’s

Additional context Add any other context or screenshots about the feature request here.

[kanderso-nrel]

Now that you mention it, this pattern could well be the issue:

    with pytest.raises(HTTPError):
        # HTTP 403 forbidden because api_key is rejected
        psm3.get_psm3(LATITUDE, LONGITUDE, api_key='BAD', email=PVLIB_EMAIL)
    with pytest.raises(HTTPError):
        # coordinates were not found in the NSRDB
        psm3.get_psm3(51, -5, DEMO_KEY, PVLIB_EMAIL)
    with pytest.raises(HTTPError):
        # names is not one of the available options
        psm3.get_psm3(LATITUDE, LONGITUDE, DEMO_KEY, PVLIB_EMAIL, names='bad')

Those could easily send more than one request per second. Maybe we should wrap the get_psm3 function calls with another function that sleeps before every call? For instance

def get_psm3(*args, **kwargs):
    time.sleep(2)
    return psm3.get_psm3(*args, **kwargs)

If that doesn’t work, here are two ideas, neither of which is nearly as good as understanding the API rate limits:

---

The API response includes two relevant headers indicating rate limit usage:

In [124]: dict(response.headers)
Out[124]: 
 ...
 'X-RateLimit-Limit': '2000',
 'X-RateLimit-Remaining': '1999',
 ...

Maybe useful for e.g. monkey-patching the network calls in test_psm3.py to print out those header values:

import requests
import sys
original_get = requests.get

def get_and_log(*args, **kwargs):
    response = original_get(*args, **kwargs)
    headers = dict(response.headers)
    for key in ['X-RateLimit-Limit', 'X-RateLimit-Remaining']:
        print(key, headers.get(key, None))
    return response

psm3.requests.get = get_and_log

Click to expand pytest output

(base) C:\Users\KANDERSO\projects\pvlib-python>pytest -s pvlib\test\test_psm3.py
================================================= test session starts =================================================
platform win32 -- Python 3.7.3, pytest-5.0.1, py-1.8.0, pluggy-0.12.0
rootdir: C:\Users\KANDERSO\projects\pvlib-python
plugins: celery-4.3.0, dash-1.6.0, arraydiff-0.3, cov-2.8.1, doctestplus-0.3.0, openfiles-0.3.2, remotedata-0.3.1
collected 4 items

pvlib\test\test_psm3.py X-RateLimit-Limit 40
X-RateLimit-Remaining 22
X-RateLimit-Limit None
X-RateLimit-Remaining None
X-RateLimit-Limit 40
X-RateLimit-Remaining 21
X-RateLimit-Limit 40
X-RateLimit-Remaining 20
.X-RateLimit-Limit 40
X-RateLimit-Remaining 19
...

---

I’ve used a retry approach like this before when I wanted to bulk download data from the API and kept getting seemingly spurious timeouts:

def retry(n=10, delay=5):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for i in range(n):
                try:
                    result = func(*args, **kwargs)
                    break
                except HTTPError:
                    time.sleep(delay)
            else:
                raise Exception('Max retries reached')
            return result
        return wrapper
    return decorator

@retry(n=3, delay=10)
def fetch(lat, lon, year):
    headers, df = pvlib.iotools.get_psm3(lat, lon, key, email, year, 30)
    return df

[CameronTStark]

Thank you for the confirmation that I’m not crazy @kanderso-nrel. I took a few iterations to be sure of what I was seeing.

To make the tests past consistently the only solutions that I can propose are to put 5 sec timers in front of every API call and/or cut down the number of API calls to the bare minimum. Both aren’t reasons to be excited to be sure. I’m not sure even NREL whitelisting Azure IPs for a higher throughput would be a workable fix. Do you have any other suggestions?

In looking at test_psm3.py some of the tests made me wonder if we were testing pvlib or the API server. Are there any API calls that could be removed to help the situation?