None in the response when the URLs list is over 100

Hi,

I am seeing a strange behavior in the response when the URLs list is over 100 or more items. Most of the responses I get from the grequests are of None.

I’ve seen the similar issue that was raised by another member https://www.reddit.com/r/Python/comments/4ifmat/using_grequests_after_a_certain_amount_of/

Here are the few things I’ve tried:

• Used one proxy per URL but still None response for some URLs.
• Iterated the None URLs with the requests library and I am getting the 200 response code.
• Created a chunk of URLs, 20 URLs per map requests and it brought the 200 response for 16 urls and 4 URLs None Response.

Is there a bug in the library or something I need to fix on my end?

Below is a code used:

HEADERS_DEFAULT = {
    "Accept-Language": "en-US,en;q=0.5",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0",
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Connection": "keep-alive"
}

cleaned_products_list = [a list of URLs I used here.]
products_rs = set()
 for u in cleaned_products_list:
    products_rs.add(grequests.get(
                             u, proxies=random.choice(PROXIES_LIST), 
                             headers=HEADERS_DEFAULT))

response_list = grequests.map(products_rs)

[willhaslam]

@azhard4int I’ve managed to solve my None responses by passing the errors to an error handler. grequests returned None if the request errors, which in my case is SSL errors.

e.g.

def exception_handler(request, exception):
    print(exception)

grequests.map(unsent_requests, exception_handler=exception_handler)

[spyoungtech]

@Jaronloch2 exception_handler is an argument to map not get.

>>> req = grequests.get('https://httpbin.org', exception_handler='foo')
>>> req.send()
>>> req.exception
TypeError("request() got an unexpected keyword argument 'exception_handler'",)

This would cause the behavior you describe. Please look more carefully at the previous comments.

That said, I don’t think this is a direct issue with grequests. Have tested extensively with sending very large numbers of requests. You’re likely running into a different underlying issue.

If this is still an issue, at a minimum, please provide the underlying exception using the method described by @willhaslam