question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

"Too many open files" error, regardless of size parameter

See original GitHub issue

I am using grequests to test the performance of my service. Both the service and the machine from which I start the requests are inside Amazon EC2. It seems that I cannot start more requests than ulimit -n, even though I explicitly specify the size parameter for map() / imap()

My setup is roughly like this:

headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
params = [json.dumps({'url': urls[i]}) for i in urls]
reqs = [grequests.post(SERVICE_URL, data=params[i % len(params)], headers=headers)
            for i in xrange(no_requests)]
res = grequests.imap(reqs, size=200)
for r in res:
    ...
    r.close()

Doesn’t the size parameter here tell grequests that it should issue no_requests, 200 at a time?

Issue Analytics

  • State:closed
  • Created 9 years ago
  • Comments:8 (1 by maintainers)

github_iconTop GitHub Comments

4reactions
cyberstipcommented, Apr 13, 2015

For posterity, just adding a session to the request seems to fix the issue (as @rtdean mentioned):

session = requests.Session() reqs = (grequests.get(url, session=session) for url in urls) results = grequests.map(reqs, size=50)

3reactions
amitsquarecommented, Feb 13, 2017

Please do mention using Session Object in README .As till you complete system shutdown , user won’t be searching the issue & end up here on this issue thread .

Read more comments on GitHub >

github_iconTop Results From Across the Web

How to Fix the 'Too Many Open Files' Error in Linux?
It means that a process has opened too many files (file descriptors) and cannot open new ones. On Linux, the “max open file...
Read more >
Fixing the “Too many open files” Error in Linux - Baeldung
It opens the file passed as an argument via the open() system call and gets assigned a file descriptor for it. Then, it...
Read more >
php - Fatal Error - Too many open files - Stack Overflow
In my case, this occurred no matter how high I set ulimit -n (I even tried 1,000,000). The problem turned-out to be bad...
Read more >
Tomcat Error: Too many open files - Support - Micro Focus
This message is stating the problem for us, and that is that Tomcat is experiencing a jni error: Too many open files.
Read more >
Linux Increase The Maximum Number Of Open Files ... - nixCraft
Icreasing the file handles is a good tip, but 5000 is very low these days. 200000 is more realistic for any modern system....
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found