"Too many open files" error, regardless of size parameter
See original GitHub issueI am using grequests to test the performance of my service. Both the service and the machine from which I start the requests are inside Amazon EC2. It seems that I cannot start more requests than ulimit -n
, even though I explicitly specify the size parameter for map() / imap()
My setup is roughly like this:
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
params = [json.dumps({'url': urls[i]}) for i in urls]
reqs = [grequests.post(SERVICE_URL, data=params[i % len(params)], headers=headers)
for i in xrange(no_requests)]
res = grequests.imap(reqs, size=200)
for r in res:
...
r.close()
Doesn’t the size
parameter here tell grequests
that it should issue no_requests, 200 at a time?
Issue Analytics
- State:
- Created 9 years ago
- Comments:8 (1 by maintainers)
Top Results From Across the Web
How to Fix the 'Too Many Open Files' Error in Linux?
It means that a process has opened too many files (file descriptors) and cannot open new ones. On Linux, the “max open file...
Read more >Fixing the “Too many open files” Error in Linux - Baeldung
It opens the file passed as an argument via the open() system call and gets assigned a file descriptor for it. Then, it...
Read more >php - Fatal Error - Too many open files - Stack Overflow
In my case, this occurred no matter how high I set ulimit -n (I even tried 1,000,000). The problem turned-out to be bad...
Read more >Tomcat Error: Too many open files - Support - Micro Focus
This message is stating the problem for us, and that is that Tomcat is experiencing a jni error: Too many open files.
Read more >Linux Increase The Maximum Number Of Open Files ... - nixCraft
Icreasing the file handles is a good tip, but 5000 is very low these days. 200000 is more realistic for any modern system....
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
For posterity, just adding a session to the request seems to fix the issue (as @rtdean mentioned):
session = requests.Session() reqs = (grequests.get(url, session=session) for url in urls) results = grequests.map(reqs, size=50)
Please do mention using Session Object in README .As till you complete system shutdown , user won’t be searching the issue & end up here on this issue thread .