"Too many open files" in debug mode

When running in the debug mode, the call to df -h /tmp will be run from https://github.com/airbnb/streamalert/blob/cf12209aa071cd4b2f85bfc5b77446bbb696d451/stream_alert/rule_processor/payload.py#L207

This may result in an exception being thrown:

[Errno 24] Too many open files: OSError
Traceback (most recent call last):
File "/var/task/stream_alert/rule_processor/main.py", line 39, in handler
StreamAlert(context).run(event)
File "/var/task/stream_alert/rule_processor/handler.py", line 148, in run
self._process_alerts(payload)
File "/var/task/stream_alert/rule_processor/handler.py", line 356, in _process_alerts
for record in payload.pre_parse():
File "/var/task/stream_alert/rule_processor/payload.py", line 155, in pre_parse
s3_file = self._get_object()
File "/var/task/stream_alert/rule_processor/payload.py", line 245, in _get_object
return self._download_object(region, bucket, key)
File "/var/task/stream_alert/rule_processor/payload.py", line 206, in _download_object
LOGGER.debug(os.popen('df -h /tmp | tail -1').read().strip())
OSError: [Errno 24] Too many open files

This will only present itself if debug output is on, and this may be masking another issue for myself, but perhaps this case should checked somehow (where you may have too many files in /tmp?). I also don’t know why my /tmp would have too many files in it, or if I should clear that out somehow. Based on lambda limits, the /tmp directory should be limited to 512MB anyway ( https://docs.aws.amazon.com/lambda/latest/dg/limits.html ), and the max file descriptors should be set to 1024.

[0xdabbad00]

I’m digging into this to find the root cause. I’ll update this as I figure things out, but I suspect the issue might be isolated to my environment.

[ryandeivert]

we’ve seen this internally an can confirm this is not fixed