Searching stops working after a few hours (give or take) of use

Hello!

I have been using CIF (primarily v2) for the past couple years (even have some commits into the CIF chrome extension). On this new CIFv3 installation, I am running into a weird problem I can’t quite fix.

Expected behavior and actual behavior.

Expected behavior is when I search CIF for an indicator (either through the CLI or API) I get results back.

Actual behavior is after searching for a little while, the search will just hang. cif -p will not work as well.

Steps to reporduce the problem

Use CIF for a period of time, usually a couple hours but sometimes as short as a few minutes.

Relevant logs as a result of the actual behavior

May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: [2018-05-29 14:17:02,845] ERROR in app: Exception on /ping [GET] May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: Traceback (most recent call last): May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1982, in wsgi_app May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: response = self.full_dispatch_request() May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1614, in full_dispatch_request May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: rv = self.handle_user_exception(e) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask_cors/extension.py”, line 161, in wrapped_function May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: return cors_after_request(app.make_response(f(*args, **kwargs))) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1517, in handle_user_exception May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: reraise(exc_type, exc_value, tb) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1612, in full_dispatch_request May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: rv = self.dispatch_request() May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask/app.py”, line 1598, in dispatch_request May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: return self.view_functionsrule.endpoint May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask/views.py”, line 84, in view May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: return self.dispatch_request(*args, **kwargs) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/flask/views.py”, line 149, in dispatch_request May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: return meth(*args, **kwargs) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/cif-3.0.0rc2-py2.7.egg/cif/httpd/views/ping.py”, line 24, in get May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: r = Client(remote, pull_token()).ping(write=write) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py”, line 110, in ping May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: return self._send(Msg.PING_WRITE) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py”, line 106, in _send May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: return self._recv(decode=decode) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/cifsdk/client/zeromq.py”, line 53, in _recv May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: mtype, data = Msg().recv(self.socket) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/cifsdk/msg.py”, line 72, in recv May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: m = s.recv_multipart() May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “/usr/local/lib/python2.7/dist-packages/zmq/sugar/socket.py”, line 395, in recv_multipart May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: parts = [self.recv(flags, copy=copy, track=track)] May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “zmq/backend/cython/socket.pyx”, line 693, in zmq.backend.cython.socket.Socket.recv May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “zmq/backend/cython/socket.pyx”, line 727, in zmq.backend.cython.socket.Socket.recv May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “zmq/backend/cython/socket.pyx”, line 150, in zmq.backend.cython.socket._recv_copy May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “zmq/backend/cython/socket.pyx”, line 145, in zmq.backend.cython.socket._recv_copy May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: File “zmq/backend/cython/checkrc.pxd”, line 19, in zmq.backend.cython.checkrc._check_rc May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: raise Again(errno) May 29 14:17:02 ip-10-225-1-233 gunicorn[1244]: Again: Resource temporarily unavailable

Specifications like the version of the project, operating system, or hardware.

CIFv3 3.0.0rc2. OS Ubuntu Server 16.04 AWS EC2 m5.2xlarge

Other Notes

When this happens, I am still able to access parts of the API like /, /help, and /help/confidence. Additionally, CIFv3 was installed using the easy button. Restarting the server fixes the problem for a short period of time.

Let me know if there are any other logs I can provide you!

[JustinAzoff]

I think I see what is going on. The logs show 2 interleaved errors. The first error was caught elsewhere in cif and wasn’t harming anything.

The 2nd error happening at the same time was from the hunter, and that one was a problem.

With CIF_HUNTER_THREADS=0 everything is fine, but the hunter isn’t catching the InvalidIndicator exception though, so if CIF_HUNTER_THREADS is 4, after 4 queries like that, all the hunters are dead and the router blocks.

I just sent over https://github.com/csirtgadgets/bearded-avenger/pull/397 which should fix it.

[Rickerd0613]

I applied the changes yesterday and so far everything appears to work! I’m not sure I’m ready to say it is 100% solved as it may still break in the next few days, but I have high hopes for the fix this time.

Thanks for taking the time to track this bug down. I really appreciate it.