RuntimeError: can't start new thread

Everything was working fine on the previous build, now that I have upgraded I am receiving this error - Traceback (most recent call last): File "sherlock.py", line 649, in <module> main() File "sherlock.py", line 613, in main timeout=args.timeout) File "sherlock.py", line 255, in sherlock timeout=timeout File "/usr/local/lib/python3.7/dist-packages/requests_futures/sessions.py", line 138, in head return super(FuturesSession, self).head(url, **kwargs) File "/usr/local/lib/python3.7/dist-packages/requests/sessions.py", line 568, in head return self.request('HEAD', url, **kwargs) File "sherlock.py", line 59, in request return super(ElapsedFuturesSession, self).request(method, url, hooks=hooks, *args, **kwargs) File "/usr/local/lib/python3.7/dist-packages/requests_futures/sessions.py", line 105, in request return self.executor.submit(func, *args, **kwargs) File "/usr/lib/python3.7/concurrent/futures/thread.py", line 160, in submit self._adjust_thread_count() File "/usr/lib/python3.7/concurrent/futures/thread.py", line 181, in _adjust_thread_count t.start() File "/usr/lib/python3.7/threading.py", line 847, in start _start_new_thread(self._bootstrap, ()) RuntimeError: can't start new thread

[0xAL3INO]

Can you try it now with the latest content on master (version 0.10.0)? In #471, I placed a limit on the number of threads that can be allocated. And, it did not seem to make any difference on performance.

It works great, thanks! I’m surprised it has no effect on performance - my pi can run a full scan in 45 seconds now, though I hadn’t timed it previously.

[hoadlck]

It is a Christmas miracle! 😃

Well…actually, I guess it is not that surprising. Trying to spawn so many threads ends up using too many resources. Which can end up causing a given thread to be even slower than normal. On my Windows 10 PC, there were some sites that did not detect usernames under the old version: Sherlock found more sites with a harder limit of threads.

Unfortunately, the tests invoke Sherlock with a limited number of sites multiple times. So, an individual test never spawned so many threads at one time…they were spread out over multiple invocations to cover the entire list. While it was fine for functionality, it missed this case.