check dns concurrently to speed up lookup

Is there any room to check dns concurrently as it might be a time consuming task? I’ve found some hacky ways to do that, but maybe it could be a flag on find_urls. I’m aware this would increase the complexity of this function a lot.

[lipoja]

OK, could you also think about how to do it so we have the code not tangled. For example: If we can separate all DNS checking parts to its own class/module? This is just and idea I did not look to the code how tricky it will be. Also it would be nice to give user a method that he/she can call when they wants. For example when they use the gen_urls() or they do some post process filtering of URLs.

[nicolasassi]

Great! I was thinking of adding concurrency in the gen_urls() and give user option from the functions above it. Achieve concurrency in the way you metioned would be a lot easier. Gonna work on that.