[Bug] ValueError with text from a reference

What a great library! I’m parsing PDF -> text files from the arXiv and a common motif that crashes the program is shown in this minimal example below:

from urlextract import URLExtract
extractor = URLExtract()

text = "et.al.[10]"
extractor.find_urls(text)

with the traceback

Traceback (most recent call last):
  File "failure.py", line 5, in <module>
    extractor.find_urls(text)
  File "/home/hoppeta/.pyenv/versions/3.6.0/lib/python3.6/site-packages/urlextract/urlextract_core.py", line 756, in find_urls
    return list(urls)
  File "/home/hoppeta/.pyenv/versions/3.6.0/lib/python3.6/site-packages/urlextract/urlextract_core.py", line 739, in gen_urls
    tmp_url = self._complete_url(text, offset + tld_pos, tld)
  File "/home/hoppeta/.pyenv/versions/3.6.0/lib/python3.6/site-packages/urlextract/urlextract_core.py", line 560, in _complete_url
    if not self._is_domain_valid(complete_url, tld):
  File "/home/hoppeta/.pyenv/versions/3.6.0/lib/python3.6/site-packages/urlextract/urlextract_core.py", line 632, in _is_domain_valid
    host = url_parts.gethost()
  File "/home/hoppeta/.pyenv/versions/3.6.0/lib/python3.6/site-packages/uritools/split.py", line 157, in gethost
    raise ValueError('Invalid host %r' % host)
ValueError: Invalid host 'et.al.[10]'

[lipoja]

Thank you for using this library. And bigger THANKS for reporting this issue! I will check it and try to fix it soon.

[thoppe]

Awesome! I don’t need it to parse as a URL, just have it not crash. It’s actually a marker for a reference in an academic paper so missing it is the correct action.

On Sun, Mar 10, 2019, 10:21 AM Jan Lipovský [e-mail] wrote:

Issue should be fixed, I’ve added both text to tests files.

@thoppe https://github.com/thoppe Right now urlextract does not return text “et.al.[10]” as URL. If you are parsing something specific you might use your own settings that fit your needs. For example by setting the stop characters using set_stop_chars_right and set_stop_chars_left.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/lipoja/URLExtract/issues/30#issuecomment-471310568, or mute the thread https://github.com/notifications/unsubscribe-auth/AClOoqVBCzPz148TSIJ2YpzdbZXH0S4sks5vVRTvgaJpZM4bDjlY .