readlines on files from hdfs using webhdfs hangs

Problem description

readlines on files from hdfs using webhdfs hangs

Steps/code to reproduce the problem

with open("webhdfs://namenode/myfile.txt", "r") as f:
    lines = f.readlines()

The issue seems to be in the following line https://github.com/RaRe-Technologies/smart_open/blob/develop/smart_open/webhdfs.py#L147

Sometimes self._response.raw.read(io.DEFAULT_BUFFER_SIZE) returns 0 bytes and StopIteration is never thrown resulting in the call hanging for ever. I fixed it locally with the following change which seems to work.

while len(self._buf) < size:
    raw_data = self._response.raw.read(io.DEFAULT_BUFFER_SIZE)
    if len(raw_data) == 0:
        break

    self._buf += raw_data

Versions

Please provide the output of:

import platform, sys, smart_open
print(platform.platform())
print("Python", sys.version)
print("smart_open", smart_open.__version__)

Linux-5.8.0-44-generic-x86_64-with-debian-bullseye-sid Python 3.7.3 (default, Sep 3 2020, 16:41:28) [GCC 9.3.0] smart_open 4.2.0

Checklist

Before you create the issue, please make sure you have:

• Described the problem clearly
• Provided a minimal reproducible example, including any required data
• Provided the version numbers of the relevant software

[traboukos]

OK @mpenkov @piskvorky makes sense. Will send a pr.

[mpenkov]

Yes, you’re right, string concatenation is awful.

The approach you suggest is definitely better. Another common way to to it is to write into an io.BytesIO buffer.

@traboukos Please go ahead with the PR, taking the above into account.