Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Different behaviour with `open`

See original GitHub issue

Problem: smart_open==1.6.0 doesn’t work correctly with numpy.frombuffer function (numpy < 1.13.0, preferable to test with numpy==1.11.3, python2.7).

Example: env:

numpy==1.11.3
gensim==3.4.0
smart_open==1.6.0

from gensim.models import FastText
from gensim.test.utils import datapath

m = FastText.load_fasttext_format(datapath("lee_fasttext"))

Trace

IOError                                   Traceback (most recent call last)
<ipython-input-1-afdc56d6c4b5> in <module>()
      2 from gensim.test.utils import datapath
      3 
----> 4 m = FastText.load_fasttext_format(datapath("lee_fasttext"))

/home/ivan/release/gensim/gensim/models/fasttext.py in load_fasttext_format(cls, model_file, encoding)
    697             model_file += '.bin'
    698         model.file_name = model_file
--> 699         model.load_binary_data(encoding=encoding)
    700         return model
    701 

/home/ivan/release/gensim/gensim/models/fasttext.py in load_binary_data(self, encoding)
    712             self._load_model_params(f)
    713             self._load_dict(f, encoding=encoding)
--> 714             self._load_vectors(f)
    715 
    716     def _load_model_params(self, file_handle):

/home/ivan/release/gensim/gensim/models/fasttext.py in _load_vectors(self, file_handle)
    819 
    820         self.num_original_vectors = num_vectors
--> 821         self.wv.vectors_ngrams = np.fromfile(file_handle, dtype=dtype, count=num_vectors * dim)
    822         self.wv.vectors_ngrams = self.wv.vectors_ngrams.reshape((num_vectors, dim))
    823         assert self.wv.vectors_ngrams.shape == (

IOError: first argument must be an open file

If I replace

with utils.smart_open(self.file_name, 'rb') as f:

from https://github.com/RaRe-Technologies/gensim/blob/875b028ccd009fe2fa7665e177b8ab0b5e2dc40d/gensim/models/fasttext.py#L711 to

with open(self.file_name, 'rb') as f:

this will works correctly (also, it works correctly with smart_open==1.5.6, i.e. bug from 1.6.0).

Issue Analytics

State:
Created 5 years ago
Comments:12 (7 by maintainers)

Top GitHub Comments

1reaction

piskvorkycommented, Jul 5, 2018

Let me add that the bug is the violation of the smart_open contract:

“For local uncompressed files, smart_open() is always a drop-in replacement for open(). There is never a reason not to use smart_open() instead of open().”

0reactions

menshikh-ivcommented, Aug 13, 2018

Fixed by #208

Top Results From Across the Web

How Openness Affects Your Behavior - Verywell Mind

Openness is one of the Big Five personality traits and involves being creative, curious, and interested in new experiences.

Open or closed behavior - Medium

In this sub-series I'll go into a few of the examples. This time: do you go for open or closed behavior? Or: are...

window.open(url) different behavior - same code, different timing

This function is called from two different places: user can scroll list of reports and click toolbar button on one of them and...

Different behaviour in OPEN DATASET <variable> FOR ...

I have strange issue here. In my source code, I am using OPEN DATASET <variable> FOR OUTPUT IN TEXT MODE statement. This works...

Failure behaviour of open and closed cracks under different ...

In this study, failure behaviour of open and closed cracks under different loading conditions was investigated. For this purpose, the compression tests on ......