Problems with memory mapped files and vectors.most_similar()
See original GitHub issueHello. I am on windows using python 3.6
There seems to be a problem with creating the magmap files for using the vectors.most_similar() method. Error:
File "\pymagnitude\__init__.py", line 1074, in get_vectors_mmap
os.rename(path_to_mmap_temp, self.path_to_mmap)
PermissionError: [WinError 32] The process cannot access the file because it is in use by another process \\AppDataTemp\\82ce74f40baf842d23c45b0e90688b9f.magmmap.tmp' -> '\AppData\\Local\\Temp\\82ce74f40baf842d23c45b0e90688b9f.magmmap'
I looked into the code and it seems you are creating and filling these files in the background. It works fine for the vectors.most_similar_approx() method. The memmap file for approx seems to be created just fine.
The problem persists even when i use blocking=True when constructing the magnitude object. Then i can’t even use the approx method. It just hangs while initializing the object.
Thank you for your time.
Issue Analytics
- State:
- Created 6 years ago
- Comments:12 (7 by maintainers)
Top Results From Across the Web
Memory-mapped files considered harmful (for databases)
Memory-mapped files create numerous performance and reliability issues and you should stay clear of DBMS that persist data using them.
Read more >Inconsistent performance of memory mapped files : r/rust
I need this to happen as fast as possible, so I am experimenting: Loading the contents into a vector runs in approximately 10...
Read more >Disadvantages of using memory mapped files - Stack Overflow
The main disadvantage of MMFs is that they consume RAM, making the file system cache less effective. Not an issue with such small...
Read more >3.4. Shared Memory With Memory-mapped Files
Memory -mapped files provide more efficient access for initial reads. When read() is used to access a file, the file contents are first...
Read more >Memory-mapped file - Wikipedia
A memory-mapped file is a segment of virtual memory that has been assigned a direct byte-for-byte correlation with some portion of a file...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
Thank you very much for your time Ajay. I will post as soon as i can, most likely tomorrow.
Your welcome and thanks for the help in making it Windows compatible! I added your handle as a Developer on the GitLab repository. If you come across any other issues, feel free to submit a pull request / merge request to get it merged into master and deployed to PyPI.
Would love to see what you working on / end up doing with Magnitude in your master’s thesis, e-mail us at opensource@plasticity.ai if you can share it.