Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
Not enough memory error when parsing pbf dataset
See original GitHub issueHello.
I was trying to parse a pbf file (approx. 65 mb) and I get an “out of memorry” error. I also have noticed that others have this issue as well. My machine specs should be sufficient for parsing this data (16gb of ram, i7 CPU),… I suppose.
I tried to investigate why this memory error occurs on small datasets and in my case it looks like it comes from node elements that are returned from the parse_osm_data (cython method) in pyrosm.py. Each node’s property that is returned is an array of size x (in my case 8000). I couldn’t manage to debug Cython code so I don’t quite understand why this behavior happens. Is there any reason why all these elements are returned only for nodes. ?
Thank you.
Issue Analytics
- State:
- Created 2 years ago
- Reactions:1
- Comments:10 (6 by maintainers)
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Can confirm this issue, i tried also to read the
mittelfrankenandniederbayernpbf files, in both cases the Kernel just died without any error message.Hi @HTenkanen I think this issue has more to it than just the Memory issues. I am currently using this library in a GCP VM where i am facing ipython notebook kernel dying for the smallest of PBF files(Comores 3.3MB) even after raising the RAM size to 500 GB. As per your suggestion i used the 0.5.3 version of the library. However i can get results if i call certain functions like get_network() but when i try to do a custom filter it kills the kernel.