Open large geotiff raster failed due to file pointer math overflow, and also very slow opening large grid
See original GitHub issueThis issue is referenced in the issue #901.
In the DemoMap application, when opening very large grid (2GB+), it failed in DotSpatial.Data.PyramidImage.WriteWindow
function as shown below:
In issue #901, @VectorZita suggested increasing pointer math variable during Seek operation to get over the overflow problem, I’ve tried this and it indeed got further, but there are other places also have this problem.
Issue Analytics
- State:
- Created 3 years ago
- Comments:16 (12 by maintainers)
Top Results From Across the Web
Open large geotif file - python
I have very large geotif file. But I can not open it in colabs. RAM is not enough. So all the time I...
Read more >Can't open large Tif image
I am trying to open a large Tif image file - 6GB. When I try to open it I get the error message...
Read more >Reduce processing time for large GeoTIFF image
I am trying to process large GeoTIFF images (128 MB) using GeoTIFF reader provided by Geotools library. However the processing time is very...
Read more >Read() when more - GitHub
(#3083) * disable LERC on big-endian hosts, as it is not big-endian ready ... more informative error message when opening raster dataset in...
Read more >CoastWatch Software Library and Utilities User's Guide
On startup, this software queries network servers for up-to-date version information, at which time data is collected and stored by the server automatically ......
Read more >
Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free
Top Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Yes. Using the
GdalImageProvider
of the older commit of @jany-tenaj is far more performant, GDAL is a lot faster indeed. @TongZhai do try to use this different version to compare for speed, using the simple code excerpt I posted above.The only problem to pay attention to is that the unsuspecting user will see an increase in their file because GDAL overviews are built internally. At the moment, I do not have enough information on how to control this, therefore, @TongZhai, be warned, when testing for this, you may want to use a copy.
I assume GDAL will have accounted for the case of very large files… maybe overviews for very large files are built externally, but I will have to double-check that.
Regarding the existing
MapRasterLayer
implementation, it is slightly off in terms of performance for large rasters. A faster method will definitely have to be implemented if a more “standalone” DotSpatial behavior is needed, which can probably do something along the lines of what I wrote earlier, i.e. what theGdalImageProvider
currently does.OK, it seems that what I am thinking of doing (switching the raster rendering with a simple raw-data write and down-sampling) is already implemented in the
GdalImageProvider
. Therefore, there is what I have been missing all along.Overviews are built externally as
.mwi
and.mwh
files.This should be faster for now. @TongZhai do give this a shot to see if the raster loads faster. You may have to adjust your
int
values tolong
once again because of the overflow problem, but thereafter, it should work out of the box. I am not entirely sure that the bug fixes I proposed earlier (#1326 and #1332) will also cover this case. If the raster does not appear immediately, you may need to zoom in a bit and let me know so that I can also look into this. However, I think these fixes should have this case covered as well.I will try the older commit of @jany-tenaj now to see how this compares in terms of performance.