Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Open large geotiff raster failed due to file pointer math overflow, and also very slow opening large grid

See original GitHub issue

This issue is referenced in the issue #901. In the DemoMap application, when opening very large grid (2GB+), it failed in DotSpatial.Data.PyramidImage.WriteWindow function as shown below:


In issue #901, @VectorZita suggested increasing pointer math variable during Seek operation to get over the overflow problem, I’ve tried this and it indeed got further, but there are other places also have this problem.

Issue Analytics

  • State:open
  • Created 3 years ago
  • Comments:16 (12 by maintainers)

github_iconTop GitHub Comments

VectorZitacommented, May 1, 2020

Yes. Using the GdalImageProvider of the older commit of @jany-tenaj is far more performant, GDAL is a lot faster indeed. @TongZhai do try to use this different version to compare for speed, using the simple code excerpt I posted above.

The only problem to pay attention to is that the unsuspecting user will see an increase in their file because GDAL overviews are built internally. At the moment, I do not have enough information on how to control this, therefore, @TongZhai, be warned, when testing for this, you may want to use a copy.

I assume GDAL will have accounted for the case of very large files… maybe overviews for very large files are built externally, but I will have to double-check that.

Regarding the existing MapRasterLayer implementation, it is slightly off in terms of performance for large rasters. A faster method will definitely have to be implemented if a more “standalone” DotSpatial behavior is needed, which can probably do something along the lines of what I wrote earlier, i.e. what the GdalImageProvider currently does.

VectorZitacommented, May 1, 2020

OK, it seems that what I am thinking of doing (switching the raster rendering with a simple raw-data write and down-sampling) is already implemented in the GdalImageProvider. Therefore, there is what I have been missing all along.

GdalImageProvider gip = new GdalImageProvider();

var imageData = gip.Open(filename);

MapImageLayer imgLayer = new MapImageLayer(imageData);


Overviews are built externally as .mwi and .mwh files.

This should be faster for now. @TongZhai do give this a shot to see if the raster loads faster. You may have to adjust your int values to long once again because of the overflow problem, but thereafter, it should work out of the box. I am not entirely sure that the bug fixes I proposed earlier (#1326 and #1332) will also cover this case. If the raster does not appear immediately, you may need to zoom in a bit and let me know so that I can also look into this. However, I think these fixes should have this case covered as well.

I will try the older commit of @jany-tenaj now to see how this compares in terms of performance.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Open large geotif file - python
I have very large geotif file. But I can not open it in colabs. RAM is not enough. So all the time I...
Read more >
Can't open large Tif image
I am trying to open a large Tif image file - 6GB. When I try to open it I get the error message...
Read more >
Reduce processing time for large GeoTIFF image
I am trying to process large GeoTIFF images (128 MB) using GeoTIFF reader provided by Geotools library. However the processing time is very...
Read more >
Read() when more - GitHub
(#3083) * disable LERC on big-endian hosts, as it is not big-endian ready ... more informative error message when opening raster dataset in...
Read more >
CoastWatch Software Library and Utilities User's Guide
On startup, this software queries network servers for up-to-date version information, at which time data is collected and stored by the server automatically ......
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Post

No results found

github_iconTop Related Hashnode Post

No results found