"sqlite3_open" error whith geopackage

The error occur in images_to_samples.py, when we have more than 1000 images to prepare.

...  
Images/mtq_f09_257000_5077000_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12404}
Images/15_2525134f09_dc_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12501}ERROR 4: sqlite3_open(chapeau_mtm09_lac_riv_build.gpkg) failed: unable to open database file

Images/mtq_f09_260000_5081000_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12516}
Images/15_2495160f09_dc_45_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12714}
Images/15_2575131f09_dc_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12714}
Images/15_2605131f09_dc_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12756}
Images/15_2455150f09_dc_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12793}
Images/mtq_f09_262000_5093000_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12799}
Images/15_2715113f09_dc_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12854}
Images/15_2505138f09_dc_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12858}
Images/15_2595126f09_dc_50_int_rou_asp_slo.tif
{'trn': 42601, 'val': 12888}
Traceback (most recent call last):
  File "images_to_samples.py", line 258, in <module>
    params['sample']['mask_reference'])
  File "images_to_samples.py", line 209, in main
    vector_to_raster(info['gpkg'], info['attribute_name'], label_raster)
  File "images_to_samples.py", line 133, in vector_to_raster
    source_layer = source_ds.GetLayer()
AttributeError: 'NoneType' object has no attribute 'GetLayer'

One of the possible solutions might include:

• Using Fiona to read/write geopackages : Fiona is OGR’s neat and nimble API for Python programmers.
• Using Rasterio for raster read/write : Rasterio reads and writes geospatial raster datasets

[Dan-Eli]

The following was tested on Windows 10 using Anaconda.

In conda 4.5.12 python 3.7.2 and activating Fiona 1.8.4 and Rasterio 1.0.13 are good alternative to direct use of OGR and GDAL as described in image_to_samples.py in the preceding comment.

I tested a prototype using Fiona and Rasterio to read a GeoPackage file containing 500 000 vector features and for each feature reading a portion of a GeoTIFF file using Rasterio and it works well.

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import fiona
schema = {'geometry': 'Point', 'properties': [('title', 'str')]}
crs = {'init': 'EPSG:4326'}
with fiona.open('foo.gpkg', 'w', layer='bar_foo', driver='GPKG',
schema=schema, crs=crs) as d:
    d.writerecords([{
                    'geometry': {'type': 'Point', 'coordinates': [0.0, 0.0]},
                    'properties': {'title': 'One'}}])

I found one problem when creating the GPKG file with Fiona it exits with the following message

ERROR 1: sqlite3_exec(CREATE VIRTUAL TABLE "rtree_line_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: table "rtree_line_geom" already exists
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'sqlite3_exec(CREATE VIRTUAL TABLE "rtree_line_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: table "rtree_line_geom" already exists'
Exception ignored in: 'fiona._shim.gdal_flush_cache'
Traceback (most recent call last):
  File "fiona/_err.pyx", line 201, in fiona._err.GDALErrCtxManager.__exit__
fiona._err.CPLE_AppDefinedError: b'sqlite3_exec(CREATE VIRTUAL TABLE "rtree_line_geom" USING rtree(id, minx, maxx, miny, maxy)) failed: table "rtree_line_geom" already exists'

Process finished with exit code 0

It’s not an error as the Process exit with code 0 and it said in the message Exception ignored. The GPKG file seems to be created correctly.

As reported here by @sgillies “This has nothing to do with fiona or the rtree python package. See section 2 of https://www.sqlite.org/rtree.html.”

I was able to run this python program without the previous error in conda 4.5.12 in python 3.7.2 by importing manually the Fiona wheel Fiona‑1.8.4‑cp37‑cp37m‑win_amd64.whl and the GDAL wheel GDAL‑2.3.3‑cp37‑cp37m‑win_amd64.whl and addiing the path to the GDAL DLL

set PATH=%PATH%;<your local path>\Lib\site-packages\osgeo

Finally, I noticed at least one difference in the .gpkh file produced. In the environement wich produced the error, the spatial index of the layers are not created whereas they are created in the environment from the wheel distribution.

So two options are possible for solving this issue:

• In anadonda installing Fiona and GDAL manually the wheel (pip install…) and it works without any Traceback
• Make a request to Anaconda project to modify SQLite product to compile the SQLite with the rtree activated.

The second avenue (request to Anaconda) should be the prefered choice.

[ymoisan]

DanEli bypassed the conda packaging problem by doing what he describes above : create a “vanilla” conda environment with Python 3.7 and manually install the wheel files and make sure they’re on the path.

We have not tested with more recent versions of conda and/or Fiona so maybe the issue is fixed now but if it’s not you’ll have to manually install stuff. Sorry we can’t be more helpful. If you do find how to set this up nice and clean, please let us know.

Good luck