Split-merge does not finalize

I have been processing large datasets (approximately 3000 images) using the split-merge function

I use the following command:
docker run -it --rm -v /images:/datasets/code opendronemap/odm --project-path /datasets --split 150 --split-overlap 10 --ignore-gsd --orthophoto-resolution 0.3 --dem-resolution 0.3 --dsm --merge pointcloud

The submodels are correctly processed, and I can get the orthophotos from there, but they are not clipped to the proper extent. The process finishes as a failed process with the following error:

How can I fix this?

e[39m[INFO] running pdal translate “ept:///datasets/code/entwine_pointcloud” "/datasets/code/odm_georeferencing/odm_georeferenced_model.laz"e[0m
(pdal translate readers.ept Warning) 492221687 will be downloaded
e[39m[INFO] Merging all crop bounds: [’/datasets/code/submodels/submodel_0000/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0001/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0002/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0003/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0004/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0005/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0006/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0007/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0008/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0009/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0010/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0011/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0012/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0013/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0014/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0015/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0016/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0017/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0018/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’, ‘/datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.gpkg’]e[0m
Traceback (most recent call last):
File “/code/run.py”, line 56, in
app.execute()
File “/code/stages/odm_app.py”, line 93, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 376, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 357, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 206, in process
Cropper.merge_bounds(all_bounds, merged_bounds_file, 0)
File “/code/opendm/cropper.py”, line 82, in merge_bounds
layer = ds.GetLayer()
AttributeError: ‘NoneType’ object has no attribute ‘GetLayer’

It seems like there is some problem with the .gpkg files, as seen in the output for every submodel:

[INFO] running gdalwarp -cutline /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.gpkg -crop_to_cutline -co BLOCKXSIZE=512 -co COMPRESS=LZW -co TILED=YES -co NUM_THREADS=8 -co BLOCKYSIZE=512 /datasets/code/submodels/submodel_0019/odm_dem/dsm.original.tif /datasets/code/submodels/submodel_0019/odm_dem/dsm.tif --config GDAL_CACHEMAX 46.55%e[0m
ERROR 1: Cannot open /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.gpkg.
e[93m[WARNING] Something went wrong while cropping: Child returned 1e[0m
e[39m[INFO] Cropping /datasets/code/submodels/submodel_0019/odm_dem/dsm.unfilled.tife[0m
e[39m[INFO] running gdalwarp -cutline /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.gpkg -crop_to_cutline -co BLOCKXSIZE=512 -co COMPRESS=LZW -co TILED=YES -co NUM_THREADS=8 -co BLOCKYSIZE=512 /datasets/code/submodels/submodel_0019/odm_dem/dsm.unfilled.original.tif /datasets/code/submodels/submodel_0019/odm_dem/dsm.unfilled.tif --config GDAL_CACHEMAX 46.55%e[0m
ERROR 1: Cannot open /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.gpkg.
e[93m[WARNING] Something went wrong while cropping: Child returned 1e[0m

Which seems to come from some issue with GDAL itself:

[INFO] Calculating cropping area and generating bounds shapefile from point cloude[0m
e[39m[INFO] running pdal translate -i “/datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.laz” -o “/datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.decimated.las” decimation --filters.decimation.step=90 e[0m
e[39m[INFO] running pdal info --boundary --filters.hexbin.edge_size=1 --filters.hexbin.threshold=0 /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.decimated.las > /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.boundary.jsone[0m
(pdal info filters.hexbin Error) GDAL failure (1) OGR_G_RemoveGeometry() not supported on polygons yet.
e[39m[INFO] running pdal info --summary /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.laz > /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.summary.jsone[0m
e[39m[INFO] running ogr2ogr -overwrite -f GPKG -a_srs “+proj=utm +zone=11 +datum=WGS84 +units=m +no_defs” /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.gpkg /datasets/code/submodels/submodel_0019/odm_georeferencing/odm_georeferenced_model.bounds.geojsone[0m
ERROR 1: sqlite3_exec(COMMIT) failed: database is locked
ERROR 1: sqlite3_exec(PRAGMA synchronous = OFF) failed: Safety level may not be changed inside a transaction
ERROR 1: sqlite3_exec(BEGIN) failed: cannot start a transaction within a transaction
ERROR 1: Transaction not established

This could be a bug. Mind opening a bug report on https://github.com/OpenDroneMap/ODM/issues?

What OS are you on?

I run it on a linux server, running CentOS Linux 7, using ODM directly. The behavior repeats if I use nodeODM too.

It is now an issue in GitHub (https://github.com/OpenDroneMap/ODM/issues/1016). Any suggestions on how to work around the issue will be highly appreciated!