Cannot process dataset, fail after 235 hours

Gordon · 26 November 2021 22:48

Continued from a thread I started in The Hangar, moved to here, a more relevant category.

I hate it even more when the task fails after over 235 hours, just before I expected it to finish! What a waste of a week and a half!

X Cannot process dataset

9354 images scaled by 0.5X

Options: crop: 1, dsm: true, matcher-distance: 60, matcher-neighbors: 24, mesh-octree-depth: 12, mesh-size: 300000, min-num-features: 11000, pc-rectify: true, split: 2000, split-overlap: 200, use-3dmesh: true, rerun-from: odm_postprocess

End of the console log, hopefully someone can figure out why it failed so close to the end. There’s plenty more log file available if it would be of use, but it wont fit here, but I could put it on Google Drive if needed.

[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm_r0.0417012088862819_x4_y3.tif (max, radius: 0.0417012088862819, resolution: 0.0834024177725638)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpox_xnt1s.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm_r0.0417012088862819_x4_y4.tif (max, radius: 0.0417012088862819, resolution: 0.0834024177725638)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpaek8d2sn.json”
[INFO] running gdalbuildvrt -input_file_list “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles_list.txt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.vrt”
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=16 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 44.75% “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.vrt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.tmp.tif”
Input file size is 18071, 16047
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=16 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 44.75% -outsize 10% 0 “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.tmp.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.small.tif”
0…10…20…30…40…50…60…70…80…90…Input file size is 18071, 16047
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdalbuildvrt -resolution highest -r bilinear “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\merged.vrt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.small_filled.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.tmp.tif”
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=16 -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=DEFLATE --config GDAL_CACHEMAX 44.75% “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\merged.vrt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\tiles.tif”
Input file size is 18071, 16047
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] Starting smoothing…
[INFO] Smoothing iteration 1
[INFO] Completed smoothing to create E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.tif in 0:01:06.691529
[INFO] Completed dsm.tif in 0:04:15.615726
[INFO] Cropping E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.tif
[INFO] running gdalwarp -cutline “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_georeferencing\odm_georeferenced_model.bounds.gpkg” -crop_to_cutline -co TILED=YES -co COMPRESS=DEFLATE -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co BIGTIFF=IF_SAFER -co NUM_THREADS=16 “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.original.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.tif” --config GDAL_CACHEMAX 43.8%
Creating output file that is 17959P x 15943L.
Processing E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.original.tif [1/1] : 0Using internal nodata values (e.g. -9999) for image E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.original.tif.
Copying nodata values from source E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.original.tif to destination E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.tif.
…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] Cropping E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.tif
[INFO] running gdalwarp -cutline “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_georeferencing\odm_georeferenced_model.bounds.gpkg” -crop_to_cutline -co TILED=YES -co COMPRESS=DEFLATE -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co BIGTIFF=IF_SAFER -co NUM_THREADS=16 “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.original.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.tif” --config GDAL_CACHEMAX 43.75%
Creating output file that is 17959P x 15943L.
Processing E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.original.tif [1/1] : 0Using internal nodata values (e.g. -9999) for image E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.original.tif.
Copying nodata values from source E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.original.tif to destination E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.tif.
…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] Computing euclidean distance: E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.euclideand.tif
[INFO] running gdal_proximity.py “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.unfilled.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0000\odm_dem\dsm.euclideand.tif” -values -9999.0

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
[Previous line repeated 6 more times]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\odm_dem.py”, line 125, in process
commands.compute_euclidean_map(unfilled_dem_path,
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\commands.py”, line 293, in compute_euclidean_map
run(‘gdal_proximity.py “%s” “%s” -values %s’ % (geotiff_path, output_path, nodata))
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.6.7 documentation
100 - done.

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\splitmerge.py”, line 158, in process
system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1
===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.6.7 documentation

itenviro · 27 November 2021 02:10

I have experienced the same multiple times, processing ~ 10 000 images. See a previous post - " Processing exited after submodel stage" for the details.
What is the greatest number of images you have successfully processed for ortho and model?
is there a size limitation on odm irrespective of the hardware?

smathermather · 27 November 2021 02:25

I’ve processed ~90,000. I haven’t found the upper limit yet.

itenviro · 27 November 2021 02:37

That is good to know. Was getting discouraged.
Can you share the parameters and number images so we can try with our data?
This was large scale computing at CERN or a commodore 64 or ??
Would be good to develop a thread on processing large datasets on single desktop node.

smathermather · 27 November 2021 02:39

Yeah, I am going to write a series on it. I have gone further over the edge of that horizon than any… Lolsob.

itenviro · 27 November 2021 02:40

Great. Thanks. I am looking forward to it.

smathermather · 27 November 2021 02:48

I don’t believe in loosing 235 hours, but we’ll need to make sure we back up your intermediate products to prevent that. How do you have it installed? Via docker on Linux or some other way?

Gordon · 27 November 2021 04:18

Just using WebODM, I had a look and found 1 submodel .obj file which looks reasonable in Meshlab, but didn’t find much else of use.
Ideally it would be good to know why it failed, so that I could try again with more suitable settings.

Gordon · 27 November 2021 04:28

I’ve made it to the end of the same 9354 image set with it failing, but the point cloud had points spread all over the place, kilometres from where I imaged, and the orthomosaic had lots of area around the edges missing. I used 0.33X scaling for that, with these settings -

Options: cameras: {“hasselblad l1d-20c 2048 1365 brown 0.7777 rgb”:{“projection_type”:“brown”,“width”:2048,“height”:1365,“focal_x”:0.7777778798476017,“focal_y”:0.7777778798476017,“c_x”:-1.4353584987681075e-9,“c_y”:2.7706085708877087e-8,“k1”:4.1401891459771804e-7,“k2”:2.502759404555409e-7,“p1”:0.0000016387400806644321,“p2”:5.352177439969789e-7,“k3”:1.504782081263804e-7}}, crop: 0, debug: true, dem-decimation: 3, dem-resolution: 5, dtm: true, feature-type: orb, matcher-distance: 60, matcher-neighbors: 20, mesh-octree-depth: 12, mesh-size: 300000, min-num-features: 15000, orthophoto-resolution: 5, pc-rectify: true, use-3dmesh: true, verbose: true
Average GSD: 8.22 cm
Area: 8,448,784.18 m²
Reconstructed Points: 91,682,871

Using an i7 3.8GHz processor and 96GB RAM

smathermather · 27 November 2021 04:37

Sounds like a good case for throwing the auto-boundary flag to constrain the model to a conservative footprint of roughly the camera inputs. I don’t know what portion of the chain that get implemented at, or if it applies to split merge, but you could edit the project, throw that flag, and resume processing and see if it works.

smathermather · 27 November 2021 04:44

(Just do it before nodeodm auto deletes your progress).

Gordon · 27 November 2021 04:53

Probably a bit late to do that now, as it was 10 days ago, the attempt before the one that failed today, and I wasn’t using split-merge on that attempt.
I can get subsets of this area to complete properly, but trying to combine a number of them fails, and not due to lack of memory.

Gordon · 28 November 2021 06:31

I restarted from meshing and got myself another submodel done (now 2/5), but it has failed again, with the same double dump after 5h20 ;(
Now over 240 hours (not counting down time) into this task!

Options: crop: 1, dsm: true, matcher-distance: 60, matcher-neighbors: 24, mesh-octree-depth: 12, mesh-size: 300000, min-num-features: 11000, pc-rectify: true, split: 2000, split-overlap: 200, use-3dmesh: true, rerun-from: odm_meshing

Working on atlas 58 99%…
Working on atlas 59 100%… done.
Finalizing texture atlases… done. (Took: 0s)
Building objmodel:
Saving model… done.
Whole texturing procedure took: 1059s
[INFO] Finished mvs_texturing stage
[INFO] Running odm_georeferencing stage
[INFO] Georeferencing point cloud
[INFO] running pdal translate -i “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_filterpoints\point_cloud.ply” -o “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz” ferry transformation --filters.ferry.dimensions=“views => UserData” --writers.las.compression=“lazip” --filters.transformation.matrix=“1 0 0 756100.0 0 1 0 6546669.0 0 0 1 0 0 0 0 1” --writers.las.offset_x=756100.0 --writers.las.offset_y=6546669.0 --writers.las.offset_z=0 --writers.las.a_srs=“+proj=utm +zone=55 +south +datum=WGS84 +units=m +no_defs +type=crs”
[INFO] Calculating cropping area and generating bounds shapefile from point cloud
[INFO] running pdal translate -i “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz” -o “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.decimated.las” decimation --filters.decimation.step=95
[INFO] running pdal info --boundary --filters.hexbin.edge_size=1 --filters.hexbin.threshold=0 “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.decimated.las” > “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.boundary.json”
[INFO] running pdal info --summary “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz” > “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.summary.json”
[INFO] running ogr2ogr -overwrite -f GPKG -a_srs “+proj=utm +zone=55 +south +datum=WGS84 +units=m +no_defs” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.bounds.gpkg” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.bounds.geojson”
[INFO] Finished odm_georeferencing stage
[INFO] Running odm_dem stage
[WARNING] Maximum resolution set to 2.0 * (GSD - 10.0%) (8.44 cm / pixel, requested resolution was 5.00 cm / pixel)
[INFO] Classify: True
[INFO] Create DSM: True
[INFO] Create DTM: False
[INFO] DEM input file E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz found: True
[INFO] Classifying E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz using Simple Morphological Filter
[INFO] running pdal translate -i E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz -o E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz smrf --filters.smrf.scalar=1.25 --filters.smrf.slope=0.15 --filters.smrf.threshold=0.5 --filters.smrf.window=18.0
[INFO] Created E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz in 0:00:47.052715
[INFO] running pdal translate -i E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz -o E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\tmp.las
[INFO] Rectifying E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz using with [reclassify threshold: 5, min area: 750, min points: 500]
[INFO] running pdal translate -i E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\tmp.las -o E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz
[INFO] Created E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz in 0:32:41.918621
[INFO] running pdal info --summary “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.laz” > “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpz6u9q7sy.json”
[INFO] Point cloud bounds are [minx: 755390, maxx: 756872.07] [miny: 6545880.91, maxy: 6547535.18]
[INFO] DEM resolution is (19606, 17565), max tile size is 4096, will split DEM generation into 25 tiles
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x0_y0.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x0_y1.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x0_y2.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x0_y3.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x0_y4.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x1_y0.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x1_y1.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x1_y2.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x1_y3.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x1_y4.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x2_y0.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x2_y1.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x2_y2.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.16875286145082338_x2_y3.tif (max, radius: 0.16875286145082338, resolution: 0.08437643072541169)

<Chopped a lot of similar lines out here, due to character limit in post>

[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpfrdm72tf.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x1_y3.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpertn024h.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x1_y4.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmps44rmid6.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x2_y0.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpu546pcrr.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x2_y1.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpas3qjhim.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x2_y2.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp926mfsth.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x2_y3.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpp0_5v730.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x2_y4.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp3x3tvbw3.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x3_y0.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpylcp494b.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x3_y1.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmprllsl7jr.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x3_y2.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmplbg2lu8p.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x3_y3.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpylhbib10.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x3_y4.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpb_pre2gl.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x4_y0.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpotrn60fx.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x4_y1.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpk9_c6ehv.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x4_y2.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpz__by2x7.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x4_y3.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpw7gk6wmg.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.08437643072541169_x4_y4.tif (max, radius: 0.08437643072541169, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp2dhkfqnf.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x0_y0.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpnd7slwk_.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x0_y1.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp6utjghl5.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x0_y2.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp2d_faysx.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x0_y3.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpu5aoan34.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x0_y4.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpi439_arg.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x1_y0.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpy7f2rnzb.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x1_y1.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp43a5hk3b.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x1_y2.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp9x6j8joa.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x1_y3.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpwzm5vfvn.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x1_y4.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp23hhv82k.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x2_y0.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpk7s18jb3.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x2_y1.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpdioyian5.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x2_y2.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp3u9oitzf.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x2_y3.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp69426ng0.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x2_y4.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpk1y2vz9k.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x3_y0.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpkf2ugd56.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x3_y1.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp6rmj9_ig.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x3_y2.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpoveueb8e.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x3_y3.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpx041ty8w.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x3_y4.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmp6l84eufg.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x4_y0.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpn_plm43z.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x4_y1.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmphoilmq8a.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x4_y2.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpt_zbpla2.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x4_y3.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpvigvd6w1.json”
[INFO] Generating E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm_r0.042188215362705846_x4_y4.tif (max, radius: 0.042188215362705846, resolution: 0.08437643072541169)
[INFO] running pdal pipeline -i “C:\Users\OTTDRO~1\AppData\Local\Temp\tmpedjdje0s.json”
[INFO] running gdalbuildvrt -input_file_list “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles_list.txt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.vrt”
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=16 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 33.85% “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.vrt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.tmp.tif”
Input file size is 17565, 19607
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=16 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 33.85% -outsize 10% 0 “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.tmp.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.small.tif”
0…10…20…30…40…50…60…70…80…90…Input file size is 17565, 19607
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdalbuildvrt -resolution highest -r bilinear “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\merged.vrt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.small_filled.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.tmp.tif”
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=16 -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=DEFLATE --config GDAL_CACHEMAX 33.85% “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\merged.vrt” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\tiles.tif”
Input file size is 17565, 19607
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] Starting smoothing…
[INFO] Smoothing iteration 1
[INFO] Completed smoothing to create E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.tif in 0:01:13.204933
[INFO] Completed dsm.tif in 0:04:14.304674
[INFO] Cropping E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.tif
[INFO] running gdalwarp -cutline “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.bounds.gpkg” -crop_to_cutline -co TILED=YES -co COMPRESS=DEFLATE -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co BIGTIFF=IF_SAFER -co NUM_THREADS=16 “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.original.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.tif” --config GDAL_CACHEMAX 33.3%
Creating output file that is 17424P x 19428L.
Processing E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.original.tif [1/1] : 0Using internal nodata values (e.g. -9999) for image E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.original.tif.
Copying nodata values from source E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.original.tif to destination E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.tif.
…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] Cropping E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.tif
[INFO] running gdalwarp -cutline “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_georeferencing\odm_georeferenced_model.bounds.gpkg” -crop_to_cutline -co TILED=YES -co COMPRESS=DEFLATE -co BLOCKXSIZE=512 -co BLOCKYSIZE=512 -co BIGTIFF=IF_SAFER -co NUM_THREADS=16 “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.original.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.tif” --config GDAL_CACHEMAX 33.25%
Creating output file that is 17424P x 19428L.
Processing E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.original.tif [1/1] : 0Using internal nodata values (e.g. -9999) for image E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.original.tif.
Copying nodata values from source E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.original.tif to destination E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.tif.
…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] Computing euclidean distance: E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.euclideand.tif
[INFO] running gdal_proximity.py “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.unfilled.tif” “E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0001\odm_dem\dsm.euclideand.tif” -values -9999.0

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
[Previous line repeated 6 more times]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\odm_dem.py”, line 125, in process
commands.compute_euclidean_map(unfilled_dem_path,
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\commands.py”, line 293, in compute_euclidean_map
run(‘gdal_proximity.py “%s” “%s” -values %s’ % (geotiff_path, output_path, nodata))
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 3.1.7 documentation
100 - done.

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\splitmerge.py”, line 158, in process
system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 3.1.7 documentation

I’ll try restarting again and see if it will produce the 3rd submodel. I wonder if I’ll get an ortho at the end of this?

Gordon · 28 November 2021 12:05

Another 5 1/2 hours leads to another fail to proceed…

EDIT- For submodel 0000 and 0001 I have .obj files and orthophotos, but only the .obj with no ortho for submodel 0002, so I’ll try restarting again to see if more progress can be made!
log ends with this:

[INFO] Finished odm_georeferencing stage
[INFO] Running odm_dem stage
[WARNING] Maximum resolution set to 2.0 * (GSD - 10.0%) (8.53 cm / pixel, requested resolution was 5.00 cm / pixel)
[INFO] Classify: True
[INFO] Create DSM: True
[INFO] Create DTM: False
[INFO] DEM input file E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz found: True
[INFO] Classifying E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz using Simple Morphological Filter
[INFO] running pdal translate -i E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz -o E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz smrf --filters.smrf.scalar=1.25 --filters.smrf.slope=0.15 --filters.smrf.threshold=0.5 --filters.smrf.window=18.0
[INFO] Created E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz in 0:00:41.206207
[INFO] running pdal translate -i E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz -o E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\tmp.las
[INFO] Rectifying E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz using with [reclassify threshold: 5, min area: 750, min points: 500]
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\commands.py”, line 55, in rectify
run_rectification(
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\ground_rectification\rectify.py”, line 24, in run_rectification
point_cloud = extend_cloud(point_cloud, kwargs[‘extend_plan’], kwargs[‘extend_grid_distance’], kwargs[‘min_points’], kwargs[‘min_area’])
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\ground_rectification\rectify.py”, line 85, in extend_cloud
new_points = __calculate_new_points(grid_inside, partition.point_cloud)
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\ground_rectification\rectify.py”, line 116, in __calculate_new_points
model = RANSACRegressor().fit(partition_point_cloud.get_xy(), partition_point_cloud.get_z())
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\linear_model_ransac.py”, line 258, in fit
X, y = self._validate_data(X, y, validate_separately=(check_X_params,
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\base.py”, line 430, in _validate_data
X = check_array(X, **check_X_params)
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\utils\validation.py”, line 63, in inner_f
return f(*args, **kwargs)
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\utils\validation.py”, line 726, in check_array
raise ValueError(“Found array with %d sample(s) (shape=%s) while a”
ValueError: Found array with 0 sample(s) (shape=(0, 2)) while a minimum of 1 is required.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\run.py”, line 53, in
retcode = app.execute()
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 130, in execute
raise e
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
[Previous line repeated 6 more times]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\odm_dem.py”, line 76, in process
commands.rectify(dem_input, args.debug)
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\commands.py”, line 72, in rectify
raise Exception(“Error rectifying ground in file %s: %s” % (lasFile, str(e)))
Exception: Error rectifying ground in file E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz: Found array with 0 sample(s) (shape=(0, 2)) while a minimum of 1 is required.

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\splitmerge.py”, line 158, in process
system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 3.1.7 documentation

Gordon · 28 November 2021 23:39

Looks like I’ve reached the end of the line, as it’s crashed again after 1h11m, end of the log file below.

It’s weird that I was able to restart a couple of times and progress without changing anything and get more orthos and .obj files, before the last 2 attempts when it wouldn’t progress any more, so no ortho for submodel 0002, even though it produced the .obj file earlier.

[INFO] Finished odm_meshing stage
[INFO] Running mvs_texturing stage
[WARNING] Found a valid ODM Texture file in: E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_texturing\odm_textured_model_geo.obj
[INFO] Finished mvs_texturing stage
[INFO] Running odm_georeferencing stage
[WARNING] Found a valid georeferenced model in: E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz
[INFO] Finished odm_georeferencing stage
[INFO] Running odm_dem stage
[WARNING] Maximum resolution set to 2.0 * (GSD - 10.0%) (8.53 cm / pixel, requested resolution was 5.00 cm / pixel)
[INFO] Classify: True
[INFO] Create DSM: True
[INFO] Create DTM: False
[INFO] DEM input file E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz found: True
[INFO] running pdal translate -i E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz -o E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\tmp.las
[INFO] Rectifying E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz using with [reclassify threshold: 5, min area: 750, min points: 500]
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\commands.py”, line 55, in rectify
run_rectification(
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\ground_rectification\rectify.py”, line 24, in run_rectification
point_cloud = extend_cloud(point_cloud, kwargs[‘extend_plan’], kwargs[‘extend_grid_distance’], kwargs[‘min_points’], kwargs[‘min_area’])
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\ground_rectification\rectify.py”, line 85, in extend_cloud
new_points = __calculate_new_points(grid_inside, partition.point_cloud)
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\ground_rectification\rectify.py”, line 116, in __calculate_new_points
model = RANSACRegressor().fit(partition_point_cloud.get_xy(), partition_point_cloud.get_z())
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\linear_model_ransac.py”, line 258, in fit
X, y = self._validate_data(X, y, validate_separately=(check_X_params,
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\base.py”, line 430, in _validate_data
X = check_array(X, **check_X_params)
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\utils\validation.py”, line 63, in inner_f
return f(*args, **kwargs)
File “E:\WebODM\resources\app\apps\ODM\venv\lib\site-packages\sklearn\utils\validation.py”, line 726, in check_array
raise ValueError(“Found array with %d sample(s) (shape=%s) while a”
ValueError: Found array with 0 sample(s) (shape=(0, 2)) while a minimum of 1 is required.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\run.py”, line 53, in
retcode = app.execute()
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 130, in execute
raise e
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
[Previous line repeated 6 more times]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\odm_dem.py”, line 76, in process
commands.rectify(dem_input, args.debug)
File “E:\WebODM\resources\app\apps\ODM\opendm\dem\commands.py”, line 72, in rectify
raise Exception(“Error rectifying ground in file %s: %s” % (lasFile, str(e)))
Exception: Error rectifying ground in file E:\WebODM\resources\app\apps\NodeODM\data\4133ad10-743b-4d2c-a07f-e52b1f07abb9\submodels\submodel_0002\odm_georeferencing\odm_georeferenced_model.laz: Found array with 0 sample(s) (shape=(0, 2)) while a minimum of 1 is required.

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 347, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 328, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\splitmerge.py”, line 158, in process
system.run(" ".join(map(double_quote, map(str, argv))), env_vars=os.environ.copy())
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 3.1.7 documentation

Saijin_Naib · 28 November 2021 23:39

Can you add auto-boundary to the processing parameters?

Gordon · 28 November 2021 23:41

OK, I’ll try that, I’ll get back to you in a couple of weeks when I hope it finishes!
With the issues that people have had with build 45, should I update? I think I’m on 43, as I updated 2 weeks ago before starting on this task.

Saijin_Naib · 28 November 2021 23:44

Do you have an Nvidia GPU in that machine? If so, you should not have any issues. I’ve been unable to reproduce the problems folks have been having, possibly because of that.

It seems the folks having issues still have Intel iGPU or AMD.

jamesgriffiths · 28 November 2021 23:53

I have been having similar problems. I have tested with and without auto-boundary with no luck. Suggestions are welcome. My system is as follows.

GUI Window 11

CPU

DMI Processor
manufacturer Intel(R) Corporation
model Intel(R) Core™ i5-10600K CPU @ 4.10GHz
clock speed 4059.0 MHz
FSB speed 100.0 MHz
multiplier 40.5x
ram 64gb

GPU
Name NVIDIA GeForce GTX 560
Board Manufacturer GIGABYTE Technology
Revision A1
Core family 0xCE (0xCE)
Memory size 1024 MB
Memory type GDDR5

Gordon · 28 November 2021 23:58

Yes, Nvidia GeForce GTX1650 Super, so I’ll update before I start it this time, with auto-boundary