I’ve set up ClusterODM and am trying to send jobs from WebODM. The entire pipeline seems to run successfully, but fails at the last step during the Report stage.
Commands pasted from WebODM:
build-overviews: true, dem-resolution: 2.0, dsm: true, dtm: true, feature-quality: medium, ignore-gsd: true, max-concurrency: 10, optimize-disk-space: true, orthophoto-compression: NONE, orthophoto-cutline: true, orthophoto-resolution: 2.0, pc-tile: true, skip-band-alignment: true, skip-report: true, sm-cluster: http://192.168.1.2:4000, split: 100, split-overlap: 50, tiles: true, verbose: true
Error return at the very end of the log:
...
[INFO] Computing feather raster: /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0000/odm_orthophoto/odm_orthophoto_feathered.tif
[INFO] Finished odm_orthophoto stage
[INFO] Running odm_report stage
[INFO] Exporting shots.geojson
[INFO] Wrote /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0000/odm_report/shots.geojson
[WARNING] Skipping report generation as requested
[INFO] Finished odm_report stage
[INFO] No more stages to run
[INFO] ODM app finished - Tue Mar 23 17:24:41 2021
[INFO] LRE: submodel_0000 finished successfully
[INFO] LRE: No remote tasks left to cleanup
Traceback (most recent call last):
File "/code/run.py", line 68, in <module>
app.execute()
File "/code/stages/odm_app.py", line 81, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 338, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 319, in run
self.process(self.args, outputs)
File "/code/stages/splitmerge.py", line 225, in process
lre.run_toolchain()
File "/code/opendm/remote.py", line 58, in run_toolchain
self.run(ToolchainTask)
File "/code/opendm/remote.py", line 252, in run
raise nonloc.error
pyodm.exceptions.TaskFailedError: (bef4bf43-620a-4e56-9d99-dd5c330ef35d) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 222, in process
cleanup_disk_space()
File "/code/stages/run_opensfm.py", line 45, in cleanup_disk_space
shutil.rmtree(folder_path)
File "/usr/lib/python3.8/shutil.py", line 706, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 704, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/bef4bf43-620a-4e56-9d99-dd5c330ef35d/opensfm/reports'
Full log saved at /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0006/error.log
Inspecting the log, it looks like there are similar errors that occur during processing, but do not crash the entire process.
For example:
...
[INFO] LRE: Upload of submodel_0008 at [57%]
[INFO] LRE: Upload of submodel_0008 at [100%]
Estimated depth-maps 1 (0.54%, 2s, ETA 0ms)...
[INFO] LRE: About to process submodel_0009 remotely
[INFO] LRE: Upload of submodel_0009 at [0%]
Estimated depth-maps 2 (1.09%, 3s, ETA 5m)...
[INFO] LRE: Upload of submodel_0009 at [37%]
[WARNING] LRE: submodel_0002 failed with: (004add26-7b3a-4169-89d7-040851eaad80) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 222, in process
cleanup_disk_space()
File "/code/stages/run_opensfm.py", line 45, in cleanup_disk_space
shutil.rmtree(folder_path)
File "/usr/lib/python3.8/shutil.py", line 706, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 704, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/004add26-7b3a-4169-89d7-040851eaad80/opensfm/reports'
Full log saved at /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0002/error.log
[INFO] LRE: Cleaning up remote task (004add26-7b3a-4169-89d7-040851eaad80)... OK
Estimated depth-maps 3 (1.63%, 6s, ETA 6m)...
[INFO] LRE: Upload of submodel_0009 at [81%]
[INFO] LRE: Upload of submodel_0009 at [100%]
Estimated depth-maps 4 (2.17%, 8s, ETA 6m)...
[WARNING] LRE: submodel_0003 failed with: (ec4c71ab-b9a4-4a78-ae66-d67360baeb1a) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 222, in process
cleanup_disk_space()
File "/code/stages/run_opensfm.py", line 45, in cleanup_disk_space
shutil.rmtree(folder_path)
File "/usr/lib/python3.8/shutil.py", line 706, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 704, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/ec4c71ab-b9a4-4a78-ae66-d67360baeb1a/opensfm/reports'
All the errors end with something like FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/5b3345eb-3359-4ee8-8dd8-6b663ed2af25/opensfm/reports'
I thought it might have something to do with the report generation, but the same error occurred with skip-report flag turned on.
Any ideas?
Edit: This same job runs successfully from WebODM using the default NodeODM instance.