FileNotFoundError from ClusterODM

I’ve set up ClusterODM and am trying to send jobs from WebODM. The entire pipeline seems to run successfully, but fails at the last step during the Report stage.

Commands pasted from WebODM:
build-overviews: true, dem-resolution: 2.0, dsm: true, dtm: true, feature-quality: medium, ignore-gsd: true, max-concurrency: 10, optimize-disk-space: true, orthophoto-compression: NONE, orthophoto-cutline: true, orthophoto-resolution: 2.0, pc-tile: true, skip-band-alignment: true, skip-report: true, sm-cluster: http://192.168.1.2:4000, split: 100, split-overlap: 50, tiles: true, verbose: true

Error return at the very end of the log:

...
[INFO]    Computing feather raster: /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0000/odm_orthophoto/odm_orthophoto_feathered.tif
[INFO]    Finished odm_orthophoto stage
[INFO]    Running odm_report stage
[INFO]    Exporting shots.geojson
[INFO]    Wrote /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0000/odm_report/shots.geojson
[WARNING] Skipping report generation as requested
[INFO]    Finished odm_report stage
[INFO]    No more stages to run
[INFO]    ODM app finished - Tue Mar 23 17:24:41  2021
[INFO]    LRE: submodel_0000 finished successfully
[INFO]    LRE: No remote tasks left to cleanup
Traceback (most recent call last):
File "/code/run.py", line 68, in <module>
app.execute()
File "/code/stages/odm_app.py", line 81, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 338, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 319, in run
self.process(self.args, outputs)
File "/code/stages/splitmerge.py", line 225, in process
lre.run_toolchain()
File "/code/opendm/remote.py", line 58, in run_toolchain
self.run(ToolchainTask)
File "/code/opendm/remote.py", line 252, in run
raise nonloc.error
pyodm.exceptions.TaskFailedError: (bef4bf43-620a-4e56-9d99-dd5c330ef35d) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 222, in process
cleanup_disk_space()
File "/code/stages/run_opensfm.py", line 45, in cleanup_disk_space
shutil.rmtree(folder_path)
File "/usr/lib/python3.8/shutil.py", line 706, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 704, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/bef4bf43-620a-4e56-9d99-dd5c330ef35d/opensfm/reports'
Full log saved at /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0006/error.log

Inspecting the log, it looks like there are similar errors that occur during processing, but do not crash the entire process.

For example:

...
[INFO]    LRE: Upload of submodel_0008 at [57%]
[INFO]    LRE: Upload of submodel_0008 at [100%]
Estimated depth-maps 1 (0.54%, 2s, ETA 0ms)...
[INFO]    LRE: About to process submodel_0009 remotely
[INFO]    LRE: Upload of submodel_0009 at [0%]
Estimated depth-maps 2 (1.09%, 3s, ETA 5m)...
[INFO]    LRE: Upload of submodel_0009 at [37%]
[WARNING] LRE: submodel_0002 failed with: (004add26-7b3a-4169-89d7-040851eaad80) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 222, in process
cleanup_disk_space()
File "/code/stages/run_opensfm.py", line 45, in cleanup_disk_space
shutil.rmtree(folder_path)
File "/usr/lib/python3.8/shutil.py", line 706, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 704, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/004add26-7b3a-4169-89d7-040851eaad80/opensfm/reports'
Full log saved at /var/www/data/5188da7e-3787-4c0f-a54a-338aa72ffe47/submodels/submodel_0002/error.log
[INFO]    LRE: Cleaning up remote task (004add26-7b3a-4169-89d7-040851eaad80)... OK
Estimated depth-maps 3 (1.63%, 6s, ETA 6m)...
[INFO]    LRE: Upload of submodel_0009 at [81%]
[INFO]    LRE: Upload of submodel_0009 at [100%]
Estimated depth-maps 4 (2.17%, 8s, ETA 6m)...
[WARNING] LRE: submodel_0003 failed with: (ec4c71ab-b9a4-4a78-ae66-d67360baeb1a) failed with task output: self.process(self.args, outputs)
File "/code/stages/run_opensfm.py", line 222, in process
cleanup_disk_space()
File "/code/stages/run_opensfm.py", line 45, in cleanup_disk_space
shutil.rmtree(folder_path)
File "/usr/lib/python3.8/shutil.py", line 706, in rmtree
onerror(os.lstat, path, sys.exc_info())
File "/usr/lib/python3.8/shutil.py", line 704, in rmtree
orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/ec4c71ab-b9a4-4a78-ae66-d67360baeb1a/opensfm/reports'

All the errors end with something like FileNotFoundError: [Errno 2] No such file or directory: '/var/www/data/5b3345eb-3359-4ee8-8dd8-6b663ed2af25/opensfm/reports'

I thought it might have something to do with the report generation, but the same error occurred with skip-report flag turned on.

Any ideas?

Edit: This same job runs successfully from WebODM using the default NodeODM instance.

1 Like

It’s a bug. Should be fixed with Fix disk cleanup bug · OpenDroneMap/ODM@6574ffd · GitHub (thanks for reporting this!)

A workaround is also not to use --optimize-disk-space, which triggers the bug.

3 Likes

Very much appreciated!

3 Likes

Confirmed, ClusterODM job completed successfully by omitting --optimize-disk-space.

It did fail with an error that seemed to be related to build-overviews. I turned that off and it succeeded. Are you interested in that log dump?

2 Likes