Tasks suddenly not completing in WebODM lightning

[kartelle]

Hello

I tried running a new task in webODM Lightning today, using fast-orthophoto option, but the task kept crashing with the error pasted below. I then tried running it with only default options (no fast-orthophoto selected), and then also tried to run an old project/task (completely different dataset and set of images) with the same settings that ran fine previously, and in each case still got the same (or very similar) error message. Does anyone have any insights please on what might be wrong? WebODM Lightning has worked great for a number of previous projects, not sure why it’s suddenly not cooperating

THanks kindly!

Your task [Task of 2021-09-07T18:29:26.657Z] could not complete. Here’s a copy of the error message:

[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Writing exif overrides
[INFO] Maximum photo dimensions: 5472px
[INFO] Photo dimensions for feature extraction: 2736px
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] [‘use_exif_size: no’, ‘flann_algorithm: KDTREE’, ‘feature_process_size: 2736’, ‘feature_min_frames: 8000’, ‘processes: 8’, ‘matching_gps_neighbors: 8’, ‘matching_gps_distance: 0’, ‘optimize_camera_parameters: yes’, ‘undistorted_image_format: tif’, ‘bundle_outlier_filtering_type: AUTO’, ‘sift_peak_threshold: 0.066’, ‘align_orientation_prior: vertical’, ‘triangulation_type: ROBUST’, ‘retriangulation_ratio: 2’, ‘feature_type: SIFT’, ‘use_altitude_tag: yes’, ‘align_method: auto’, ‘local_bundle_radius: 0’]
[INFO] Wrote reference_lla.json
[INFO] running /code/SuperBuild/install/bin/opensfm/bin/opensfm extract_metadata “/var/www/data/0ce6ac19-f83c-45af-a7be-e47779164c21/opensfm”
/code/SuperBuild/install/bin/opensfm/bin/opensfm: line 12: python: command not found

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 127
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 94, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 340, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 340, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 340, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 321, in run
self.process(self.args, outputs)
File “/code/stages/run_opensfm.py”, line 33, in process
octx.extract_metadata(self.rerun())
File “/code/opendm/osfm.py”, line 284, in extract_metadata
self.run(‘extract_metadata’)
File “/code/opendm/osfm.py”, line 34, in run
system.run(’%s %s “%s”’ %
File “/code/opendm/system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 127

[Saijin_Naib]

Welcome!

Sorry you hit a snag.

It looks like we’re having a bit of a moment, unfortunately:
https://status.webodm.net/

I would expect things to be resolved shortly

[pierotofy]

Thanks for reporting this, and apologies for the inconvenience.

This was a strange bug due to an older version of docker running on some nodes. I’ve never seen this issue before, and this is why we didn’t catch it before deploying in production. We always test new updates before deploying to lightning, but this one was new and unexpected.

If you reprocess it should work this time around.