After a few successful jobs, I am now consistently getting exit code 1 errors from WebODM when doing a 3D Model. I am using a data set that has been processed with success several times last week.
Docker is configured with 16 GB of RAM and 250 GB of disk space, which should be a lot more than needed.
Have tried to do an ‘update’ of WebODM, but the result is stil exit code 1.
How do i proceed ?
This is the log data containing the error:
2020-11-20 15:09:11,550 DEBUG: Computing sift with threshold 0.06666666666666667
2020-11-20 15:09:11,625 DEBUG: Found 16 points in 21.634562969207764s
2020-11-20 15:09:11,643 DEBUG: reducing threshold
2020-11-20 15:09:11,644 DEBUG: Computing sift with threshold 0.06666666666666667
2020-11-20 15:09:11,789 DEBUG: Found 724 points in 21.803841590881348s
2020-11-20 15:09:11,805 DEBUG: reducing threshold
2020-11-20 15:09:11,805 DEBUG: Computing sift with threshold 0.06666666666666667
Traceback (most recent call last):
File “/code/SuperBuild/src/opensfm/bin/opensfm_main.py”, line 8, in
commands.command_runner(commands.opensfm_commands)
File “/code/SuperBuild/src/opensfm/opensfm/commands/command_runner.py”, line 27, in command_runner
command.run(args)
File “/code/SuperBuild/src/opensfm/opensfm/commands/command.py”, line 12, in run
self.run_impl(data, args)
File “/code/SuperBuild/src/opensfm/opensfm/commands/detect_features.py”, line 10, in run_impl
detect_features.run_dataset(dataset)
File “/code/SuperBuild/src/opensfm/opensfm/actions/detect_features.py”, line 21, in run_dataset
parallel_map(detect, arguments, processes, 1)
File “/code/SuperBuild/src/opensfm/opensfm/context.py”, line 66, in parallel_map
res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
File “/usr/local/lib/python3.8/dist-packages/joblib/parallel.py”, line 1061, in call
self.retrieve()
File “/usr/local/lib/python3.8/dist-packages/joblib/parallel.py”, line 940, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File “/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py”, line 542, in wrap_future_result
return future.result(timeout=timeout)
File “/usr/lib/python3.8/concurrent/futures/_base.py”, line 439, in result
return self.__get_result()
File “/usr/lib/python3.8/concurrent/futures/_base.py”, line 388, in __get_result
raise self._exception
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
The exit codes of the workers are {SIGKILL(-9)}
Traceback (most recent call last):
File “/code/run.py”, line 69, in
app.execute()
File “/code/stages/odm_app.py”, line 86, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 361, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 361, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 361, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 342, in run
self.process(self.args, outputs)
File “/code/stages/run_opensfm.py”, line 30, in process
octx.feature_matching(self.rerun())
File “/code/opendm/osfm.py”, line 273, in feature_matching
self.run(‘detect_features’)
File “/code/opendm/osfm.py”, line 22, in run
system.run(’%s/bin/opensfm %s “%s”’ %
File “/code/opendm/system.py”, line 79, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1
- so adjusted docker down to 16 - and disabled autostart.
