Process exited with code 1 - adjusted memory, min-num-features but failed in the middle of processing. PLS HELP!


#1

Hello there,

The processing of 100 images on WedODM stopped with the following task output:
https://drive.google.com/file/d/1wP_fkMOqFqRaFpxljNYGGqZraE4yAzSF/view?usp=sharing

For the images and gcp_list.txt file processed on WebODM, please refer to the following:
https://drive.google.com/open?id=14fDfJuE3LKOnl7fPYvBxXDKlIahUvb7g

In the first time, processing was complete without any problem. After replacing a graphic card on my workstation from Nvidia quadro FX4800 to Nvidia GTX 1080, processing started “stopped”.

I have tried to solve this issue with the following adjustments:

  1. Increased Memory to 32 GB ~ 40 GB on Advanced in Setting of Docker
  2. Increased min-num-features option in WebODM

However, processing images on WebODM stopped in the middle without “complete”.

My workstation operates with 2 Xeon processors, 64 GB Ram, GTX 1080 graphic card and 512 GB SSD, which I believe has capacity enough to process images with this project size.

It would be highly appreciated if anybody with the same issue experiences could help solve this issue.

Best regards,

Hakchul Lee


#2

Related: https://github.com/OpenDroneMap/OpenDroneMap/issues/895


#3

@flyfuture are you able to process the dataset if you lower the --max-concurrency parameter?


#4

Do you also get these lines presented??

  • O

[INFO] Found 25 usable images
[DEBUG] running /code/build/bin/odm_extract_utm -imagesPath /var/www/data/30178ac3-546b-42d3-9b7d-5eb67caa705b/images/ -imageListFile /var/www/data/30178ac3-546b-42d3-9b7d-5eb67caa705b/img_list.txt -outputCoordFile /var/www/data/30178ac3-546b-42d3-9b7d-5eb67caa705b/odm_georeferencing/coords.txt -logFile /var/www/data/30178ac3-546b-42d3-9b7d-5eb67caa705b/odm_georeferencing/odm_georeferencing_utm_log.txt
Warning: Directory Photo has an unexpected next pointer; ignored.
Warning: Directory Iop has an unexpected next pointer; ignored.
Warning: Directory GPSInfo has an unexpected next pointer; ignored.


#5

Thanks for the reference link.

I didn’t get those lines presented on task output. I got these lines presented on the last part of the task output:

==============================================================
2018-09-21 09:47:58,676 DEBUG: Full matching 5760 / 5835, time: 3.657777071s
[DEBUG] running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm create_tracks /var/www/data/25b7afc5-37f1-4005-9a11-8571577e5ad7/opensfm
2018-09-21 09:47:59,515 INFO: reading features
2018-09-21 09:48:13,556 DEBUG: Merging features onto tracks
2018-09-21 09:48:40,498 DEBUG: Good tracks: 384071
[DEBUG] running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm reconstruct /var/www/data/25b7afc5-37f1-4005-9a11-8571577e5ad7/opensfm
2018-09-21 09:49:55,495 INFO: Starting incremental reconstruction
Traceback (most recent call last):
File “/code/SuperBuild/src/opensfm/bin/opensfm”, line 34, in
command.run(args)
File “/code/SuperBuild/src/opensfm/opensfm/commands/reconstruct.py”, line 21, in run
report = reconstruction.incremental_reconstruction(data)
File “/code/SuperBuild/src/opensfm/opensfm/reconstruction.py”, line 1161, in incremental_reconstruction
pairs = compute_image_pairs(common_tracks, data)
File “/code/SuperBuild/src/opensfm/opensfm/reconstruction.py”, line 416, in compute_image_pairs
result = parallel_map(_compute_pair_reconstructability, args, processes)
File “/code/SuperBuild/src/opensfm/opensfm/context.py”, line 38, in parallel_map
return list(e.map(func, args))
File “/usr/local/lib/python2.7/dist-packages/loky/process_executor.py”, line 788, in _chain_from_iterable_of_lists
for element in iterable:
File “/usr/local/lib/python2.7/dist-packages/loky/_base.py”, line 589, in result_iterator
yield future.result()
File “/usr/local/lib/python2.7/dist-packages/loky/_base.py”, line 433, in result
return self.__get_result()
File “/usr/local/lib/python2.7/dist-packages/loky/_base.py”, line 381, in __get_result
raise self._exception
loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.

Traceback (most recent call last):
File “/code/run.py”, line 47, in
plasm.execute(niter=1)
File “/code/scripts/run_opensfm.py”, line 133, in process
(context.pyopencv_path, context.opensfm_path, tree.opensfm))
File “/code/opendm/system.py”, line 34, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

After lowering --max-concurrency parameter from 2 to 1, processing the dataset exited with code 1 and produced the following last part of task output different from the above task output:

================================================================
2018-09-26 07:49:04,732 DEBUG: Full matching 5741 / 5806, time: 3.9042699337s
[DEBUG] running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm create_tracks /var/www/data/c00f196a-d6f0-42e3-9271-b49348a19933/opensfm
2018-09-26 07:49:05,581 INFO: reading features
2018-09-26 07:49:20,161 DEBUG: Merging features onto tracks
2018-09-26 07:49:49,459 DEBUG: Good tracks: 384427
[DEBUG] running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm reconstruct /var/www/data/c00f196a-d6f0-42e3-9271-b49348a19933/opensfm
2018-09-26 07:51:07,011 INFO: Starting incremental reconstruction
Illegal instruction
Traceback (most recent call last):
File “/code/run.py”, line 47, in <module> plasm.execute(niter=1)
File “/code/scripts/run_opensfm.py”, line 133, in process (context.pyopencv_path, context.opensfm_path, tree.opensfm))
File “/code/opendm/system.py”, line 34, in run raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 132

My workstation running this WebODM is a computer with a 64-bit CPU though.


#6

max-con


#7

Ok this helps a lot. It’s not a memory issue, but a compilation issue.

Keep an eye on for updates. https://github.com/OpenDroneMap/OpenDroneMap/issues/895


#8

Thanks @pierotofy for the amazing work being done!! Truly appreciated!


#9

Ok, please try to update WebODM via ./webodm.sh update and reprocess the dataset. Is the problem still happening?


#10

Thanks a lot @pierotofy ! Reprocessing the dataset on updated WebODM was successful without any problem. It was the first success.