"Exception: Child returned 139" when "Running georeferencing with OpenSfM transformation matrix"

srogerio · 2018-10-25 02:37:27 UTC

Hi, there!

I’m trying to process a big project with 4890 images (still more to come. I estimate +1000 images to complete this job) with ODM Native running on a Droplet (Digital Ocean) with 192GB RAM and 32 vCPUs + 1.8TB volume, Ubuntu 16.04.

The last issue I’ve stumble on when ODM starts to process “OpenSfM transformation matrix”, with the following ouput:

[INFO] Running ODM Texturing Cell - Finished
[INFO] Running ODM Georeferencing Cell
[INFO] Running georeferencing with OpenSfM transformation matrix
[DEBUG] running /code/build/bin/odm_georef -bundleFile /home/webodm/proj_fazendaOmega/opensfm/bundle_r000.out -inputTransformFile /home/webodm/proj_fazendaOmega/opensfm/geocoords_transformation.txt -inputCoordFile /home/webodm/proj_fazendaOmega/odm_georeferencing/coords.txt -inputFile /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model.obj -outputFile /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /home/webodm/proj_fazendaOmega/smvs/smvs_dense_point_cloud.ply -outputPointCloudFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferenced_model.ply -logFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_log.txt -outputTransformFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_transform.txt -georefFileOutputPath /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_model_geo.txt
Segmentation fault (core dumped)
Traceback (most recent call last):
File “/code/run.py”, line 47, in
plasm.execute(niter=1)
File “/code/scripts/odm_georeferencing.py”, line 127, in process
‘-logFile {log} -outputTransformFile {transform_file} -georefFileOutputPath {geo_sys}’.format(**kwargs))
File “/code/opendm/system.py”, line 34, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 139

Here is the command I used, with the only option to start-with odm_georeferencing, since the first part took almost 48 hours to run…:
python /code/run.py proj_fazendaOmega --start-with odm_georeferencing 2>&1 | tee ./proj_fazendaOmega/log-24out2018.txt

The process just begins to write on files odm_georeferencing_transform.txt, odm_geocoords_transformation.txt (just 4 lines) and quits with the message above. The Droplet have lots of disk space.
The process apparently is able to complete writing the file coords.txt, with 4892 lines (4890 ref to the images + 2 headlines, I think) then stops when begining to write those two txt files.
Strangely, the last values are zeroes:
odm_georeferencing_transform.txt:
|[ 1.0004297,|0.0114977,|0.0000000,|769106.5625155 ]|
|[ -0.0114977,|1.0004297,|0.0000000,|-1685242.3864058 ]|
|[ 0.0000001,|0.0000001,|1.0000000,|-0.0000000 ]|
|[ 0.0000000,|0.0000000,|0.0000000,|1.0000000 ]|

odm_geocoords_transformation.txt:
1.0004296840634197 0.011497707571834326 0.0 769106.5625155289
-0.01149771036580205 1.0004296808037907 2.3283064365386963e-10 -1685242.3864057753
7.82310962677002e-08 8.009374141693115e-08 1.0 -1.862645149230957e-09
0.0 0.0 0.0 1.0

The coords.txt file, on the other side, has no zero values.
I’m not sure those figures means something, but I’m running crazy trying to find what is going on…

Does anyone know what is the meaning of the “139” error or what to do on this situation?

Any help is more than welcome!
Thanks a lot!

pierotofy · 2018-10-25 16:14:04 UTC

139 is the error code for SIGSEGV (segmentation fault). This error can occur when a value has been written on a buffer past the memory allocated for it (among many other possible causes).

If you can upload these files somewhere:

opensfm/bundle_r000.out
opensfm/geocoords_transformation.txt
odm_georeferencing/coords.txt
odm_texturing/odm_textured_model.obj
smvs/smvs_dense_point_cloud.ply

We could try to reproduce the issue and debug it.

srogerio · 2018-10-25 18:37:54 UTC

Hello, @pierotofy! Thanks for your help!!! again…

Here are the links for the files you requested:
http://it4.com.br/odm-temp/bundle_r000.out
http://it4.com.br/odm-temp/geocoords_transformation.txt
http://it4.com.br/odm-temp/coords.txt
http://it4.com.br/odm-temp/odm_textured_model.obj
http://it4.com.br/odm-temp/smvs_dense_point_cloud.ply

I really hope you can find something on those. I am stuck on this.

Thanks again.
Best regards.
Roger.

pierotofy · 2018-10-26 02:16:39 UTC

Hey @srogerio could you double check the smvs_dense_point_cloud.ply file? It looks like it might have been corrupted during upload (I cannot open it in Meshlab).

srogerio · 2018-10-26 03:24:18 UTC

Hi, @pierotofy.
It’s a huge file.
Please find it at:
https://it4systems-1.nyc3.cdn.digitaloceanspaces.com/odm-tmp/smvs_dense_point_cloud.ply
24.3GB

pierotofy · 2018-10-26 16:28:00 UTC

That is a pretty big file

I won’t be able to run it on my machine (not enough RAM).

I did run the program with a mock point cloud and the rest of the files, and odm_georef worked properly. I suspect the size (or more likely, the number of points) is what’s causing the problem.

Could you run /code/SuperBuild/build/pdal/bin/pdal info /home/webodm/proj_fazendaOmega/smvs/smvs_dense_point_cloud.ply and paste the output? (Replace the /code path to wherever you’ve installed ODM natively)

srogerio · 2018-10-26 21:11:48 UTC

Hi! Thank you for your interest and help!
Here is the file:
https://it4.com.br/odm-temp/pierotofy.txt

Best of luck!

pierotofy · 2018-10-27 06:33:21 UTC

Thanks @srogerio. It’s very likely this is an overflow problem in the PCL library. The number of points in this point cloud is only about 3x smaller than the largest value for a signed int32. This is obviously a problem and should be handled.

If you pass --depthmap-resolution 320 (which should lower the number of points) does processing continue?

srogerio · 2018-10-27 23:31:05 UTC

Hi! Thanks again.
I’ve tried your suggestion but, unfortunately, it didn’t work…

$ python /code/run.py proj_fazendaOmega --start-with odm_georeferencing --depthmap-resolution 320 2>&1 | tee ./proj_fazendaOmega/log-27out2018.txt

(is it correct??)
The result, after loading the images:
…
INFO] Found 4890 usable images
[DEBUG] running /code/build/bin/odm_extract_utm -imagesPath /home/webodm/proj_fazendaOmega/images/ -imageListFile /home/webodm/proj_fazendaOmega/img_list.txt -outputCoordFile /home/webodm/proj_fazendaOmega/odm_georeferencing/coords.txt -logFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_utm_log.txt
[DEBUG] Line: WGS84 UTM 22S
[INFO] Running ODM Load Dataset Cell - Finished
[INFO] Running ODM OpenSfM Cell
[WARNING] Found a valid OpenSfM reconstruction file in: /home/webodm/proj_fazendaOmega/opensfm/reconstruction.json
[WARNING] Found a valid Bundler file in: /home/webodm/proj_fazendaOmega/opensfm/reconstruction.json
[DEBUG] running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm export_geocoords /home/webodm/proj_fazendaOmega/opensfm --transformation --proj '+units=m +no_defs=True +datum=WGS84 +proj=utm +zone=22 ’
[INFO] Running ODM OpenSfM Cell - Finished
[INFO] Running SMVS Cell
[WARNING] Found a valid SMVS reconstruction file in: /home/webodm/proj_fazendaOmega/smvs/smvs_dense_point_cloud.ply
[INFO] Running ODM SMVS Cell - Finished
[INFO] Running ODM Meshing Cell
[WARNING] Found a valid ODM Mesh file in: /home/webodm/proj_fazendaOmega/odm_meshing/odm_mesh.ply
[WARNING] Found a valid ODM 2.5D Mesh file in: /home/webodm/proj_fazendaOmega/odm_meshing/odm_25dmesh.ply
[INFO] Running ODM Meshing Cell - Finished
[INFO] Running MVS Texturing Cell
[WARNING] Found a valid ODM Texture file in: /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model.obj
[WARNING] Found a valid ODM Texture file in: /home/webodm/proj_fazendaOmega/odm_texturing_25d/odm_textured_model.obj
[INFO] Running ODM Texturing Cell - Finished
[INFO] Running ODM Georeferencing Cell
[INFO] Running georeferencing with OpenSfM transformation matrix
[DEBUG] running /code/build/bin/odm_georef -bundleFile /home/webodm/proj_fazendaOmega/opensfm/bundle_r000.out -inputTransformFile /home/webodm/proj_fazendaOmega/opensfm/geocoords_transformation.txt -inputCoordFile /home/webodm/proj_fazendaOmega/odm_georeferencing/coords.txt -inputFile /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model.obj -outputFile /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /home/webodm/proj_fazendaOmega/smvs/smvs_dense_point_cloud.ply -outputPointCloudFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferenced_model.ply -logFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_log.txt -outputTransformFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_transform.txt -georefFileOutputPath /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_model_geo.txt
Segmentation fault (core dumped)
Traceback (most recent call last):
File “/code/run.py”, line 47, in
plasm.execute(niter=1)
File “/code/scripts/odm_georeferencing.py”, line 127, in process
‘-logFile {log} -outputTransformFile {transform_file} -georefFileOutputPath {geo_sys}’.format(**kwargs))
File “/code/opendm/system.py”, line 34, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 139

pierotofy · 2018-10-30 15:25:03 UTC

Mm, what if you use --use-opensfm-dense?

srogerio · 2018-10-31 01:15:51 UTC

Hi! Same result, as you can see below.
Do you think that the “Splitting Large Datasets” method could solve it?

2018-10-30 21:13:28,550 INFO: Pruning depthmap for image DJI_30278.JPG
2018-10-30 21:13:29,290 INFO: Pruning depthmap for image DJI_60016.JPG
2018-10-30 21:13:37,514 INFO: Merging depthmaps
[WARNING] Found a valid Bundler file in: /home/webodm/proj_fazendaOmega/opensfm/reconstruction.json
[DEBUG] running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm export_geocoords /home/webodm/proj_fazendaOmega/opensfm --transformation --proj '+units=m +no_defs=True +datum=WGS84 +proj=utm +zone=22 ’
[INFO] Running ODM OpenSfM Cell - Finished
[INFO] Running ODM Meshing Cell
[WARNING] Found a valid ODM Mesh file in: /home/webodm/proj_fazendaOmega/odm_meshing/odm_mesh.ply
[WARNING] Found a valid ODM 2.5D Mesh file in: /home/webodm/proj_fazendaOmega/odm_meshing/odm_25dmesh.ply
[INFO] Running ODM Meshing Cell - Finished
[INFO] Running MVS Texturing Cell
[WARNING] Found a valid ODM Texture file in: /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model.obj
[WARNING] Found a valid ODM Texture file in: /home/webodm/proj_fazendaOmega/odm_texturing_25d/odm_textured_model.obj
[INFO] Running ODM Texturing Cell - Finished
[INFO] Running ODM Georeferencing Cell
[INFO] Running georeferencing with OpenSfM transformation matrix
[DEBUG] running /code/build/bin/odm_georef -bundleFile /home/webodm/proj_fazendaOmega/opensfm/bundle_r000.out -inputTransformFile /home/webodm/proj_fazendaOmega/opensfm/geocoords_transformation.txt -inputCoordFile /home/webodm/proj_fazendaOmega/odm_georeferencing/coords.txt -inputFile /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model.obj -outputFile /home/webodm/proj_fazendaOmega/odm_texturing/odm_textured_model_geo.obj -inputPointCloudFile /home/webodm/proj_fazendaOmega/opensfm/depthmaps/merged.ply -outputPointCloudFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferenced_model.ply -logFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_log.txt -outputTransformFile /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_transform.txt -georefFileOutputPath /home/webodm/proj_fazendaOmega/odm_georeferencing/odm_georeferencing_model_geo.txt
Segmentation fault (core dumped)
Traceback (most recent call last):
File “/code/run.py”, line 47, in
plasm.execute(niter=1)
File “/code/scripts/odm_georeferencing.py”, line 127, in process
‘-logFile {log} -outputTransformFile {transform_file} -georefFileOutputPath {geo_sys}’.format(**kwargs))
File “/code/opendm/system.py”, line 34, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 139

smathermather-cm · 2018-10-31 11:46:23 UTC

Yes, I’d say this is a good candidate for split merge.

pierotofy · 2018-11-02 19:13:21 UTC

I’m out of ideas. Perhaps something went wrong when you compiled the odm_georef program. I would try to use the docker images to see if those work.