Process exited with code 1 - Exception: Child returned 139

WebODM
#1

Hello,

We have are running the latest docker versions images (as of 28/02/2020) on a workstation that is configured with 300GB of RAM, 64 cores. Project had 408 images of 20mpix each

[INFO]    Finished odm_dem stage
[INFO]    Running odm_orthophoto stage
[INFO]    running /code/build/bin/odm_orthophoto -inputFiles /var/www/data/09eb361a-62c2-4eee-92ea-d4fab595962d/odm_texturing_25d/odm_textured_model_geo.obj -logFile /var/www/data/09eb361a-62c2-4eee-92ea-d4fab595962d/odm_orthophoto/odm_orthophoto_log.txt -outputFile /var/www/data/09eb361a-62c2-4eee-92ea-d4fab595962d/odm_orthophoto/odm_orthophoto_render.tif -resolution 100.0  -outputCornerFile /var/www/data/09eb361a-62c2-4eee-92ea-d4fab595962d/odm_orthophoto/odm_orthophoto_corners.txt
Segmentation fault (core dumped)
Traceback (most recent call last):
File "/code/run.py", line 57, in <module>
app.execute()
File "/code/stages/odm_app.py", line 92, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 445, in run
self.process(self.args, outputs)
File "/code/stages/odm_orthophoto.py", line 89, in process
'-outputCornerFile {corners} {bands}'.format(**kwargs))
File "/code/opendm/system.py", line 76, in run
raise Exception("Child returned {}".format(retcode))
Exception: Child returned 139
#2

Smaller project of 343 images:

[INFO]    Finished odm_dem stage
[INFO]    Running odm_orthophoto stage
[INFO]    running /code/build/bin/odm_orthophoto -inputFiles /var/www/data/c157ef9f-67a8-4607-88c3-a1026059988b/odm_texturing_25d/odm_textured_model_geo.obj -logFile /var/www/data/c157ef9f-67a8-4607-88c3-a1026059988b/odm_orthophoto/odm_orthophoto_log.txt -outputFile /var/www/data/c157ef9f-67a8-4607-88c3-a1026059988b/odm_orthophoto/odm_orthophoto_render.tif -resolution 50.0  -outputCornerFile /var/www/data/c157ef9f-67a8-4607-88c3-a1026059988b/odm_orthophoto/odm_orthophoto_corners.txt
Couldn't allocate enough memory to render the orthophoto (55497x53712 cells = 11923419456 bytes). Try to increase the --orthophoto-resolution parameter to a larger integer or add more RAM.
Traceback (most recent call last):
File "/code/run.py", line 57, in <module>
app.execute()
File "/code/stages/odm_app.py", line 92, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 464, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 445, in run
self.process(self.args, outputs)
File "/code/stages/odm_orthophoto.py", line 89, in process
'-outputCornerFile {corners} {bands}'.format(**kwargs))
File "/code/opendm/system.py", line 76, in run
raise Exception("Child returned {}".format(retcode))
Exception: Child returned 1
#3

It’s very strange.
Last one
“Couldn’t allocate enough memory to render the orthophoto (55497x53712 cells = 11923419456 bytes). Try to increase the --orthophoto-resolution parameter to a larger integer or add more RAM “

Can you post a graph or statistics of RAM usage during the processing.

I have faced a problem , that if Evan had enough RAM and SWAP Linux kernel does not allocate needed.

OR
You can increase the --orthophoto-resolution parameter to a larger integer.

Can you post initial parameters?

#4

It is running through WebODM with a customised preset:

[{"name": "ignore-gsd", "value": true}, {"name": "dsm", "value": true}, {"name": "depthmap-resolution", "value": "1000"}, {"name": "dem-resolution", "value": "2.0"}, {"name": "orthophoto-resolution", "value": "1.0"}]

Yeah, given that the amount of RAM that it is asking for is 12GB, and it has 300GB (it can expand up to 700GB as the system requires) available, it seems odd. Swap has been disabled.

Do you have any specific tools that you recommend for stats recording? I usually use glances for realtime monitoring, but that doesn’t seem to record anything.

I am also re-running one that I know worked, with only 172 images.

#5

I got both data-sets to process by increasing the orthophoto-resolution to 5, now I am going to re-run them and get smaller until I see where it breaks.

1 Like
#6

Yes, --ignore-gsd is a great but dangerous flag: you can run out of memory at an exponential rate.

1 Like
#7

Is the effect of --ignore-gsd that it processes at the actual resolution that the GSD should be. i.e.if we targeted a flight height of X to get a GSD of say 3cm, it will process at 3cm without us having to tell the software that we want a GSD of 3cm?

#8

orthophoto-resolution dictates the target resolution, but resolution is capped at the GSD estimate (if you fly to get 3 cm/px and ask for 1 cm/px, you will get 3 cm/px), unless --ignore-gsd is used, in which case you will get 1 cm/px outputs.

Setting orthophoto-resolution to a very low value will always attempt to give you the highest resolution possible for the flight based on GSD.

–ignore-gsd also turns off certain memory and speed optimizations, which can produce slightly sharper results, but at the expense of higher memory usage and slower processing.

3 Likes
#9

Great, then I can set our internal default to be 1cm and known that it will always process at the highest resolution possible!

Alright, so I have got it down to 3.5cm and now I am stuck at:

Found point cloud: odm_georeferencing/odm_georeferenced_model.laz

Entwine point cloud is already built.

Postprocessing: done (•̀ᴗ•́)و!

Compressing all.zip

It has been at this step now for about 24 hours.

Which is this one: Stuck on Compressing all.zip