Abort during local seam leveling

Lars · 2 November 2021 04:12

Update:
The recommendations worked!
I increased --min-num-features (from 8000 to 16000) and --matcher-neighbors (from 8 to 16) and the mapping did not abort!
Thanks, Saijin!!!

Now I try it with –resize-to -1 for better resolution
and with –mesh-octree-depth 14 to decrease the glitches at the vegetation.

Saijin_Naib · 2 November 2021 13:28

Sounds like you’re well on your way!

Please post what you get if you’d like. We love seeing what folks are doing!

Lars · 13 November 2021 07:04

Unfortunatley it crashed again (I was on leave for a week so I could not follow up earlier).

I run the same dataset but this time without resizing (using 45MP photos)
(–dsm --dtm --dem-resolution 4 --resize-to -1 --orthophoto-kmz --mesh-octree-depth 14 --min-num-features 16000 --matcher-neighbors 16 )

Here is the last bit of the log. Do you (or anybody) have an idea what went wrong?

Filtering /datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/parts/part_226.ply (statistical, meanK 16, standard deviation 2.5)
[INFO]    running pdal pipeline -i /tmp/tmpr7asunuw.json > /dev/null 2>&1
[INFO]    Filtering /datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/parts/part_478.ply (statistical, meanK 16, standard deviation 2.5)
[INFO]    Filtering /datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/parts/part_264.ply (statistical, meanK 16, standard deviation 2.5)
[INFO]    running pdal pipeline -i /tmp/tmp2_7c23bn.json > /dev/null 2>&1
[INFO]    running pdal pipeline -i /tmp/tmpw327acs1.json > /dev/null 2>&1
[INFO]    Merging 541 point cloud chunks to /datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/point_cloud.ply
[INFO]    Finished odm_filterpoints stage
[INFO]    Running odm_meshing stage
[INFO]    Writing ODM Mesh file in: /datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.ply
[INFO]    running "/code/SuperBuild/install/bin/PoissonRecon" --in "/datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/point_cloud.ply" --out "/datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.dirty.ply" --depth 14 --pointWeight 4.0 --samplesPerNode 1.0 --threads 31 --maxMemory 185 --bType 2 --linearFit 
Segmentation fault (core dumped)
[WARNING] Child returned 139
[WARNING] PoissonRecon failed with 15 threads, let's retry with 7...
[INFO]    running "/code/SuperBuild/install/bin/PoissonRecon" --in "/datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/point_cloud.ply" --out "/datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.dirty.ply" --depth 14 --pointWeight 4.0 --samplesPerNode 1.0 --threads 15 --maxMemory 185 --bType 2 --linearFit 
Segmentation fault (core dumped)
[WARNING] Child returned 139
[WARNING] PoissonRecon failed with 7 threads, let's retry with 3...
[INFO]    running "/code/SuperBuild/install/bin/PoissonRecon" --in "/datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/point_cloud.ply" --out "/datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.dirty.ply" --depth 14 --pointWeight 4.0 --samplesPerNode 1.0 --threads 7 --maxMemory 185 --bType 2 --linearFit 
Segmentation fault (core dumped)
[WARNING] Child returned 139
[WARNING] PoissonRecon failed with 3 threads, let's retry with 1...
[INFO]    running "/code/SuperBuild/install/bin/PoissonRecon" --in "/datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/point_cloud.ply" --out "/datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.dirty.ply" --depth 14 --pointWeight 4.0 --samplesPerNode 1.0 --threads 3 --maxMemory 185 --bType 2 --linearFit 
Segmentation fault (core dumped)
[WARNING] Child returned 139
[WARNING] PoissonRecon failed with 1 threads, let's retry with 0...
[INFO]    running "/code/SuperBuild/install/bin/PoissonRecon" --in "/datasets/Tropical_Regen_Block1_basemap_highres/odm_filterpoints/point_cloud.ply" --out "/datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.dirty.ply" --depth 14 --pointWeight 4.0 --samplesPerNode 1.0 --threads 1 --maxMemory 185 --bType 2 --linearFit 
Segmentation fault (core dumped)
[WARNING] Child returned 139
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/ReconstructMesh" -i "/datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.dirty.ply" -o "/datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.ply" --remove-spikes 0 --remove-spurious 20 --smooth 0 --target-face-num 400000 
14:36:15 [App     ] Build date: Oct 13 2021, 21:19:36
14:36:15 [App     ] CPU: Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz (32 cores)
14:36:15 [App     ] RAM: 251.83GB Physical Memory 29.92GB Virtual Memory
14:36:15 [App     ] OS: Linux 5.11.0-37-generic (x86_64)
14:36:15 [App     ] SSE & AVX compatible CPU & OS detected
14:36:15 [App     ] Command line: -i /datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.dirty.ply -o /datasets/Tropical_Regen_Block1_basemap_highres/odm_meshing/odm_mesh.ply --remove-spikes 0 --remove-spurious 20 --smooth 0 --target-face-num 400000

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 94, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  [Previous line repeated 3 more times]
  File "/code/opendm/types.py", line 328, in run
    self.process(self.args, outputs)
  File "/code/stages/odm_meshing.py", line 24, in process
    mesh.screened_poisson_reconstruction(tree.filtered_point_cloud,
  File "/code/opendm/mesh.py", line 207, in screened_poisson_reconstruction
    system.run('"{reconstructmesh}" -i "{infile}" '
  File "/code/opendm/system.py", line 106, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

Andreas_Olsson · 13 November 2021 07:19

Avoid ignore-GSD? Can it make things bad?

Saijin_Naib · 13 November 2021 13:32

Looks like Out-Of-Memory again. Maybe drop mesh octree to 13?

israelbar · 14 November 2021 16:40

It seems --ignore-gsd causes a higher memory usage.

The danger with --ignore-gsd is that it throws away some memory optimization and other goodies, so you may need more RAM, if your dataset is sufficiently large.

Saijin_Naib · 14 November 2021 16:47

Massively more. It’s pretty drastic, to be honest.

Andreas_Olsson · 14 November 2021 17:08

But ignoring GSD is better for urban settings right or when there’s just many hight differences?

The strange thing is that during the matching and dense reconstruction ram isn’t a problem, but at the end there’s a huge spike in ram use. What is happening at the end that makes the use of ram spike? Maybe that part needs to be optimised.

Saijin_Naib · 14 November 2021 17:53

I don’t think it’d be as important for a regular urban scene as you may think.

Last few steps are point cloud filtering and composition, and model meshing and texture mapping.

Andreas_Olsson · 14 November 2021 17:58

I have that book on ODM, as it is explain it looks like it has a huge impact on areas with big differences in hight.

I’ve set the Ortophoto and DSM resolution to 20mm, will unchecking ignore-GSD change anything? Does ignore-GSD affect the pointcloud?

smathermather · 14 November 2021 18:10

The book is fantastic, but was written prior to lots of massive changes. I would test to verify you need ignore-gsd. Almost every memory failure we see is due to using this flag, which throws away most of the memory optimization in ODM, often with (IMO) little improvement.

Andreas_Olsson · 14 November 2021 18:19

So maybe checking that box is the source of a lot of problems I’ve been experiencing.

Saijin_Naib · 14 November 2021 18:21

Yes, as I’ve been trying to convince folks of

My next step is to make it ask you to confirm like, 30 times before it lets you use it

Andreas_Olsson · 14 November 2021 18:24

But does it affect the pointcloud?

Saijin_Naib · 14 November 2021 18:32

It can, but I don’t think it is as important for the point cloud as other, less problematic settings are.

ODM does a pretty fantastic job of accurate GSD estimation, so the onus of tuning points mostly falls on --pc-quality, --pc-filter, --pc-sample.

–ignore-gsd is really only useful in certain edge cases, and even then carries a ton of baggage that makes it often not the right or optimal choice.

Sort of like using a cannon to scratch an itch on your leg, if you get me.

Andreas_Olsson · 14 November 2021 18:38

I’ve taken drone photos with a GSD of 10mm to get good point clouds. It’s been said that you get an uncertainty of gsd * 2 in x/y and gsd * 3 in hight. So I would not want to make things worse.

Lars · 14 November 2021 23:53

Update:
It worked with mesh-octree-depth set to 13, but the output is low resulution. (as you can see in the uploaded photos)
original:

orthophoto (low res and glitchy):

I tried to go back to set --ignore-gsd and --orthophoto-resolution 1 , but it aborted again. Porbably due to the memory issues discussed here.

I read that without setting --ignore-gsd the resolution of the othrophoto cannot be changed. How can one get to the resoution that are given in the original pictures? The discrepancy is rather large.

Saijin_Naib · 15 November 2021 00:44

What is the GSD being calculated as ?

Lars · 15 November 2021 01:18

I am not sure where to find that info, but looking around, I could find that the odm_orthophoto_log.txt file has written:
-resolution
20.0
From the picture I would think it is a higher resolution than that, maybe 10cm/pixel. THe original photo has 1cm/pixel

Saijin_Naib · 15 November 2021 01:21

If you run the job to completion, the Report.pdf will have the GSD as calculated.