Floating point exception (Texturing phase)

nhavas · 2 June 2021 09:44

Hi All,

I run into an error processing a fairly large dataset (793 images).

Options: build-overviews: true, crop: 1, dem-resolution: 2.0, fast-orthophoto: true, orthophoto-resolution: 0.51, pc-quality: high, skip-3dmodel: true, tiles: true, use-exif: true, verbose: true, rerun-from: odm_meshing

Environment:
Ubuntu 18.04.5 LTS (GNU/Linux 5.4.0-1048-aws x86_64)
AWS c5.9xlarge, 36 cores, 72Gb memory, 400Gb SWAP
files are on S3 mapped local with goofys

Dataset (4.1Gb): Dropbox - webODM-dataset - Simplify your life

Entry in syslog:
Jun 2 09:24:05 ip-172-31-2-193 kernel: [ 1292.270081] traps: texrecon[5650] trap divide error ip:5651e35bc12e sp:7ffcda556550 error:0 in texrecon[5651e351d000+175000]

Any ideas?

Thanks,
n.

Console (full consol log is in Dropbox, did not fit here):

399921 faces have not been seen
Took: 88.678s
Generating texture patches:
Running…
done. (Took 2.92s)
0 texture patches.
Running global seam leveling:
Create matrices for optimization…
Floating point exception (core dumped)
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 136
===== Done, human-readable information to follow… =====

[ERROR] The program exited with a strange error code. Please report it at https://community.opendronemap.org
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
[Previous line repeated 3 more times]
File “/code/opendm/types.py”, line 319, in run
self.process(self.args, outputs)
File “/code/stages/mvstex.py”, line 108, in process
system.run(’"{bin}" “{nvm_file}” “{model}” “{out_dir}” ’
File “/code/opendm/system.py”, line 90, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 136
100 - done.

nhavas · 2 June 2021 10:03

UPDATE:

After switching off global seam leveling (texturing-skip-global-seam-leveling: true) run into another fail:
Unsupported bit depth value: -1

Full console in DropBox (console2.txt)
Rendering the ortho photo…
… model rendered
Writing ortho photo to /var/www/data/1f22a895-893f-44fe-b126-b54e324a4f95/odm_orthophoto/odm_orthophoto_render.tif
Unsupported bit depth value: -1
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.4.7 documentation
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
[Previous line repeated 6 more times]
File “/code/opendm/types.py”, line 319, in run
self.process(self.args, outputs)
File “/code/stages/odm_orthophoto.py”, line 73, in process
system.run(’"{odm_ortho_bin}" -inputFiles {models} ’
File “/code/opendm/system.py”, line 90, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

Saijin_Naib · 2 June 2021 11:28

Welcome!

Sorry you’re having difficulty.

Are you using the absolute latest/current ODM release? We should be on v2.5.1 at the moment, but your log is referring to v2.4.7

nhavas · 2 June 2021 11:56

Oh. Thank you, let me update and retry. Will get back here with the news.
I appreciate your reply!

nhavas · 7 June 2021 08:32

We have updated to the latest version, still get the same error when resolution is set to 0,51 or 0,9mm/px

Saijin_Naib · 7 June 2021 16:58

Oooh, I think we ran into a fun internationalization issue!

Are you passing the resolution exactly as you’ve typed here on the forum, with the comma as the decimal separator?

On this side of the pond, we use/expect a period. Can you try passing values like 0.51 or 0.9 instead of 0,51 and 0,9

This may have to be something we address. I’ll try to get back to you on that.

pierotofy · 7 June 2021 17:25

Very strange error. It comes from odm_orthophoto/OdmOrthoPhoto.cpp at 5567fd1d238ca0a57126d519830947ceff954a91 · OpenDroneMap/odm_orthophoto · GitHub which suggests some possible error in reading the input texture files. Not sure of the root cause.

nhavas · 8 June 2021 06:15

We have used dot as decimal separator when set the resolution as it should be. See my first post, when I pasted the options. When I wrote the comment this was just muscle memory.
Thank you for keep keep posting hints!

nhavas · 8 June 2021 06:17

I get this when global seam leveling is OFF. When it is ON we get a floating point exception.

nhavas · 9 June 2021 15:11

I get this same error with another dataset and driving me crazy. Do you guys have any clue how can I get through this?

Saijin_Naib · 9 June 2021 15:35

Have you updated again to v2.5.3 and/or v.2.5.4? Any changes/improvements?

nhavas · 9 June 2021 17:03

Yes, we are on the latest version.

Saijin_Naib · 10 June 2021 20:10

Could you give us a better idea of your flight plans?

Overlap, sidelap, and height above ground?

Looking at the dataset you linked above, it is a pretty difficult dataset since each image is so homogenous both within itself and between other frames.

Saijin_Naib · 11 June 2021 12:30

I’ve employed a work-around I’ve used in the past a few times, and I was able to process your dataset (mostly) successfully!

The images were pretty washed out, low sharpness, and had very low scene detail.

So, I pre-processed them lightly in XNConvert with an Enhance Detail and Auto-Contrast filter.

Original Image:

Pre-Processed Image:

The difference is subtle, but it is enough to really help during the matching phase.

Processing using pretty much default settings worked great:
Options: crop: 0, debug: true, dsm: true, verbose: true

Orthophoto:

DSM:

Point Cloud:

The Point Cloud is where we see some of the limitations of this dataset, this pre-processing workflow, and my settings come together.

The strong “waffling” is just a consistent lack of tiepoints extracted during matching. Maybe we could do significantly better by forcing the min-num-features up, or by cranking up pc-quality, feature-quality, and turning off resizing.

To that end, Piero is tracking a feature request for OpenDroneMap to do this pre-processing in the background transparently for the user just for the matching phase, and then pass the original/unprocessed images to the pipeline for point-cloud coloration, orthophoto generation, and model texturing.

nhavas · 14 June 2021 17:01

Hi,

Thank you for your post, I truly appreciate you put this issue on your agenda. We have had success generating the orthomosaic with the default settings problems start to occure when we go high-res and define the resolution. It would be useful to see if these enhancements help with the following settings:
dem-resolution: 0.9, dsm: true, ignore-gsd: true, orthophoto-resolution: 0.9, pc-quality: high

When we try using them we run into a an error: empty initial point-cloud followed by a opendm.system.SubprocessException: Child returned 1

Saijin_Naib · 14 June 2021 17:15

Interesting! I was getting a failure to stitch with the settings above (Options: crop: 0, debug: true, dsm: true, verbose: true) until I pre-processed the images.

I can try again with Lightning with the same pre-processing I did before and the settings you included above. I’ll update you in time when it finishes.

Trying locally:

nhavas · 14 June 2021 17:27

Amazing, thank you!

nhavas · 14 June 2021 18:42

What was the error? Empty initial point-cloud?

Saijin_Naib · 14 June 2021 21:32

If I recall correctly, yes.

At the moment, I don’t think I can reproduce. My system runs out of RAM almost immediately during feature extraction at the settings you’ve indicated (32GB RAM, 1GB SWAP).

I can’t reproduce on Lightning as the values for ortho and dem resolution are clamped, along with ignore-gsd not being valid.

nhavas · 14 June 2021 21:45

Thank you for the update. I have registered on Lightning to give it a try myself. Also have made the suggested conversion on the images and running on my installation. Will get back to you on the results. Even if the image quality is not the best these errors are disturbing. I have run the processing on Pix4D and got the desired outcome in 20 mins. It seems it that can deal with the image quality.