ODM fails due to "Strange values in the reconstruction"

ODM
1

I’m having issues getting a map to build. I’ve used Docker command-line OpenDroneMap multiple times (starting over from scratch twice) and WebODM multiple times (starting over from scratch twice) – with the same result each time. It gets to the postprocessing stage and dies at “Processing stopped because of strange values in the reconstruction,” etc. Have I followed best practices for data acquisition? I’m not sure (the link to “flying” is dead), but I did the best I knew how. I flew my DJI Mavic Air using the latest firmware and software, flying a 2D mapping grid using Pix4Dcapture. I used the default ODM parameters in all cases. I have successfully built a map with this drone and this configuration before, but it’s been a couple years.

Here is a link to my project directory including the images:
https://drive.google.com/drive/folders/1-2WUaCNCbYrKs-oNGGPjQuEb7BwKBIgf?usp=sharing

I put the full dump of the latest output in output.txt. This is the end of it:

[INFO]    running "C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud" "C:\WebODM\resources\app\apps\NodeODM\data\253782e9-5e76-48e3-bfd4-20ef417fd7fd\opensfm\undistorted\openmvs\scene.mvs"  --resolution-level 3 --dense-config-file "C:\WebODM\resources\app\apps\NodeODM\data\253782e9-5e76-48e3-bfd4-20ef417fd7fd\opensfm\undistorted\openmvs\Densify.ini" --max-resolution 4056 --max-threads 12 --number-views-fuse 2 --sub-resolution-levels 2 -w "C:\WebODM\resources\app\apps\NodeODM\data\253782e9-5e76-48e3-bfd4-20ef417fd7fd\opensfm\undistorted\openmvs\depthmaps" -v 0 --cuda-device -1
[INFO]    running "C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\DensifyPointCloud" --filter-point-cloud -20 -i "C:\WebODM\resources\app\apps\NodeODM\data\253782e9-5e76-48e3-bfd4-20ef417fd7fd\opensfm\undistorted\openmvs\scene_dense.mvs" -v 0 --cuda-device -1
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221225477
Traceback (most recent call last):
File "C:\WebODM\resources\app\apps\ODM\stages\odm_app.py", line 81, in execute
self.first_stage.run()
File "C:\WebODM\resources\app\apps\ODM\opendm\types.py", line 398, in run
self.next_stage.run(outputs)
File "C:\WebODM\resources\app\apps\ODM\opendm\types.py", line 398, in run
self.next_stage.run(outputs)
File "C:\WebODM\resources\app\apps\ODM\opendm\types.py", line 398, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File "C:\WebODM\resources\app\apps\ODM\opendm\types.py", line 377, in run
self.process(self.args, outputs)
File "C:\WebODM\resources\app\apps\ODM\stages\openmvs.py", line 227, in process
raise e
File "C:\WebODM\resources\app\apps\ODM\stages\openmvs.py", line 221, in process
system.run('"%s" %s' % (context.omvs_densify_path, ' '.join(config + gpu_config + extra_config)))
File "C:\WebODM\resources\app\apps\ODM\opendm\system.py", line 110, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221225477

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/

I appreciate any help. Thank you.

1 Like
2

I am using Windows 10 Pro (Win32_x64) 10.0.19044, Google Chrome 112.0.5615.140, an Intel Core i7-9750H CPU, 16 GB RAM, an NVIDIA GeForce GTX 1650 GPU, Docker Desktop v4.19.0, and a fair bit of coffee.

2 Likes
3

Looked at some of the pictures. Looks like you are using enough overlap, though some pictures are very blurry. I also use Pix4Dcapture as a flight planner and it generally works great.
Only sometimes when I fly over very contourless grounds (big meadows, forrests) do I run into problems with ODM processing the images.
Might be that you would have to tweak some settings like increasing feature_quality or min_num_features. Maybe set the first to high and the later to something like 24000 or 32000.

When you are not sure your setup works, you could try a known to work dataset from Github

3 Likes
4

Thanks for looking at it. I tried fiddling with the settings you suggested, in several permutations, and though it seemed to last longer (it went longer before it died), it still died. A few updates:

  • I observed the same apparent blurriness in the photos, but I don’t think the photos are actually blurry. Something about the texture of the ground (which has recently been cleared with a bulldozer) makes it appear blurry, but even in the same photos, other features appear sharp.
  • I tried to build the same map with WebODM on a different machine (Windows 11 x64, AMD Ryzen 5 3600, 16 GB RAM, NVIDIA GeForce 1650 SUPER GPU) with the same results.
  • I downloaded and tried a different dataset from Github per your suggestion and it worked fine.

One concerning thing I notice when I look at my project in WebODM: when I click on “View Map”, it tells me “Bounds are not valid” and can’t center on the location. I imported a GeoJSON polygon of the boundary, but despite trying it several times and turning off “auto-boundary,” it didn’t appear to save the setting.

So, a few questions:

  • Why “bounds are not valid”? Does that mean a problem with my drone images, or with my GCP file? My drone images, when I view them in gdalinfo, have the GPS location as part of the EXIF tag but aren’t actually geotagged – should they be? I used GCP Editor Pro to write the GCP file, so I thought that ought to be okay. Is there any way to validate the GCP file?
  • What does “Strange values in the reconstruction” actually mean? It would be helpful if I could understand what has actually gone wrong, if there is anything I can do to fix it, and if there is anything I can do to improve when I try again.
5

Strange values can be that it can’t connect all images, might be because of the blurry images.

6

These photos are taken on a mountaintop. The cemetery at the top of the mountain appears mostly to be in focus. Parts of the surrounding slope appear blurry – but that’s not the part I care about at all. Both of these are crops from the same photo. (It only let me post one photo.) Notice in the first one that the treetop is in focus but the lower ground is not.

DJI_0601_cr
DJI_0601_cr1623×1082 335 KB

7

I tried removing about 50 of the peripheral images and running it again – same thing. :confused: Seriously, I’d like a technical explanation of what is going wrong. What does “strange values” mean, from a technical perspective?

8

I’ve had that with bad overlap or bad images

9

I suspect the technical explanation is the blurring as identified above. Have you tried the settings that shiva recommended?

10

It might be the images, it also might not. I’d try a higher overlap (like 90%) so you get more areas in focus from picture to picture. I have had this issue with tall trees as well where the treetops are in focus but then the ground isn’t. Increasing overlap helped a good deal.

I also say it might not be the images because I’ve had the same set of images successfully create orthos and then also fail with different settings with the same “strange values” error…usually (but not always) on “ultra” settings, which makes me think it could sometimes be a misreported out-of-memory issue (which potentially caused the strange values)

2 Likes
11

I suspect the technical explanation is the blurring as identified above.

Thanks, but that’s not what I meant by a technical explanation. I mean, what goes wrong when there are blurry images? What part of the image processing breaks down? The final failure appears to take place in the dense point-cloud filtering. The error message “something was weird” is frustratingly unhelpful.

Have you tried the settings that shiva recommended?

Oh yes, I did try those settings, in several permutations. Sorry I didn’t mention that.

I finally got it to finish and produce an acceptable map! I discarded all the noticeably blurry images and kept only the ones of the area I really cared about – narrowed the dataset down to 92 images (out of 195 I started with).

Friendship-Road-5-7-2023-orthophoto_cr
Friendship-Road-5-7-2023-orthophoto_cr1920×2065 694 KB

3 Likes
12

Rolling shutter isn’t the only problem with taking an image.

If you fly at 2m/s with a shutter of 1/1000 the drone will move 2mm during every shutter release. But it gets worse when you take rotation into account, a small rotation doesn’t mean much in the centre of the image but a lot out in the edge.

When I’m taking images of flying birds I need a shutter of 1/2000 to get a clean image so imagine what’s needed in photogrammetry.

If you can’t have a high speed shutter then let the drone stop at every shutter release.

I’m always aiming for the highest degree of accuracy though.

1 Like
13

So is the blurriness in these photos due to motion blur (rather than depth of field)? I suppose the treetops would not suffer as much from motion blur because they are closer to the lens? Most of these photos had a shutter speed of between 1/600 and 1/1000 – which is not nearly as fast as I would have hoped since the platform is moving.

14

Things that move can cause a blur depending of GSD vs shutter. A treetop is closer and therefore has a lower GSD than the ground, if the ground is at 10mm GSD the treetop might be at 5mm. And if something that you also want to model is closer you need to increase overlap to get both the ground and the high part we’ll defined.

In pointclouds I do leafs tend to produce void because they move between images. That’s also a good thing wen making pointclouds of roads because moving cars are absent in the pc.

1 Like
15

And choosing a higher iso to get a faster shutter isn’t an option either on a small sensor as the ones in consumer drones as it leads to to much noise which degrades the images to much.

I suggest testing a lot, you can use most cameras.
Phones, actioncameras, slr or other. I’ve actually tried it them all.

I might go into to much detail but anyway…

There’s many common problems that needs to be thought of when choosing a camera for photogrammetry. The sensor, small sensors doesn’t handle low light well. The lens, cheap lenses usually have problems like chromatic aberration, flaring, coma, vignetting and uneven focus. This can affect the processing.

1 Like
16

I appreciate the detail! I would like to learn as much about this as I can.

1 Like
17

Being closer, they would be more blurry, as they move through the FOV more quickly.

Depending on the type of mechanical shutter in use, very short exposures can also suffer from rolling shutter distortion, very similar to an electronic shutter. Short exposures will reduce motion blur though, and the shutter speed required can be determined quite easily, see:

1 Like
18

I guess I need to spend some time reading a general intro to photogrammetry.

Since the “strange values” error was occurring in the dense point cloud filtering – I ran it again completely disabling pc-filtering, and it did run to completion. In my limited application for this one – making a 2D map of a cemetery – it seems not to have affected the result.

19

I feel this frustration deeply. When I started in this field almost 10 years ago, we really didn’t understand at all when and why photogrammetry models failed. I could ask a question at a table full of computer vision PhDs and they would shrug and talk about the mysteries.

We’ve progressed a long way since that time. At this stage, we generally know what works and have fewer failures (ensure sufficient overlap, minimize blurring, ensure having enough views of the elements of interest, etc.), but it’s not uncommon to get a dataset that is a bit mysterious.

This one is a bit of a mystery, and I don’t have a lot of bandwidth for doing a deep dive on it, but did run it on a machine with a lot of memory and still hit the ceiling.

This is the failure that I got with the following settings:

auto-boundary: true, dem-resolution: 2.0, dsm: true, min-num-features: 32000, orthophoto-resolution: 1.0, pc-quality: high, rerun-from: dataset

Fused depth-maps 69 (40.59%, 968ms, ETA 1s)...
Fused depth-maps 92 (54.12%, 1s, ETA 908ms)...
Fused depth-maps 131 (77.06%, 1s, ETA 349ms)...
Fused depth-maps 170 (100%, 1s257ms)
20:55:28 [App     ] Densifying point-cloud completed: 0 points (7m35s767ms)
20:55:28 [App     ] MEMORYINFO: {
20:55:28 [App     ] 	VmPeak:	17046788 kB
20:55:28 [App     ] 	VmSize:	13699268 kB
20:55:28 [App     ] } ENDINFO
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud" --filter-point-cloud -20 -i "/var/www/data/f7074ff2-8a0b-4b6d-8dc7-3438fbe6cb7e/opensfm/undistorted/openmvs/scene_dense.mvs" -v 0 --cuda-device -2
20:55:29 [App     ] Build date: May  8 2023, 19:37:41
20:55:29 [App     ] CPU: Intel(R) Xeon(R) CPU E5-4657L v2 @ 2.40GHz (58 cores)
20:55:29 [App     ] RAM: 503.70GB Physical Memory 0B Virtual Memory
20:55:29 [App     ] OS: Linux 5.4.0-148-generic (x86_64)
20:55:29 [App     ] Disk:
20:55:29 [App     ] SSE & AVX compatible CPU & OS detected
20:55:29 [App     ] Command line: DensifyPointCloud --filter-point-cloud -20 -i /var/www/data/f7074ff2-8a0b-4b6d-8dc7-3438fbe6cb7e/opensfm/undistorted/openmvs/scene_dense.mvs -v 0 --cuda-device -2
20:55:29 [App     ] error: no valid point-cloud for the ROI estimation
Segmentation fault (core dumped)

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 139
Traceback (most recent call last):
File "/code/stages/odm_app.py", line 81, in execute
self.first_stage.run()
File "/code/opendm/types.py", line 398, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 398, in run
self.next_stage.run(outputs)
File "/code/opendm/types.py", line 398, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File "/code/opendm/types.py", line 377, in run
self.process(self.args, outputs)
File "/code/stages/openmvs.py", line 225, in process
raise e
File "/code/stages/openmvs.py", line 219, in process
system.run('"%s" %s' % (context.omvs_densify_path, ' '.join(config + gpu_config + extra_config)))
File "/code/opendm/system.py", line 110, in run
raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 139

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/
2 Likes
20

Thank you for looking at it!

2 Likes