Reconstructing models from multiple levels (aerial and ground), initial findings

smathermather · 28 May 2022 04:29

Ok my friends. I present to you my findings on combining ground level photos with aerial imagery in a cohesive and fantastic model. Now, there are some limitations to this model: one set of photos was taken on a sunny day at sunset and the other on a partially cloudy day a couple hours before sunset. In addition to lighting differences, we can expect challenges from differences in GPS between days (we cannot rely on lower relative error for short flights). But the challenges are an important part of understanding how to do this even under less-than-ideal conditions.

I have been trying to think through the problem space of combining aerial and ground photos in an effective and scalable way for a while. At first I thought it would be easy (circa February 2021), and now I know it is not as easy as it could be.

But let’s start with some results and the successful endpoint(cloud):
ezgif-4-c49b83c717

And mesh:
ezgif-4-43b79ee594

So, let’s talk about failures when this doesn’t work. This was the first result I got when I combined ground images and low flight images with higher orbits:

The ortho isn’t quite… I mean, I suppose it is orthorectified, but the position from which its been orthorectified is way off… .

And this has everything to do with the fact that OpenSfM is now doing a more meaningful job integrating all possible data into the model (yay!) but not doing a good job initializing the model (awwww!).

To test this theory, I tried setting the GPS accuracy for the imagery to encourage the SfM to prioritize appropriately the good data when setting up the model. This didn’t work well:

See how we can see the side of the buildings? It is better, but not good enough. What we need to do instead is initialize the models using the best available data so that the orientation is better fixed from go.

Baring changes to OpenSfM (which I will be proposing, I suspect it’s at worst a 10-line fix), I simply stripped all the GPS tags out of the less accurate data, and reprocessed. This ensures that no orientation data is taken from the images with poor GPS. It means we have to use BOW matching, which isn’t as fast usually as the current incremental approach.

smathermather · 28 May 2022 04:36

Reading through the OpenSfM docs here:
https://opensfm.org/docs/reconstruction_module.html?highlight=initialization#finding-good-initial-pairs

I suspect we need to add a weighting factor for GPSDOP/GPSXYAccuracy/GPSZAccuracy right around here (assuming that is feasible):

github.com

mapillary/OpenSfM/blob/main/opensfm/reconstruction.py#L188

      
        
                    return outliers
                else:
                    return 0
            
            

            
TPairArguments = Tuple[
                str, str, np.ndarray, np.ndarray, pygeometry.Camera, pygeometry.Camera, float
            ]
            
            

            
def compute_image_pairs(
                track_dict: Dict[Tuple[str, str], tracking.TPairTracks], data: DataSetBase
            ) -> List[Tuple[str, str]]:
                """All matched image pairs sorted by reconstructability."""
                cameras = data.load_camera_models()
                args = _pair_reconstructability_arguments(track_dict, cameras, data)
                processes = data.config["processes"]
                result = parallel_map(_compute_pair_reconstructability, args, processes)
                result = list(result)
                pairs = [(im1, im2) for im1, im2, r in result if r > 0]
                score = [r for im1, im2, r in result if r > 0]

Opened an issue on github:

github.com/OpenDroneMap/OpenSfM

Feature request: add GPS Accuracy to score in compute image pairs

opened 04:49AM - 28 May 22 UTC

smathermather

Model initialization for reconstructing models with datasets that contain variab…le accuracy GPS, like when combining aerial and terrestrial data where the terrestrial imagery may have lower confidence GPS than the aerial imagery. The choice of initial camera pairs can drive the creation of the entire model resulting in wildly skewed models like this: ![image](https://user-images.githubusercontent.com/1174901/170810292-5d1d63f1-eb7d-43fb-9b4e-4edae4e45c39.png) When the model when orthorectified should look like this: ![image](https://user-images.githubusercontent.com/1174901/170810319-f160d630-9132-45ea-921d-348d46b9319d.png) Removing GPS data for shots with low quality GPS can result in much better models, but forces the use of BOW matching which can be much slower than OpenSfM's default incremental approach: ![image](https://user-images.githubusercontent.com/1174901/170810368-738b1d42-8ad1-45af-861a-9c62ea95a7f4.png) I propose adding an optional scoring factor to [compute_image_pairs](https://github.com/mapillary/OpenSfM/blob/main/opensfm/reconstruction.py#L188 ) that prioritizes high quality estimates of GPS over low quality, when estimates are available I have also posted about this on the community forum here: https://community.opendronemap.org/t/reconstructing-models-from-multiple-levels-aerial-and-ground-initial-findings/11817

pierotofy · 29 May 2022 01:14

If the goal is to get a better orthophoto, have you tried using --use-3dmesh?

smathermather · 29 May 2022 02:16

The goal is the 3D model itself correctly oriented. It’s just easier to show the model initialization issues and how they propitiate through by using the orthophoto.

pierotofy · 29 May 2022 02:24

Ah, I see; how about patching/forcing ODM/osfm.py at master · OpenDroneMap/ODM · GitHub align_method: orientation_prior (which by default is auto?)

smathermather · 29 May 2022 04:04

I’ll give it a wing.

smathermather · 29 May 2022 16:44

Forcing orientation prior resulted in the same / similar outcome:

smathermather · 29 May 2022 16:56

My try naïve next just for giggles.

israelbar · 29 May 2022 18:20

Point cloud and mesh look great. Are you using the same camera?

is the model tilted or deformed? is it possible to control model orientation and position by using GCP’s?

smathermather · 29 May 2022 18:53

Yes, point cloud and mesh are frankly fantastic. No distortion, just rotation.

One camera for all, including carrying the drone around the yard with rotors removed with a child operating the trigger for the lower shots.

GCPs are an option, but seem a bit silly when most or all the gps info is good enough, but orientation priors are just estimated wrong.

smathermather · 29 May 2022 19:15

Naïve is well named:

Models themselves are quite fantastic. One additional anecdote is that matching results in 1 model if I force BOW, but generates 3 models (which get blissfully merged) otherwise.

smathermather · 30 May 2022 00:34

Aside from orientation, the model is soooo good:

(BTW, the cars moved several times, so they are a mess)

smathermather · 1 June 2022 23:24

Any more thoughts / alternatives for this?

pierotofy · 2 June 2022 04:33

I’d try to remove the GPS coordinates from the ground shots. It’s strange that OpenSfM can’t orient the model properly though, but I haven’t seen the dataset.

smathermather · 2 June 2022 15:56

I can share the dataset later today.

nsicart · 5 June 2022 07:55

Hello folks!

I was also going to ask if ODM could work for on ground photogrametry. I mean walking around with my phone taking pictures.

What would be the best processing configuration?

Thanks

GaryStebbins · 5 June 2022 18:24

It sure can. There are a number of posts about doing so. Here is one of mine: My simple WebODM project - print a rock! . I think I processed with mostly default settings, probably changing the initial option to don’t resize photos.

Also, look at more detail on settings in Kerrowman’s thread here: Creating 3D model from phone photos - best settings .

Kerrowman · 5 June 2022 20:40

I did another ground one a few days ago using my iPhone 12. I also find it helps using a remote for the shutter in one hand and the phone in the other.

Here is the end result: Kerrow Lintel Stone - Download Free 3D model by Kerrowman [2bde0e3] - Sketchfab

It took 261 images and I set my RAM useage to 15GB out of 16 available and these settings:

Pixel: 2,000, Feature Quality: Ultra, PCQuality: High, Mesh: 300,000, Min Features: 50,000 and all others on default settings.

The pixel settings is not the resizing but set underneath the edit button.

Kerrowman · 5 June 2022 20:43

Also PC Geometric: True, Use 3D Mesh: True and Auto boundary

Gordon · 6 June 2022 01:32

Julian, are you finding that some models are tilted when viewed from some angles, such as your lintel, which is tilted when viewed end-on. It’s as if WebODM isn’t finding the ground plane correctly, which I guess is a bit difficult when there is minimal flat area around the object of interest.
I’ve had that with a number of objects, both with drone and phone photos, but I can view them untilted in Meshlab.