Sift, Surf or Orb?

Andreas_Olsson · November 5, 2021, 2:55pm

Found a very interesting paper about sift/surf/orb. It seems that you need to choose the right one for the job.

I can’t try Surf so can’t say anything about it, Sift and Orb seems to have differences, I found that a point cloud gets much more points with Sift than with Orb and that Orb can’t even get me a result in some special cases. Both seem to do well with drone nadir images but when it’s photos from many different angles taken with a phone Orb can’t handle it.

This is a link to an interesting comparison of them:

smathermather · November 9, 2021, 4:36am

I think we have to be careful about how we interpret the results of a paper like this. From my reading, if using Sift I get a 30% increase in matches, but computationally I can add 10x Orb features in the same processing time, then: by the first metric, I can declare Sift the better approach; but if Orb is 10x or more the speed, then match rate is much less important.

Also, what I have seen in other posts regarding reduced residuals in the images is also an artifact: if per this paper, orb tends toward features in the middle of the image, then the reduction of distortion issues at the edge of the images is an artifact of sampling, not of an actual decrease. That said: a feature extraction algorithm which prioritizes the least distorted portion of the lens is super useful from a practical perspective, as this reduces overall error and could allow for lens models which ignore optimizing for problematic portions of the lens while also avoiding those problematic portions.

Anyway, I’ve had some anecdotal challenges with Orb as well, so I’ll try Saijin_Naib’s approach of throwing a whole bunch more features at it and see what comes out.

Over time, with all our disparate and varied datasets, experiences, and papers, we’ll together arrive at better recommendations, which makes this community process really powerful. Thanks for sharing this: I may have to read it a couple more times… .

Cheers.

Saijin_Naib · November 9, 2021, 4:47am

To add to this, I’m 99% certain we have what I’ve seen called ORB15K, or ORB that can extract up to 15k features per image, so throwing any more at it will not actually do anything. I’ve been giving out bad advice that way, haha. Apologies for everyone who used ORB with anything over 15K. Didn’t help, but at least it didn’t hurt

The Processing Report from 2.6.5+ has a really great section which shows min/mean/max matches and ORB will not exceed 15K no matter what you set it at, and what the subject is. SIFT/HAHOG will approach what you set it to.

SIFT64K:

ORB15K:

I’ve seen ORB30K used in a few papers looking at things like cellular structure and chemical compound structure, so there must be a way to add this version, or, what I actually hope for, enable ORB to take an arbitrary number of features much like SIFT and HAHOG currently can.

I’m not seeing where I can change this yet, so currently, consider 15K for ORB our actual practical maximum.

Gordon · November 9, 2021, 5:57am

I’ve certainly noticed the change in distribution of detected features with ORB, below typical SIFT then ORB heatmaps for large data sets.

However, as you say this can be, and I think definitely is for the M2P, a good thing, by preferentially selecting features in what is the zone of best image quality for most camera lenses.

Andreas_Olsson · November 9, 2021, 7:27am

Small cameras has much more depth of field so that’s no problem and barrel distortion is removed by software. So what quality are we looking for?

Even chromatic abberation is taken care of by software nowadays.

Gordon · November 9, 2021, 7:48am

Depth of field isn’t an issue with wide-angle or even short telephoto lenses when the target is 50 or 100m away and relatively planar. Barrel distortion is common in wide-angle lenses and as you say, easily corrected for.

Spherical aberration, focal plane not being perpendicular to optical axis (tilt), coma - plenty to choose from when looking at parts of the image that are well off-axis, and most are worse when operating with the aperture at or near wide-open, and not really very correctable with software without introducing artifacts.

Gordon · November 10, 2021, 2:14am

It appears that isn’t always the case, perhaps related to overlap?
I was going to use an old house to check whether or not it failed with ORB, rather than SIFT, until I noticed that I’d done it successfully using ORB initially!

Camera positions, very few nadir images in the camera cloud, most a long way from nadir.
(Open images in new tab to view at full resolution)

The resulting 3D, although I should have taken some lower closer images to better see under verandas.

smathermather · November 10, 2021, 2:52am

Classic. We always want more photos after the flights are done… But it looks great.

Saijin_Naib · November 10, 2021, 3:10am

Just so everyone knows:
I did not understand ORB well when I added it. It is heavily reduced in power due to how I implemented it.

It is currently broken:
https://github.com/OpenDroneMap/ODM/issues/1370

We also do not pass --matcher-type properly:
https://github.com/OpenDroneMap/ODM/issues/1368

I’m working to add AKAZE and BRUTEFORCE:
https://github.com/OpenDroneMap/ODM/pull/1369

There looks to be an upstream OpenSFM bug that further reduces the effectiveness of Binary Features descriptors:
https://github.com/mapillary/OpenSfM/pull/805

Current state with ORB/AKAZE with Binary Feature-Compatible BRUTEFORCE:
2-3x the number of features reconstructed VS current broken pairing with FLANN while still retaining multiple times faster processing than SIFT with BRUTEFORCE.

Next work:
Add optimized Binary Features-Compatibile matcher (like FLANN is for SIFT/SURF/HAHOG) to improve matching robustness and speed.

Further work:
Allow for more than 15k features, and expose tuning options to optimize and improve matching performance.

If it works at all for you now, it’ll only get much better as I fix my mistakes

APOS80 · November 10, 2021, 11:10am

Try these photos.

I tried Sift and ORB, ORB didnt work. No crop and ultra settings.

Saijin_Naib · November 10, 2021, 2:06pm

So, here’s what I have tested locally:

No tuning, and not using the optimal matcher, but there is certainly a big improvement in performance.

It should get even better once we can get ORB/AKAZE above 15k features, as SIFT is usually hovering around 30k-45k, giving it the ability to reconstruct as much as it does.

Saijin_Naib · November 10, 2021, 8:25pm

ORB+FLANN

Created on: 11/10/2021, 10:25:37 AM
Processing Node: Lightning - UAV4Geo (manual)
Options: auto-boundary: true, crop: 0, debug: true, dem-gapfill-steps: 4, dem-resolution: 1, dsm: true, dtm: true, feature-type: orb, matcher-neighbors: 0, mesh-size: 300000, min-num-features: 15000, orthophoto-resolution: 1, pc-classify: true, pc-filter: 1, pc-geometric: true, pc-quality: high, resize-to: -1, use-3dmesh: true, verbose: true
Average GSD: 0.07 cm
Area: 871.58 m²
Reconstructed Points: 11,724

ORB+BRUTEFORCE+DTYPE Fix

Created on: 11/10/2021, 1:32:03 PM
Processing Node: node-odm-1 (manual)
Options: auto-boundary: true, crop: 0, debug: true, dem-gapfill-steps: 4, dem-resolution: 1, dsm: true, dtm: true, feature-quality: ultra, feature-type: orb, matcher-neighbors: 0, matcher-type: bruteforce, mesh-size: 300000, min-num-features: 15000, orthophoto-resolution: 1, pc-classify: true, pc-filter: 1, pc-geometric: true, pc-quality: ultra, pc-rectify: true, resize-to: -1, use-3dmesh: true, verbose: true
Average GSD: 0.03 cm
Area: 917.96 m²
Reconstructed Points: 133,456

Still a broken reconstruction here, but 10x the points reconstructed…

I noticed that your iPhone 12 isn’t in the upstream OpenSFM Sensor database, so I’m inquiring as to how we should support such a multi-sensor device. It may help the reconstruction a bit.

Gordon · November 11, 2021, 1:18am

I’ve just tried with some of my phone photos using ORB, but it would not process, I have previously successfully run the dataset (a basket of Macadamia nuts), but I’m doing it again currently, and noticed this in the console log:
[INFO] No GPS information, using BOW matching
[WARNING] Using BOW matching, will use HAHOG feature type, not SIFT

Andreas_Olsson · November 11, 2021, 4:21am

Database of devices? Is that for ORB only because SIFT manage to give me a complete reconstruction.

I have an iPhone XR by the way.

Saijin_Naib · November 11, 2021, 4:26am

It is for any camera system that OpenSFM works with data from.

Sure, you have an iPhone XR but apos80 has the iPhone 12.

iPhone XR isn’t present upstream, either btw.

Have a JPEG with EXIF intact that you can give me so I can make another MR?

Gordon · November 11, 2021, 4:30am

apos80 = Andreas Olsson!

Saijin_Naib · November 11, 2021, 4:34am

Intrigue! Drama! Excitement!

But still, that dataset is tagged with iPhone 12 If iPhone XR needs/wants to be upstreamed, I need an image and I don’t feel like crawling flickr haha

Andreas_Olsson · November 11, 2021, 8:23am

iPhone XR might be a variant of 12. The images you got is from the XR.

Gordon · November 12, 2021, 7:55am

Not strictly related to the question in the subject line, but since I’m using ORB matching, and watching the console log in a big task…

From my current log:

2021-11-12 09:15:57,817 INFO: Reading data for image DJI_0498_7.JPG (queue-size=1
2021-11-12 09:15:57,818 DEBUG: Found 15000 points in 0.11483049392700195s
2021-11-12 09:15:57,830 DEBUG: Found 15000 points in 0.09299826622009277s

2021-11-12 09:15:58,007 INFO: Extracting ROOT_ORB features for image DJI_0498_7.JPG
2021-11-12 09:15:57,994 DEBUG: Found 15000 points in 0.09551215171813965s

2021-11-12 09:15:58,157 DEBUG: No segmentation for DJI_0498_7.JPG, no features masked.

2021-11-12 09:30:43,651 DEBUG: No segmentation for DJI_0498_7.JPG, no features masked.

2021-11-12 09:41:53,095 DEBUG: No segmentation for DJI_0498_7.JPG, no features masked.

2021-11-12 09:45:51,207 DEBUG: No segmentation for DJI_0498_7.JPG, no features masked

2021-11-12 10:21:35,958 DEBUG: No segmentation for DJI_0498_7.JPG, no features masked.

2021-11-12 10:35:20,695 DEBUG: No segmentation for DJI_0498_7.JPG, no features masked.

2021-11-12 11:35:07,855 DEBUG: No segmentation for DJI_0498_7.JPG, no features masked.

etc, in the log for all the matched pairs.

Does this represent any duplication of processing effort for this (and all other) images in this dataset, or is it just noting a previously determined ‘no features masked’ before attempting to match with another neighbour?

itenviro · November 13, 2021, 2:34pm

my 2 cents
fyi:
Images: 5933
processing parameters
Options: build-overviews: true, camera-lens: brown, crop: 0, debug: true, dem-resolution: 2, dsm: true, dtm: true, feature-type: orb, matcher-distance: 50, matcher-neighbors: 25, mesh-octree-depth: 8, min-num-features: 160000, orthophoto-kmz: true, orthophoto-png: true, pc-classify: true, pc-csv: true, pc-ept: true, pc-geometric: true, pc-las: true, pc-quality: high, pc-rectify: true, split-overlap: 0, verbose: true, rerun-from: dataset
64:26

report available at:
https://drive.google.com/file/d/12KrSL8ijE5GaQAd-Pmu8T4izZo2o5_bt/view?usp=sharing