WebODM Now Taking Much Longer to Process Datasets With Matcher Process in Latest Version

I decided to run my test area (1500 images) from a project I have referenced in another thread one last time utilizing the latest version of WebODM over Docker on Windows 10 Pro that now includes OPK Triangulation to determine how much faster the processing times are before touching off the entire 8300 image project. I wanted to use my last run as a benchmark and keep all settings identical except using Triangulation instead of incremental. However, the latest version now leaves out Matcher Distance which I used in my original and successful run. That run utilized Matcher Distance = 100 and Matcher Neighbor = 24 and took 13 hours to process and the image matching part of the process took about an hour and a half. So I kept the matcher Neighbor at 24 for the benchmark test. That process took 2 days before it stopped running and hung up in the matching part of the process. Another thing I noticed was my Docker virtual disk exploded in size (I had it set to 1TB on a SSD). It did not run out of space but it grew to 400GB filled of that 1TB. I decided to download the Native Windows WebODM installer and try that to see if I could change my luck. The same test area is now 22 hours into processing and the console shows that it is still in the matching process and still churning. Like I stated earlier that same test area ran through the matcher process in an hour and a half. Has anyone else experienced this same issue with projects now taking longer in the latest version? My Laptop Workstation specs are 12 core 128GB RAM 1TB SSD C: and 2TB SSD D: with 256GB Virtual Memory. Below is reference to other link describing project.

1 Like

Does your dataset have OPK information to use triangulation with? Do you have the full console log thus far? It might have fallen back to incremental.

Yes my DJI Phantom 4 Pro has Omega Phi Kappa data in the images. And even if the process reverted back to incremental I still do not know why the matcher process is taking this long when it took an hour and a half with the previous release.

Actually I just checked the log and it says there is no OPK data in file and reverted to Incremental. Here is a snippet of that message:
[INFO] Running dataset stage
[INFO] Loading dataset from: D:\WebODM\resources\app\apps\NodeODM\data\a32be467-1af8-4a08-a00a-601ef87fe0e7\images
[INFO] Loading 1500 images
[INFO] Found 1500 usable images
[INFO] Parsing SRS header: WGS84 UTM 18N
[WARNING] No omega/phi/kappa angles found in input photos (Plan03ReDo_1.JPG), switching sfm-algorithm to incremental
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[INFO] Maximum photo dimensions: 4864px
[INFO] Photo dimensions for feature extraction: 2432px
[INFO] Using GPU for extracting SIFT features
[INFO] Altitude data detected, enabling it for GPS alignment
[INFO] Enabling hybrid bundle adjustment

I ran the identified image file through Exif Checker and here are those results:

ExifTool Version Number : 12.39
File Name : Plan03ReDo_1.JPG
Directory : D:/Sand Pond Ortho Raw Images
File Size : 7.0 MiB
File Modification Date/Time : 2021:12:29 12:13:36-05:00
File Access Date/Time : 2022:01:14 08:34:22-05:00
File Creation Date/Time : 2022:01:02 14:25:36-05:00
File Permissions : -rw-rw-rw-
File Type : JPEG
File Type Extension : jpg
MIME Type : image/jpeg
Exif Byte Order : Little-endian (Intel, II)
Image Description : DCIM\100MEDIA\DJI_0002.JPG
Camera Model Name : FC6310
Orientation : Horizontal (normal)
X Resolution : 72
Y Resolution : 72
Resolution Unit : inches
Software : v01.07.1641
Modify Date : 2020:05:05 09:40:58
Y Cb Cr Positioning : Centered
Exposure Time : 1/200
F Number : 5.0
Exposure Program : Program AE
ISO : 100
Exif Version : 0230
Date/Time Original : 2020:05:05 09:40:58
Create Date : 2020:05:05 09:40:58
Components Configuration : -, Cr, Cb, Y
Compressed Bits Per Pixel : 3.163009742
Shutter Speed Value : 1/200
Aperture Value : 5.0
Exposure Compensation : 0
Max Aperture Value : 2.8
Subject Distance : 0 m
Metering Mode : Center-weighted average
Light Source : Daylight
Flash : No flash function
Focal Length : 8.8 mm
Make : DJI
Speed X : -0.40
Speed Y : -2.10
Speed Z : +0.50
Pitch : -14.80
Yaw : -64.30
Roll : +2.20
Camera Pitch : -90.00
Camera Yaw : -62.80
Camera Roll : +0.00
Flashpix Version : 0010
Color Space : sRGB
Exif Image Width : 4864
Exif Image Height : 3648
Interoperability Index : R98 - DCF basic file (sRGB)
Interoperability Version : 0100
Exposure Index : undef
File Source : Digital Camera
Scene Type : Directly photographed
Custom Rendered : Normal
Exposure Mode : Auto
White Balance : Auto
Digital Zoom Ratio : undef
Focal Length In 35mm Format : 24 mm
Scene Capture Type : Standard
Gain Control : None
Contrast : Normal
Saturation : Normal
Sharpness : Hard
Subject Distance Range : Unknown
Serial Number : c91ab1ac3c9e59dfa94847718b3d2eec
GPS Version ID : 2.3.0.0
GPS Altitude Ref : Above Sea Level
GPS Time Stamp : 13:40:58
GPS Date Stamp : 2020:05:05
XP Comment : Type=N, Mode=P, DE=TrueColor
XP Keywords : v01.07.1641;1.2.0;v1.0.0
Compression : JPEG (old-style)
Thumbnail Offset : 2408
Thumbnail Length : 10445
Current IPTC Digest : 83c87141bf6db41fb4599582a00690e7
Caption-Abstract : DCIM\100MEDIA\DJI_0002.JPG
Application Record Version : 4
XMP Toolkit : Image::ExifTool 10.96
About : DJI Meta Data
Rating Percent : 50
Description : DCIM\100MEDIA\DJI_0002.JPG
Format : image/jpg
Absolute Altitude : +620.34
Calibrated Focal Length : 3666.666504
Calibrated Optical Center X : 2432.000000
Calibrated Optical Center Y : 1824.000000
Cam Reverse : 0
Flight Pitch Degree : -14.80
Flight Roll Degree : +2.20
Flight Yaw Degree : -64.30
Gimbal Pitch Degree : -90.00
Gimbal Reverse : 0
Gimbal Roll Degree : +0.00
Gimbal Yaw Degree : -62.80
Relative Altitude : +121.40
Rtk Flag : 0
Self Data : 5eb16c79c3c76749cab7207d4.4.1
Date/Time Digitized : 2020:05:05 09:40:58-04:00
GPS Map Datum : WGS-84
Rating : 3
Already Applied : False
Has Crop : False
Has Settings : False
Version : 7.0
MPF Version : 0010
Number Of Images : 2
MP Image Flags : Dependent child image
MP Image Format : JPEG
MP Image Type : Large Thumbnail (VGA equivalent)
MP Image Length : 254235
MP Image Start : 7034167
Dependent Image 1 Entry Number : 0
Dependent Image 2 Entry Number : 0
Image UID List : (Binary data 66 bytes, use -b option to extract)
Total Frames : 1
Image Width : 4864
Image Height : 3648
Encoding Process : Baseline DCT, Huffman coding
Bits Per Sample : 8
Color Components : 3
Y Cb Cr Sub Sampling : YCbCr4:2:2 (2 1)
Aperture : 5.0
Image Size : 4864x3648
Megapixels : 17.7
Scale Factor To 35 mm Equivalent: 2.7
Shutter Speed : 1/200
Thumbnail Image : (Binary data 10445 bytes, use -b option to extract)
GPS Altitude : 632.2 m Above Sea Level
GPS Date/Time : 2020:05:05 13:40:58Z
GPS Latitude : 43 deg 56’ 49.50" N
GPS Longitude : 75 deg 10’ 4.09" W
Preview Image : (Binary data 254235 bytes, use -b option to extract)
GPS Latitude Ref : North
GPS Longitude Ref : West
Circle Of Confusion : 0.011 mm
Field Of View : 73.7 deg
Focal Length : 8.8 mm (35 mm equivalent: 24.0 mm)
GPS Position : 43 deg 56’ 49.50" N, 75 deg 10’ 4.09" W
Hyperfocal Distance : 1.41 m
Light Value : 12.3
– press ENTER –

1 Like

My guess is that matcher distance probably trimmed significantly the number of matches? How far apart are the shots on this dataset? 24 might be excessive.

1 Like

Actually it failed to process at 24 then I set Matcher to 0 and re running it. That 24 setting did work with the initial process prior to latest update.

1 Like

Right, but it (probably?) worked because it worked in conjunction with matcher-distance. If matcher-distance pruned a lot of matches, it probably took priority over matcher-neighbors.

1 Like

Ok that makes sense. I’ll wait till this process completes then I will set it to the default of 8 and see what happens. Thanks for your input!

2 Likes

And in regards to the OPK data I was doing a little research and realized the Phantom 4 Pro only has Pitch Roll and Yaw and need to be converted to OPK. I am assuming that I have to convert those values in my images utilizing the opensfm/exif.py script found here?

1 Like

In theory we should handle the conversion from PYR → OPK automatically. But we need to add support for various drones and perhaps the P4Pro needs special attention.

Upload a sample image here: DJI OPK - Google Drive

Also related thread: Sfm-algorithm: triangulation - #19 by pierotofy

1 Like

I just uploaded a sample for a Phantom 4 Pro V1. I see that you already have one for V2 and do not know if there are any differences. So you are saying that this conversion is supposed to occur automatically when selecting the Triangulation option?

1 Like

Was the drone oriented north when you shot that picture? (It doesn’t look like it is). See the instructions document in the drive folder.

:+1:

1 Like

I just uploaded one from a flight line oriented north. Sorry about that I didn’t see the initial instructional text.

3 Likes