Match_Features not muti-threaded?

Windows 10 Pro
Intel Core i9-10885H @ 2.40GHz
65GB Memory
Graphics: NVIDIA GeForce GTX 1650 Ti (apparently not being used)

Windows Native WebODM 1.9.14

I am attempting to build a 3D model (textured mesh) of a complex sculpture. Because of the complexity, I have shot ~350 images to try to cover everything. When I process this it takes “forever” to complete the match-features stage. What I mean by forever is that I have never successfully completed it. I have had issues where I thought it stopped processing because the log file did not update (see Windows Native WebODM hung matching images) for a long period of time - over 12 hours in one instance. I have had my computer crash after 40+ hours of processing. So, I have given up processing all images and am trying to do a subset, playing with masking and parameters, etc.

It appears to me that match_features is not running multi-threaded. My system has 8 cores, 16 threads, yet during match_features it does not appear that all cores/threads are being used. For testing purposes I am running with 50 images and trying to log what is happening.

The log file shows:

 [INFO]    running "C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm" match_features "C:\WebODM\resources\app\apps\NodeODM\data\92e945b6-70dd-4365-9f00-f2ade721920e\opensfm"
 2022-06-11 18:23:58,770 INFO: Matching 1225 image pairs
 2022-06-11 18:23:58,770 INFO: Computing pair matching with 16 processes

During detect_features, I see CPU usage pegged at 100% (the 100% CPU usage in screen capture of Task Manager below). I did the screen capture during match_features (right portion of screen capture):

Note that overall CPU is < 50%.
Likewise, Resource Monitor shows similar information:

The Python task at the top is averaging between 12 and 13%. In a test last week with more images, that was consistently around 8%. That Python task’s PID is 39744. Here is the command line for that task:

Command Line:
C:\WebODM\resources\app\apps\ODMWpython38\python.exe "C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\\opensfm_main.py" match_features "C:\WebODM\resources\app\apps\NodeODM\data\92e945b6-70dd-4365-9f00-f2ade721920e\opensfm"
Path:
C:\WebODM\resources\app\apps\ODM\python38\python.exe

I don’t know what to conclude other than match_features is not running 16 processes as it claims. If I could get it from 1 (or 2) processes to 16 processes, I might be able to complete my ~350 images in a reasonable amount of time. On one pass with just over 200 images, I projected that it would take ~120 hours to complete, and it was using about 7% of the total CPU. If I could get it even to 50%, that would be 7x faster, and the total time for match_feaatures would be <20 hours instead of the 120 hours I projected.

Bottom line: Is match_features working as intended?

3 Likes

What are your thermals like? Can you use something to monitor them? Laptop chips, especially in the i9 class, can be tricky as they’re very prone to throttling as most laptops can’t dissipate that amount of heat continuously.

TJunction is 100C (potential hard-shutoff), so expect some throttling and slowdown before 100C for sure. What that ramp looks like, only Intel knows for certain.

Also, make sure your laptop remains on AC power and your Power Profile is set to “Performance” not Balanced or Power-Save. With the newer chips, this is more of a hint/guideline than a hard performance state like in the older generation CPUs, but it should still help.

1 Like

Thanks, Saijin. I did find a program to monitor thermals. You are right - I am seeing throttling during match_features and several other steps.

2 Likes

Sorry, Gary! Then it is unfortunately out of our hands… These chips have incredibly complicated logic that allows them to run as fast as they safely can for as long as possible, so if the chip has decided it can only give you one or two cores then that’s all the thermal overhead you have. At least you know it is all working properly! When I switched to Linux I forgot to install thermald and I kept coming back to my desktop “crashed” off every time I was processing. Turns out my CPU was happily not throttling and running itself to thermal shutdown in a few minutes. Who knows how many years I knocked off of it with that!

You can try checking out things like spacers (so airflow beneath is increased), laptop active cooling mat/tray (extra fans), and/or possibly refreshing the heatsink compound, though on that generation CPU and it being an i9, I’d imagine the OEM was using decent stuff that should have a few years of service life left.

I’m also curious if tuning your Power Profile or --max-concurrency might help by limiting yourself to say half the cores and seeing if that gives you enough overhead that they can run faster than just a single thread pegged or all of them throttled aggressively.

2 Likes

Saijin, thanks for the help. Now I understand the problem. I hope I can eventually build myself a nice system, but for now I’ll have to work with what I have. I’ve noticed this laptop has very little clearance between the table and the air intake vents. I’ll start by improving that, and then work from there.

2 Likes