Difference Windows and Docker version

[MarkoM]

Im having troubles with windows version of WebODM.
After some time i decided to support it with buying it (i was using docker before) and now its causing me troubles.
I would think its the same as docker version but somehow its not… to worse sadly.
Processing takes longer even with GPU now working its share. I notice that it doesnt use 100% of CPU as docker did but somehow around 25% or so. Thats why processing time got longer i guess. It doesnt use GPU as much either - like 10% doubt it makes a difference in processing.
Then with same settings as docker verion it didnt process test dataset. It removed like 2/3 of the area at start. With some tweaking with setting i managed to make it process whole thing but it takes 25% longer at least.
I hope i am just doing something wrong and its solvable. Worse case ill just go back to docker and wait for it to support GPU processing. I tried with playing around with setting but then i rather paid for full version as it was suppose to work out of the box.
It works the problem is how… And i have something to compare it to and its not ok.
Any ideas apprecieted.

Edit: looks like its using only e-cores. Thats why its slow. Docker is using all 8+8+4 cores

Edit 2: scratch that… its not that… but it should be using 100% of cores until job is done at least docker did that.
Now im thinking if that number of concurrent processes is set to 20 = number of cores or its % of CPU time?

[Saijin_Naib]

The --max-concurrency variable controls how many threads it will divide the jobs suitable for multi-thread processing into.

I’m not sure we’ve seen many folks on heterogenous CPUs like the 12th gen Alder Lake CPUs quite yet…

Are you on Windows 11? It looks like Linux (therefore Docker) and Windows 11 have the best support for Thread Director currently.

[MarkoM]

So i thought right. Should have used all the processors for computing. I have noticed at some point it actually did but just for few seconds. But last 4 cores were used most of the time.
I will put back docker and run same set with same settings in the morning.
I’m using Windows 10. I have new computer because 10yr old one was not suitable for processing anymore. You can see hardware from screens if it helps. Should work fast.

[Saijin_Naib]

There’s a complication, however, Marko.

Intel Thread Director is a kernel-level feature to work with the new scheduler on-CPU in the Alder Lake CPUs. Windows 11 and recent Linux versions have support for it, Windows 10 does not (and by current news, will not).

Without Intel Thread Director, you can not expect full cooperation between the Performance Cores and Efficiency Cores, as Intel Thread Director handles that in concert with the CPU’s thread scheduler.

Can you take the Windows 11 upgrade early to see if it improves the behavior for you?

Some more detailed information here:

Intel Architecture Day 2021: Alder Lake, Golden Cove, and Gracemont Detailed

[MarkoM]

Oh thank you for this.
That’s why it works on docker and not windows… Looks like it has the support for it though wsl.

I’ll check what’s with upgrade to win11. From what I briefly checked the 12gen processors are made for win11.
I’ll be back with the results once i figure it out how to update windows.

[Saijin_Naib]

@MarkoM

I’ll be back with the results once i figure it out how to update windows.

https://www.microsoft.com/en-us/software-download/windows11

You can start here with the Upgrade Advisor, or use the tool to make Installation Media on a USB. Once you have the USB, run the setup.exe from the USB and go through the install/upgrade process. This is what is called an in-place upgrade.
Here’s a short guide (ignore their suggestion to use a virtual ISO mounted as a drive, actually write it to a USB flash drive):

How to run In-place upgrade in Windows 11

[Samdrone]

I have same results. The docker version are much faster then windows version. 2-4 times faster. sometimes, the windows version just can’t finish any project after 24 hours.
Mine is old computer with 256G RAM.

I think the windows version still have some big problems. If I start windows version, it just start to run out all 256G RAM. Now I have few times can really get into windows version so far.

[Saijin_Naib]

@Samdrone

I have same results.

What is your CPU? If you aren’t using an Alder Lake CPU, you probably are not experiencing the same issue Marko is.

[Samdrone]

I only talk about the speed.

[Samdrone]

just run another set to test both docker and windows version.
Docker, finish in 1h16min.

windows 1h 50min, still running, looking only 10% finish.

CPU and RAM has a lot of extra to use.

[MarkoM]

Ok im back with new win 11 (liking it) and results… not what you would expect.
Same dataset. Almost 50% difference in favor of Docker.
Windows version uses GPU in start for some computing (i guess extracting matches) where Docker is not.

And i see @Samdrone have similar results.

Edit: funny how results are different (GSD, area and reconstructed points) even results are different. Not much but it makes a difference.

[Saijin_Naib]

Do either of you have any Anti-Virus or other security software running in the background with Realtime Scanning turned on?

[MarkoM]

Just windows defender which last scan was 30s when i installed windows 11.
Shouldnt be causing troubles. Task manager shows no activity worth mentioning out of usual.

[Saijin_Naib]

Thank you for confirming that. It doesn’t interfere in my work either, but I have seen 3rd Party Anti-Virus do so.

Something else is up.

What is the processing time/CPU usage delta like between Windows 10 and your Windows 11? Has it improved or changed now that you’re using Intel Thread Director?

[MarkoM]

I dont think it uses the cores corectly. From what i see theres 3 very similar threads from today. (maybe merge or something?)
Difference is like 1min or not even that from what i remember. I didnt write down processing time from windows 10 but i think it was like 19m:15s or something like that. Docker was about same in win10 as it is in win11 (around 10min).

I think the problem is in the program. I would say Docker works as intendent. Could be even faster with GPU support but i havent figured it out how to make it work. Maybe i will try that solutions to enable it in other thread again with win11.

[MarkoM]

GOOD NEWS! (NOT… read below edit2)
Was reading about this Thread director around and what it does and came across an article about the problem with it in win 11.
From here

Now usually when I’m dealing with video exports, it’s the video throughput that is my limiting factor. I need the video to complete, regardless of what I’m doing in the interim. By defocusing the video export window, it now moves to the slower E-cores. If I want to keep it on the P-cores in this mode, I have to keep the window in focus and not do anything else. The way that this is described also means that if you use any software that’s fronted by a GUI, but spawns a background process to do the actual work, unless the background process gets focus (which it can never do in normal operation), then it will stay on the E-cores.

In my mind, this is a bad oversight. I was told that this is explicitly Microsoft’s choice on how to do things.

The solution, in my mind, is for some sort of software to exist where a user can highlight programs to the OS that they want to keep on the high-performance track. Intel technically made something similar when it first introduced Turbo Max 3.0, however it was unclear if this was something that had to come from Intel or from Microsoft to work properly. I assume the latter, given the OS has ultimate control here.

I was however told that if the user changes the Windows Power Plan to high-performance, this behavior stops. In my mind this isn’t a proper fix, but it means that we might see some users/reviews of the hardware with lower performance if the workload doing the work is background, and the reviewer is using the default Balanced Power Plan as installed. If the same policy is going to apply to Laptops, that’s a bigger issue.

After i changed power plan to high performance it finished in 11:50. Still slower than Docker but we are getting there.
Lets see if it can be tweaked more. We still have 10% or more to gain to get on pair with Docker not counting GPU processing.

Edit: Looks like its not “fixable” link

Edit 2: Scratch everything… it only processed half of data somehow… back to square 1.
Any idea why results are so unconsistent?

Edit 3. Yup… All above makes no difference in processing. 27min40s for this:

[MarkoM]

Tried with upping page file size as there was solution in other thread but no difference.

I think problem is new CPU.
Could be something else too but i cant figure out and ran out of ideas…
So Windows native is slow… Docker doesnt support GPU… zzz ill have to wait for someone to come up with somesort of fix eventually when more users get new CPUs or Docker version getting CUDA support with Windows. It works… just not in WebODM.

[Saijin_Naib]

Marko, samdrone,

Could you both find/attach your log.json files for me? It would help to investigate which stages are taking how long (and hopefully come to a conclusion as to why).

It should be at the root level of your all.zip export (next to cameras.json, images.json, task_output.txt, etc).

[MarkoM]

Sorry for late reply. I ran some tests and as it looks like windows version takes twice as long as docker.
By default docker and windows have different min max settings so thats adjusted so its comparable.
Maybe theres a bug or something that windows version doesnt use all cores all the time. Thats the root of the slow processing… it uses just one or two cores most of the time. Otherwise the result is pretty similar.

Docker CPU.txt

Docker GPU.txt

Win11.txt

[smathermather]

Saijin may be able to help you here. This is less a bug and more a “feature” of Windows 11, and there are workaround.