How to get the processing stage of my task?

WebODM
#1

Hello Forum :slight_smile:

I would like to get more information about at what stage my current project is. The last message in the Web-Dashboard Log-Field is from hours ago:

...
2022-05-01 14:28:49,545 INFO: -------------------------------------------------------
2022-05-01 14:28:49,552 INFO: DJI_0118_QW90wbu.JPG resection inliers: 124 / 141
2022-05-01 14:28:49,556 INFO: Adding DJI_0118_QW90wbu.JPG to the reconstruction
2022-05-01 14:28:50,535 INFO: -------------------------------------------------------
2022-05-01 14:28:50,541 INFO: DJI_0119_rhig42E.JPG resection inliers: 175 / 203
2022-05-01 14:28:50,545 INFO: Adding DJI_0119_rhig42E.JPG to the reconstruction

I am a little surprised that the last message doesn’t mention anything about a finished task or a newly initiated task. But I assume that’s the way it is.

So far everthing indicates that the node is still processing, but it probably run out of RAM and is swapping to the M2 drive. (CPU usage at about 20% with spikes of 100%, M2 drive is working non-stop and free space is decreasing (will probably run out of free space before project can finish))

I already logged into the docker container (webodm) with this guide but I couldn’t find anything that indicates at what stage the processing is.

Could somebody point me into the right direction to figgure out at what stage the processing is?

btw. I’m not in a hurry nor do I worry. I just try to understand it better in order to estimate better how long projects will take to process with certain settings.

1 Like
#2

I would check the RAM and swap usage on the system that is running the assembly. I have experienced similar things when a processing node runs out of RAM and starts using swap. It should still finish (eventually), if you have enough swap space. ODM uses a lot of RAM for big datasets.

2 Likes
#3

Welcome, Joku!

As probably-not-porter said, you’re likely swapping pretty heavily, so you’ve got a ways to go before the log updates and things progress with visibility in the log.

Hang tight!

#4

Thanks for your answers. But my question was another one: How can I find what step of the whole process is currently running?

The Log-Output on WebODM got updated and last messages are:

2022-05-01 19:49:35,959 DEBUG: Undistorting image DJI_0067_nNdVoxX.JPG
2022-05-01 19:49:36,368 DEBUG: Undistorting image DJI_0853_Gn15KM3.JPG

Unfortunately I don’t know what step would give such an output. Is it SFM? Meshing? DepthMap?

Following the guide I linked in the initial post I run find . -name depth-L1.mvei | wc -l inside the webodm docker container and get a 0 as output, which seems to indicate that there’s no data for DepthMap from MVE yet.
But what step is being processed at the moment? And where would I have to reiniciate the process in case if processing is getting stoped (for not having enough free space i.E.)?

Is there any guide or wiki which explains the different processing steps in detail? (and yes, I did google but couldn’t find what I was looking for)

1 Like
#5

Undistorting should occur during the OpenSFM stage.

If you have a failure, then I’d recommend rerun-from opensfm at the least (unless you change parameters in which case I generally recommend rerun-from dataset).

No guide yet! It’s on the list, however. You may have found this already, but maybe this can help for the time being:

1 Like
#6

Thanks for the Link. It gave me an overview of the processing pipeline and the idea to search for other keywords. That helped :slight_smile:

The easiest way was to download the log file from the WebODM Dashboard, open the file and search for running. This way I can see every command that was run, hence I know now that the last command was
[...]opensfm export_geocoords --reconstruction [...]
I kind of got confused initially because the output in WebODM just shows the last few lines of the log file. How could I not come up with the Idea of downloading the log file :man_facepalming: :sweat_smile:

But I also found other interesting indicators. I’ll write it down as reference:
ODM Output and Results (Folder structure) on GitHub
Opensfm files and folders
Opensfm reporting

But it’s late already. I’ll dive more into it tomorrow.

2 Likes
#7

So after a few days the processing stopped, I reloaded the webodm-page and processing time went from 104 hours down to 20 hours (but still processing). I saw that there were additional docker containers running (I think 5 in total). The worker-container hat the status ‘restarting’, but didn’t do anything.
So I klicked the ‘play’-symbol of the worker-container which caused the processing to stop with an error message. Unfortunately I didn’t make screenshots…
But that’s actually not the topic here haha.

I found another useful Link to get information about the ongoing processing step in OpenSFM:
OpenSFM: Reconstruction Commands
It seems as if the commands are listed in a chronological order.

1 Like