Webodm for Out and Indoor with different cameras + Cluster

ClusterODM
#1

Hey all, i would like to process a dataset with this area:

https://1drv.ms/u/s!An38OVIiD8ij3c4aQzC0IlEI_6EvxA?e=HDRTez
PW: Bitmap

We are currently planning the dataset and how to process it. We guess there will be more than 5000 Picuteres since we want to use the DJI Osmo+ Gimbal for indoor and close distance photos and a mavic 2 pro and maybe the mini 2 for air photos. We currently don’t know how we can use different cameras on the same dataset since earlier tests where not that succesfull.

However we were also not able to get the cluster working. We tried to use the windows cluster-odm package and the linux docker version, we were never able to get the node online. We were able to start ClusterODM, use telnet to start the nodes, but they were never usable (allways got error messages directly after connecting).

We also dont know how the token stuff works and we may just need a red line to walk by. This was not really usefull for me, i guess i just miss the part were the api keys are defines?
Cluster ODM + WebODM + ODM Installation baby steps - ClusterODM - OpenDroneMap Community

This Area has around 16 Acres on the normal ground floor plus dozen acres indoor and i want to process it fast with the cluster (at least 2 Server with 64gb, 16 cores and 2TB SSD per Server, Ubuntu 20.04) maybe much more at a later point.

1 Like
#2

Welcome!

Thanks for sharing part of your data, your setup, and your plan for this collect.

This is quite a large task!

Have you searched around the Community for threads about ClusterODM and NodeODM deployments? Many Community members here have shared lots of good advice.

#3

https://bitmap.bitblade.io/public/task/363a8c00-d4fa-42d8-b8e0-cc29943e45a3/map/

Here you can have a closer look at the area.

I had a look at different Topics like the one linked above. But even this own does not totally explain how to correctly realize a correct environment.

As far as i actually get it is possible to have webodm and clusterodm on one small server, after this i can have multiple cluster odms (standalone?) on other servers.

Setting up WebODM, ClusterODM, and multiple ODM nodes across separate hardware - WebODM - OpenDroneMap Community

2 Likes
#4

Almost! I believe the current best practice is to have at least one sufficiently large machine hosting ClusterODM and NodeODM, as some of the final reconstruction work needs to take place on a single node (and that should be the default node since it hosts ClusterODM).

On your other processing machines, you’ll just need NodeODM for processing, and to set them up as remote nodes on the ClusterODM host.

For viewing, WebODM can theorhetically be on any of the machines, but it might just be easiest to have it on your one “main” machine that’s hosting ClusterODM and your default NodeODM.

That being said, I’ve not done a deploy like this, so folks here may well have much better advice for you on how to do such a deployment at scale from experience.

#5

I found an other thread Remote Processing - General Help - OpenDroneMap Community

But this still not answers my questions in regard to the security.

Does this bassicly means that i can run a normal full node with everything on the first server withthe cluster odm, plus on the second server i only need to run nodeodm?

#6

I’m not sure what you mean here. That you couldn’t bring the nodes online?

Pretty much, yeah! And same goes for other machines you bring online for processing. Just NodeODM and configured to talk to ClusterODM.

#7

I mean, if i start a clusternode, this must be added as node, and there is a field for the Token, do i need to create a token for this with a json? WebODM Documentation

1 Like
#8

The token isn’t required and aside from digging into the nodeodm and clusterodm code is really a bit undocumented.

Reality is, you should use a dedicated network for WebODM, clusterodm and the nodes and you can safely skip using a token.

If you require use of a token for compliance reasons or similar, I can see if I can dig out my notes on that.

1 Like
#9

I would really like to make it safe, therefore i have the option with the password and the resulting JWC Token. (If i see this correctly), or to define some ufw firewall rules. Both options are ok for me so i will focus first on creating a working cluster.

Both Servers have 64G Ram and 512G Swap 16 cores

Unbenannt

How do I install the Server0 ?

For this I use a minimal Ubuntu 20.04 Server with SSH already installed.

apt-get install update
apt-get install git
sudo apt install docker.io

sudo curl -L "https://github.com/docker/compose/releases/download/1.29.2/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

check docker with the command → docker-compose –version

git clone https://github.com/OpenDroneMap/WebODM --config core.autocrlf=input --depth 1

cd WebODM

wget https://bootstrap.pypa.io/pip/2.7/get-pip.py

./webodm.sh restart --ssl --hostname yourdomain.org --port 443

-After this I setup the admin and then I am done with the basic installation.

Type docker container ls to see 5 running rontainers: webapp, worker,db,broker and webodm_node-odm_1

?Question

According to Topic : “cluster-odm-webodm-odm-installation-baby-steps/5377 “ I shouldn’t start the odmnode on the server0, instead I should use 0. To archive this I assume that I can use
./webodm.sh down && ./webodm.sh update && ./webodm.sh restart --default-nodes 0 --ssl --hostname yourdomain.org --port 443

After this I would type on Server0
Docker
run -p 3001:3000 opendronemap/nodeodm --max_concurrency 16 --max_images 1000000
And on Server1
docker
run -p 3001:3000 opendronemap/nodeodm --max_concurrency 16 --max_images 1000000
(according to the mentioned usage thread OpenDronemap/ClusterODM#usage I should type (docker run --rm -ti -p 3000:3000 -p 8080:8080 opendronemap/clusterodm --ssl --hostname yourdomain.org --port 443)

On Server1 again (according to the OpenDronemap/ClusterODM#usage

I should type telnet localhost 8080
NODE ADD NODEODM-HOST 3001)

Then open http://server1:10000 in the webbrowser

/?Question

1 Like
#11

After the Cluster is now working, i still see no usage of the second node.

Please don’t use my installation manual above because it is incomplete.

I used the following parameters to start a process, but only node 0 is used, node1 has no resource usage:

auto-boundary: true, dsm: true, dtm: true, feature-quality: ultra, mesh-size: 3999999, pc-geometric: true, pc-quality: ultra, pc-rectify: true, split: 200, split-overlap: 60, rerun-from: odm_postprocess

1 Like
#12

I now tried: auto-boundary: true, dsm: true, dtm: true, feature-quality: ultra, mesh-octree-depth: 12, mesh-size: 3999999, min-num-features: 40000, orthophoto-resolution: 1, pc-geometric: true, pc-quality: ultra, pc-rectify: true, rerun-from: odm_postprocess, sm-cluster: http://externalnodeadress:3000, split: 40, split-overlap: 150

But still only node 0 is used.

I checked also with telnet if both nodes are online - they are, and they are accessible from the primary machine
console (1).pdf (1011.4 KB)

I found Cluster not being used · Issue #1013 · OpenDroneMap/WebODM · GitHub which has the suggestion to use the pbulic parameter for clusterodm. Will try this later.

Other thing to check later: Multi-Node Cluster - #17 by pierotofy

Cluster says

[INFO]    LRE: Upload of submodel_0004 at [90%]
[INFO]    LRE: Upload of submodel_0004 at [100%]
[WARNING] LRE: submodel_0004 (30a671cf-48e3-4d9b-a09a-86ff7d30bb7a) turned from RUNNING to QUEUED. Re-adding to back of the queue.
[WARNING] LRE: submodel_0004 failed with: Delayed task limit reached
[INFO]    LRE: Cleaning up remote task (30a671cf-48e3-4d9b-a09a-86ff7d30bb7a)... OK
[INFO]    LRE: Re-queueing submodel_0004 (retries: 0)
[INFO]    LRE: About to process submodel_0005 remotely
1 Like
#13

I guess it is working now. I will write a documentation for this and how to avoid all the errors I had. Will inform Saijin. But i still have not figured out how to make clusterodm persistent - not yet, and i think that the password and token option is set in the same file or parameter set.

2 Likes
#14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.