Help me understand how to make use of ClusterODM in Google Compute Engine (I'm a noob!)

[cmccurdy]

So it took me about a week of rubbing my 2 brain cells together, but I finally learned enough about Google Compute Engine & WebODM to be able to successfully deploy WebODM to a GCE instance and process some imagery.

Then I realized I could create a machine image of that instance and create new instances from it, so I could then scale up my processing power and process multiple datasets simultaneously.

I now realize this isn’t the most efficient or even the intended way to use WebODM. I mean, it works, but it’s also not ideal to have to use separate tabs to access each instance running WebODM.

Say, for instance, my setup looks like this:

[VM1 running WebODM1] > accessed via web interface #1 on desktop PC
[VM2 running WebODM2] > accessed via web interface #2 on desktop PC
[VM3 running WebODM3] > accessed via web interface #3 on desktop PC

Essentially 3 GCE instances running WebODM/NodeODM

What would the equivalent setup with ClusterODM look like?

Something like this?

[VM1 running NodeODM1]
[VM2 running NodeODM2] - [VM4 running ClusterODM & WebODM] > web interface on desktop PC
[VM3 running NodeODM3] /

Is VM4 necessary, or can I just run ClusterODM & WebODM on my desktop?

I’m not looking for someone to hold my hand through every step of setting this up (yet), but I guess I’m just looking for some clarity on what the general structure should look like, as far as what needs to be installed on what machine for everything to work properly.

Sorry if these are dumb questions! I am very new to this kind of stuff

[mbohman]

are you still looking for help?

[cmccurdy]

I never figured out how to use ClusterODM, but I figured out how to get multiple instances of NodeODM running and access them via the WebODM Lightning Windows 10 app. My biggest annoyance with my original setup was having to have separate WebODM windows open to access each node, so I’ve resolved that issue now.

My current processing workflow looks something like this:

1. Spin up 2-3 VM instances in Google Compute Engine (I use a machine image with WebODM/NodeODM already installed on it, and I have a startup script that launches a NodeODM server on startup)
2. Open the WebODM Lightning Windows 10 app
3. Copy the external IPs for each VM instance and add them to WebODM Lightning app as processing nodes
4. Select imagery for Project #1, adjust settings, choose Node #1 as the processing node, start processing
5. Select imagery for Project #2, adjust settings, choose Node #2 as the processing node, start processing
6. etc…
7. Review & download outputs
8. Remove processing nodes from WebODM Lightning
9. Delete Google Compute VM instances

I’m not sure if this is the most efficient way of doing things, but I’m open to suggestions!

Would ClusterODM simplify or speed up my workflow? My datasets are usually 200-300 images each, and they are all oblique imagery (of cell towers), consisting of multiple sub-missions (usually several “orbit” sub-missions, and a “column” sub-mission, as described here: https://support.pix4d.com/hc/en-us/articles/360041078692-How-to-map-cell-towers). I’m just not sure if splitting my datasets arbitrarily and merging them at the end will be as effective as with nadir imagery datasets, but I’d be willing to try it out.

edit: I just read some of the documentation (https://docs.opendronemap.org/large.html#local-split-merge) and I see that you can manually define how the split occurs by creating a image_groups.txt file, but I’m still not sure how I would benefit from splitting up my datasets.

[smathermather-cm]

Doing splits isn’t a requirement of ClusterODM: it’s a NodeODM proxy that can also help orchestrating resources when spli-merge is useful.

As to allocation and deallocation of resources, this is ClusterODM’s strongest point. Take a look at the Autoscale-setup section of the README for ClusterODM: https://github.com/opendronemap/clusterodm#autoscale-setup

[cmccurdy]

I see.

I guess I need to familiarize myself with AWS lol. It took me a few days to figure out how to use GCE, and another week or so to get things running smoothly. I’m VERY new to all of this stuff.

If you think it would be appropriate and/or helpful, I put together a “for dummies” style tutorial for setting up WebODM in GCE, with step-by-step instructions and screenshots. It’s basically fleshed out version of this section of the WebODM documentation.

Is this something that you think would be a good addition to the documentation?

[coreysnipes]

I would love to see this, myself. Yes please do. If you don’t want to dive in to the ODM docs setup you can just provide text and images somewhere and I can help pull it in to the main docs.

[cmccurdy]

Cool.

I’ll try to do it myself, but if I can’t figure it out I’ll send you a Dropbox link with the doc and screenshots!