Processing large jobs on amazon AWS EC2

Hi all,

I have tried WebODM on my laptop worked well for small/mid jobs of up to 100-200 photos but not for larger jobs. So I have set up an Amazon AWS t2.xlarge EC2 (16GB RAM & 100GB Hard) to test a large 645 photos job. It failed at my first attempt after 2 hours and now trying it for the second time, it’s about 3 hours and its still processing but not much activity is recorded in AWS CPU/RAM/SD usage logs - I can see max 20% of SD usage and 70% of RAM usage at peaks.

Can anyone give me an indication of time/resources on AWS to finish processing large jobs with over 500 and possibly over 1000 photos please? Not sure whether I’m using the right Instance or there’s something else I could do to optimise the time/cost of AWS processing!

Any thoughts?

Ok the second attempt on t2.xlarge EC2 has faild too so I’m trying it again with an auto-scale upgrade to see whether more CPU & RAM can help processing this job. Fingers crossed!

If this doesn’t work I might have to try WebODM Lightning too!

MeSaberi,

The amount of RAM used, depends on the WebODM settings/parameters you are using, and the dataset size- 16GB of RAM is tiny for any PC/Processing of any kind in my own opinion (Its not 2004 anymore). Try running a much bigger instance 128GB of RAM or more for 600+/1000+ pictures and go from there. You will then have a completed dataset that you can use and review logs/performance metrics to tweak and optimize to better match your budget and needs.

Thanks @korn704, I thought using AWS auto scaling on my t2.xlarge EC2 would increase RAM and CPU and HDD as much as it needs to but apparently this is not the case when I only have one active instance coz my third attempt processing a 650-photo job failed after 3 hours error code 1. I guess I’m still new with AWS :slight_smile:

Not sure whether my auto scaling didn’t work or I have to use a better EC2 in the first place. Am trying a 128G ram instance. Let’s see how it goes

Good Luck Mel, takes some time. But very good when you have it all configured and tweaked. Google Compute is great as well.

Sorry for the delay updating this. AWS successfully set up and tested on a lareg 650-photo project and also a 1200-photo project!

All I needed was a RAM optimized r5ad.4xlarge EC2!! Ran perfectly smooth, absolutely no problems. 2 Hours uploading and processing time for 650-photo project and 3 hours uploading and processing time for 1200-photo project. Can’t be happier to be honest!

Will certainly donate as this level of service, accuracy and compatibility is out of this world for a FREE open source software. Well done everyone who contributed to this!

8 Likes

Glad it worked!

While the project doesn’t accept donations, there’s lots of ways to contribute back. See https://github.com/OpenDroneMap/WebODM/#support-the-project for ideas that apply to ODM as well. Also keep an eye out for funding opportunities on https://community.opendronemap.org/c/funding/31 :+1:

4 Likes

Thanks for sharing MelSaberi! I’ve been curious about AWS myself. I’m a bit surprised that, even with upload time included, your AWS runs seem to be going at least ~2-3 times faster than my local processing on a decent workstation I have! I may have to dive in and see roughly what the costs for processing would run with AWS running ODM for higher priority projects!

2 Likes

Hi, just found this old thread, as I’ve just been through the same process with AWS. Started off with T2, and am now trying to process 1400+ 12M photos DJI phantom 3S, and have moved to r5.2xlarge . . .to get more processors and mainly RAM 8/64G. I don’t really understand the many AWS families, but have just read that the ‘c’ are for compute bound processes, which I guess webodm is, or is it memory bound? has anyone tried a c5 instance? My slightly smaller job with 1025 photos has been running for 2 hours (but the green progress bar in webodm beside ‘running’ doesn’t ever seem to indicate how far through the job it is?).

1 Like

Costs. Do remember to stop the instance as soon as your job finishes (after downloading the results) so you only pay for the time the instance is running! Storage is different, and I’m not sure whether you have to pay for a full month even if not used. But storage costs are much less than instance.

1 Like