ODM Benchmark Data

I’ve started a GitHub repo for aggregating ODM benchmark data. It’s under my GitHub account, but if it seems useful to the community I would like to see it moved over to the ODM github account. I’m happy to manage the contributions if that is useful.

EDIT 3/27: Official repo is now https://github.com/OpenDroneMap/odm-benchmarks

Please have a look and let me know how we might improve it. If anyone has benchmark data to add, post the info on this thread or send it to me in a PM and I will add. You can also submit a pull request if that’s easier for you.

I selected 14 datasets to start. There are probably too many datasets in the 0-200 photo range. It would be good to add a couple of larger sets, too. I am still filling in some details here, but wanted to get it out to everyone for comment.

I’m interested in people’s opinions on everything, but especially:

  • Does the presentation of this info make sense?
  • Does using GitHub for this make sense?
  • Suggested changes to the list of datasets?
  • Other parameters we should capture?
2 Likes

I think github is a fine place to start this, and I’m happy to pull it into an official repo. If you want larger datasets, check out the Red Cross ones here: American Red Cross projects

3 Likes

@smathermather-cm I think my benchmarks repo is ready for you to pull over as an official ODM repo, if you’re game. Let me know if you’d like to see any changes.

I still want to select and run a couple of larger datasets but I figure this is probably enough to start. If someone else could run a couple of these datasets and post their results here, I’d love to see how processing times compare across different systems.

1 Like

I’ll start running these as soon as processing space is available on my windows machine with 12gb.

Question, I have some absolutely immense datasets that I would be happy to share, is there any interest in adding some “insane” options as a means to set a ceiling? I’ve got some that go from 2,000 images -> ~15,000 images.

1 Like

@RainyRockies I would think so. 2-3 large datasets would be useful for finding the upper limits of some configurations, and also for benchmarking split/merge. I’m still working with a single processing node at the moment, but I’m planning to dig in to Cluster ODM in a few weeks.

I struggle with massive downloads sometimes, so splitting those datasets up into a few zipfiles might be helpful. Really appreciate the offer!

Check out downthemall for resumable downloads (if you are using a gui). For wget see e.g. https://www.cyberciti.biz/tips/wget-resume-broken-download.html and for CURL: https://www.cyberciti.biz/faq/curl-command-resume-broken-download/

That said – it is kinder to bundle things up into smaller zips. :slight_smile:

2 Likes

Cool! Fantastic work. This will be a great contribution.

I think you have to initiate the transfer and then I accept it. Let me know when you have initiated it, and I will also add you as a collaborator.

1 Like

OK, I found the transfer functions but I think I need slightly expanded permissions on the ODM account beforehand. I initiated the transfer but am getting an error from Github: "You don’t have the permission to create public repositories on OpenDroneMap "

I’m following these instructions. I think this is the key bit: “you must have permission to create a repository in the target organization.” I can hop on gitter to work through this, if helpful @smathermather-cm .

Added you to a benchmarking team. It’s been since WebODM since I have done this, so let me know if I need to make additional changes.

Success! https://github.com/OpenDroneMap/odm-benchmarks
Thanks @smathermather-cm

2 Likes

Looking good! Thank Corey! I’ll start running tests on my hardware and post them.

2 Likes