Largest dataset so far… to come

The Hangar Off Topic
#1

Talked to a colleague about drone photogrammetry, where going to test it on a project.

I estimated it will be about 2500 images, largest dataset I will be taking care of.

I’ll probably have to use split for this, I don’t think my 64gig ram and 8 cores will manage it as a hole. And I also estimated it will take at least 24h to process.

Anny recommendations for a larger set like this?

3 Likes
#2

Sounds exciting! Will you be able to share any screenshots or anything once it is done?

I think Gordon and Stephen will likely be your best resources for something like this at the moment.

2 Likes
#3

Yes, I might publish something if everything works out.

4 Likes
#4

You should be right at your limit. Set your swap == RAM or 1.5x RAM or even 2x RAM and you should be fine, as long as you aren’t turning anything up to ULTRA.

    "imageSizeMapping": [
        {"maxImages": 40, "slug": "s-2vcpu-2gb"},
        {"maxImages": 250, "slug": "s-4vcpu-8gb"},
        {"maxImages": 500, "slug": "s-6vcpu-16gb"},
        {"maxImages": 1500, "slug": "s-8vcpu-32gb"},
        {"maxImages": 2500, "slug": "s-16vcpu-64gb"},
        {"maxImages": 3500, "slug": "s-20vcpu-96gb"},
        {"maxImages": 5000, "slug": "s-24vcpu-128gb"}
    ]
2 Likes
#5

69 hours into 1320 images on ultra quality (CPU feature extraction 12MB images too big for 4GB CPU memory), 96GB RAM and pagefile 145GB, I’ve seen 99% of RAM used, and 67% of RAM + Paging so far

screen20220129
screen202201291920×1080 142 KB

1 Like
#6

I need ultra, quality is key here.

#7

Then you probably need more RAM. I haven’t done profiling on the differences, but almost all RAM failures we’ve seen on here have been folks turning those knobs. I recommend trying first medium, then high, then ultra, just so you have a sense at what quality output you are getting before it breaks.

1 Like
#8

But doesn’t split reduce ram usage?

1 Like
#9

For sure. You could split this into a few parts. Set your split size to maybe 1500 and turn the knobs up to ultra. But it’s good to run a the lower settings so you understand, when all is said and done, whether and when turning up the settings and increasing complexity through split-merge is worth it.

2 Likes
#10

What would be a good split-overlap for split 1500?

1 Like
#11

That really depends on your between-camera spacing. You probably want 3 or so cameras on each side of the overlap. So if your camera spacing is roughly 10m, you can set your overlap to 30-40 meters.

2 Likes
#12

If only WebODM could calculate the right split based on the hardware. But that might be hard.

2 Likes
#13

Agreed! This would be super useful, and probably not impossible. At that point, the flag might go away and it could automatically do it when needed.

2 Likes