Dataset with 1441 images fails unless images are resized

Have been having trouble processing a recent dataset with ~1441 images with WebODM and Lightning. The errors have either been “not enough memory” or “cannot process dataset”. It happens about the 10 to 12 hour mark.

I have tried about 7 processes with several different settings based on my limited experience and searches on this forum. The only one that worked is to resize the images, but then my GSD was higher than I wanted.

I have posted some sample data here in the following google drive folder if anyone feels like taking a look. I would like to understand what it is about this dataset that is causing the problem and if there is any way to process it with maximum possible resolution. Included in the folder is console output for 3 of the failed attempts, sample raw images unedited, and the orthophoto from the 1 attempt that did work with the resized images (orthophoto has been compressed with jpg post-process).

https://drive.google.com/drive/folders/1t_YFYxaXSlgUK-JW7_Nqp4TL9_f8wmzB?usp=share_link

1 Like

Not enough RAM and/or virtual memory.
What are your system specifications?

1 Like

This was using Lightning, 1500 image tier

1 Like

We can sometimes still OOM on Lightning, especially when being fed the newer quad-bayer imagery which is just massively inflated megapixel count vs actual resolved detail (and therefore, significantly higher resource requirements).

We see this sometimes with the 48MP quad-bayers which can (and should be) safely downsampled to 12MP.

Do you know if your sensor is a pixel-binning sensor?

1 Like

Not sure. I will look. So far I have found the following specs:

DJI Mavic 3 Enterprise

4/3 CMOS, Effective pixels: 20 MP

FOV: 84°
Format Equivalent: 24 mm
Aperture: f/2.8-f/11
Focus: 1 m to ∞

ISO Range: 100-6400

Shutter Speed:
Electronic Shutter: 8-1/8000 s
Mechanical Shutter: 8-1/2000 s

Max Image Size: 5280×3956

Still Photography Modes

Single: 20 MP
Timed: 20 MP
JPEG: 0.7/1/2/3/5/7/10/15/20/30/60 s
JPEG+RAW: 3/5/7/10/15/20/30/60 s
Smart Low-light Shooting: 20 MP
Panorama: 20 MP (raw image)

1 Like

What are the actual settings for the images you took- did you use 4:3 ratio image, or 16:9, shutter speed, flight speed, GSD etc?

1 Like

When running mapping missions with this drone in DJI Pilot 2 application, it appears it automatically applies what it thinks are the optimal mapping settings.

Max resolution of the camera is used and doesn’t appear I can override it. Mechanical shutter is enabled and shutter speed seems to hover around 1:500, if that makes any sense? (I’m not a camera expert, so not sure if I am using correct number)

Flight speed is set at the software recommended level which was 12 m/s

Altitude was 76 m above takeoff, and GSD 2.04 cm/pixel. Both overlaps were set at 84%.

Terrain following was not used, and the site does have some elevation change. There is one area of the site that has fairly generic crop pattern which I wonder could be causing an issue.

2 Likes

You should be able to find all the info in EXIF.
Not really an issue for flat ground in your case, but at 12m/s you don’t want to be exposing for any longer than 1/500sec, as image motion will be an issue. As it is, there is 2.4cm motion at 1/500, slightly more than your GSD, for 76m AGL.

If there are areas of higher ground, then overlap will be reduced, and images may be slightly blurred, which can lead to problems. Lower ground areas will have more overlap and less image motion on the sensor.

1 Like

Areas with elevation change were depressions so should improve overlap like you said.

I will look through the exif data when at computer next. There are also sample images in that Google drive link.

As far as the exposure speed, it doesn’t seem like DJI let’s me change it at all, but I’ll look again. I think 1/500 is the lowest I have seen. I will look through the exif and find the range for these images.

2 Likes

I tried to download, but my internet connection is not up to the task of large downloads today. I haven’t been able to update to the latest WebODM build either.

1 Like

Luc,

Are you able to share the full dataset?

I’m wondering if we might not have knocked this error back down in a recent update that has not made its way to Lightning quite yet.

Gladly! Here are all 1441 images in a zip file. It’s about 19.5gb so I had to use my OneDrive instead of Google. Thanks for taking a look.

Taking a random sample of some of the images the shutter speed does seem to vary between 1/500 and 1/1000.

1 Like

Is this something you’d be able to share on our GitHub so Piero can try and track down where exactly this is failing?

I’m getting the same failure locally as well as on Lightning.

I’m going to see about quickly pre-processing the images a bit and re-running it.

@Saijin_Naib
What quantity of images from this 1441 image dataset would you consider the minimum needed to be a representative sample to try processing…50, 100, 200?

1 Like

Great question… The scenery is pretty variable, so hard to say depending upon which area of the survey you are going to use.

Probably 50ish over the house, but I’m not sure the dirt and snow field will be able to reconstruct just by itself, and it may require more.

1 Like

Sorry for the slow reply. Have been working days (collecting more data) and spending nights learning how to run WebODM on AWS EC2. I didn’t even know what EC2 was a couple days ago. I had a colleague beginning to hunt for a used server box for us to set up for ODM processing, but that excellent documentation on your Github made it fairly easy to setup WebODM on EC2 and I think that is a far better solution for us when we can’t use Lightning.

Back on topic in regard to this particular 1441 image dataset, it is at least a relief that you are getting the same error as me, so it is something about the dataset rather than something I am doing incorrectly on my end. Given that the dataset processed fine with the resized images, I am really starting to wonder if certain pattern areas of the farmers field are just giving the reconstruction some difficulty? Could it be one of those scenarios where feature quality to ultra would help? Perhaps I should have flown a bit higher for the entire dataset or perhaps done a second flight at higher altitude?

I can definitely share this data on Github. I have a Github account, but I don’t have any experience making contributions. Can you point me in the right location where I should place the images so I don’t make a mess?

1 Like

No worries!

I think for now if you could open an issue on our ODM repo, that’s a great start:

You can share the photos and logs wherever is most convenient for you, provided it can be accessed easily (not by request/invite)

@saigin @piero
I tried to process this dataset and processing “hung” after a couple of days. I left it a few more days to see if it was just slow processing, but it definitely had hung and with no error reported.
The last message in the log was:
2023-01-23 22:52:55,320 INFO: DJI_20230111125751_0004_V.JPG resection inliers: 5472 / 7549
2023-01-23 22:52:55,404 INFO: Adding DJI_20230111125751_0004_V.JPG to the reconstruction
I checked and didn’t find any blurred images or images with divergent GPS EXIF data.

I processed it with a single node in a cluster - that node being limited to using 20 concurrent CPU processors and maximum 4000 images. The server has 2 x 20 core Xeon CPUs and 512GB RAM and an Nvidia K80 (comprises 2 x 12GB GPUs).

Some statistics about the maximum resource consumption during this processing task (up until I cancelled it):
62 GB disk space
0 Swap space
234GB RAM
115GB RAM Cache

And adding some OpenSFM processing time details:
Detect Features: 5 min
Match Features: 56 min
Create Tracks: 1 hour 16 min
Reconstruct: hung after 16 hours

My resource consumption logs indicate it actually entered a “hung” state after less than 24 hours.

I just noticed unexpected log messages during OpenSFM “match_features”:
INFO: Altitude is negative (-1084.0651326475377) : viewing directions are probably divergent. Using default altitude of 1.0
INFO: Altitude for orientation based matching 1.0

This is unexpected because the image EXIF data altitudes are all within 334 metres +/- 0.5m.

1 Like

I see this weird message very often, even though the EXIF altitudes are all good.

1 Like

I haven’t found any logical explanation for this message. Might be a red herring…

2 Likes