Have been having trouble processing a recent dataset with ~1441 images with WebODM and Lightning. The errors have either been “not enough memory” or “cannot process dataset”. It happens about the 10 to 12 hour mark.
I have tried about 7 processes with several different settings based on my limited experience and searches on this forum. The only one that worked is to resize the images, but then my GSD was higher than I wanted.
I have posted some sample data here in the following google drive folder if anyone feels like taking a look. I would like to understand what it is about this dataset that is causing the problem and if there is any way to process it with maximum possible resolution. Included in the folder is console output for 3 of the failed attempts, sample raw images unedited, and the orthophoto from the 1 attempt that did work with the resized images (orthophoto has been compressed with jpg post-process).
We can sometimes still OOM on Lightning, especially when being fed the newer quad-bayer imagery which is just massively inflated megapixel count vs actual resolved detail (and therefore, significantly higher resource requirements).
We see this sometimes with the 48MP quad-bayers which can (and should be) safely downsampled to 12MP.
Do you know if your sensor is a pixel-binning sensor?
When running mapping missions with this drone in DJI Pilot 2 application, it appears it automatically applies what it thinks are the optimal mapping settings.
Max resolution of the camera is used and doesn’t appear I can override it. Mechanical shutter is enabled and shutter speed seems to hover around 1:500, if that makes any sense? (I’m not a camera expert, so not sure if I am using correct number)
Flight speed is set at the software recommended level which was 12 m/s
Altitude was 76 m above takeoff, and GSD 2.04 cm/pixel. Both overlaps were set at 84%.
Terrain following was not used, and the site does have some elevation change. There is one area of the site that has fairly generic crop pattern which I wonder could be causing an issue.
You should be able to find all the info in EXIF.
Not really an issue for flat ground in your case, but at 12m/s you don’t want to be exposing for any longer than 1/500sec, as image motion will be an issue. As it is, there is 2.4cm motion at 1/500, slightly more than your GSD, for 76m AGL.
If there are areas of higher ground, then overlap will be reduced, and images may be slightly blurred, which can lead to problems. Lower ground areas will have more overlap and less image motion on the sensor.
Areas with elevation change were depressions so should improve overlap like you said.
I will look through the exif data when at computer next. There are also sample images in that Google drive link.
As far as the exposure speed, it doesn’t seem like DJI let’s me change it at all, but I’ll look again. I think 1/500 is the lowest I have seen. I will look through the exif and find the range for these images.
Sorry for the slow reply. Have been working days (collecting more data) and spending nights learning how to run WebODM on AWS EC2. I didn’t even know what EC2 was a couple days ago. I had a colleague beginning to hunt for a used server box for us to set up for ODM processing, but that excellent documentation on your Github made it fairly easy to setup WebODM on EC2 and I think that is a far better solution for us when we can’t use Lightning.
Back on topic in regard to this particular 1441 image dataset, it is at least a relief that you are getting the same error as me, so it is something about the dataset rather than something I am doing incorrectly on my end. Given that the dataset processed fine with the resized images, I am really starting to wonder if certain pattern areas of the farmers field are just giving the reconstruction some difficulty? Could it be one of those scenarios where feature quality to ultra would help? Perhaps I should have flown a bit higher for the entire dataset or perhaps done a second flight at higher altitude?
I can definitely share this data on Github. I have a Github account, but I don’t have any experience making contributions. Can you point me in the right location where I should place the images so I don’t make a mess?
I tried to process this dataset and processing “hung” after a couple of days. I left it a few more days to see if it was just slow processing, but it definitely had hung and with no error reported.
The last message in the log was:
2023-01-23 22:52:55,320 INFO: DJI_20230111125751_0004_V.JPG resection inliers: 5472 / 7549
2023-01-23 22:52:55,404 INFO: Adding DJI_20230111125751_0004_V.JPG to the reconstruction
I checked and didn’t find any blurred images or images with divergent GPS EXIF data.
I processed it with a single node in a cluster - that node being limited to using 20 concurrent CPU processors and maximum 4000 images. The server has 2 x 20 core Xeon CPUs and 512GB RAM and an Nvidia K80 (comprises 2 x 12GB GPUs).
Some statistics about the maximum resource consumption during this processing task (up until I cancelled it):
62 GB disk space
0 Swap space
115GB RAM Cache
And adding some OpenSFM processing time details:
Detect Features: 5 min
Match Features: 56 min
Create Tracks: 1 hour 16 min
Reconstruct: hung after 16 hours
My resource consumption logs indicate it actually entered a “hung” state after less than 24 hours.
I just noticed unexpected log messages during OpenSFM “match_features”:
INFO: Altitude is negative (-1084.0651326475377) : viewing directions are probably divergent. Using default altitude of 1.0
INFO: Altitude for orientation based matching 1.0
This is unexpected because the image EXIF data altitudes are all within 334 metres +/- 0.5m.