Go check the readme.md as it pretty much explains everything visually.
Hi,
I’d like to share with the ODM community an open source DIY project we worked on with my dad for 3 years to help the botanists of the village get multispectral images on a very low budget. (you can read the story below if you’re interested).
It took me a while to discover ODM but I was very pleased to see the results & such a beautiful open source project . We use ODM as the multi-spectral stitching post-processing engine with the infamous --skip-band-alignment option as everything is already done in IRdrone.
Quickly introducing myself: I’m Balthazar Neveu from France, an image processing engineer with about 11 years of experience in the field of computation photography (DxO , GoPro , Xiaomi now) .
End of 2019, it all started when my dad ask me if we could remove the IR filter from a camera to help the botanists in his village take Infrared pictures with a drone…
I thought this was going to be a sort of easy project but nearly 3 years later, we have a full DIY hardware+software solution to capture visible + NIR images
from a DJI drone
an action camera SJCAM with removed optical filter.
The concept is simple: why adding a dedicated visible camera when you could simply be using the gimbal + the DJI camera? it brings the opportunity for incredible visible images… you just need an additional NIR camera and it saves some weight!
Everything is shot in raw/dng to access linear data, aligned geometrically with an image registration engine. RawTherapee is used to perform the bayer to rgb processing.
Every fly has to be shot in timelapse. Synchronization has been painful but we managed to get a software solution + a dedicated synchronization procedure using aruco detection (spin the drone or spin the aruco).
Software was made “easy” for the botanists to use (.bat files under windows with file selection) but there’s also a docker version and CLI interfaces.
Features
What does the core software do?
Choose paired images (with minimal delay)
Raw processing is a wrapper for RawTherapee
Analyze exif to retrieve gimbal angles & provide a coarse estimation of the pitch/yaw camera misalignments
Align images using global rigid estimation (distorsions and lens shading corrections included), refined local alignment.
Outputs 4 linear tifs (R, G, B , NIR) and copy the original metadata exifs back to the output images. Ready to be processed by ODM multispectral.
What’s the preparation phase for ?
Synchronize two timelapses from the 2 cameras . You can fly the drone above a Aruco chart and make the drone spin around.
Because spinning the drone around requires a bit of attention and the drone pilot may be focused elsewhere, my dad made a spinning table where the aruco is rotating so you can keep the drone in stationary fly.
If anyone is interested, I’ll be more than happy to share more information.
Again, thank you so much for ODM ! It made the last piece of the problem much easier, I didn’t have to develop orthophoto mapping back from scratch .
Whoever wrote opendm/multispectral.py which I discovered way later than starting the IRdrone project may be interested in my following notes.
Multispectral alignment is a tricky problem.
When I first started this projects, I thought it wouldn’t be too hard to match the fisheye NIR image with the DJI visible images based on SIFT . That was a mistake, it worked for a while on the test images shot during winter but when spring came around… almost everything stopped working.
We polished the synchronization strategy
I experienced with many different approaches for multispectral registration.
tried to tweak SIFT parameters, tried to get SIFT describe absolute gradients rather than the rgb & NIR image → it didn’t work either
tried phase correlation + Fourier Mellin (log polar) as an article said it was robust to the multispectral component. This was a misleading information in my experience! Registration of visible and near infrared unmanned aerial vehicle images based on Fourier-Mellin transform. Using the log/polar representation is an interesting component, but using Fourier phase shift estimation instead of block matching did not bring any advantage for the multispectral difficulty / contrast invariances.
I ended up using minimizing the NTG (normalized total gradients) distance by brute force block matching (find the argmin of the matching cost function),
you end up with a vector field. fit an affinity on it to find a rigid global transform
I didn’t finish the iterative gradient descent mentioned in the paper … short story : I became a dad in-between). For the record, nowdays I’d rather code this with a pytorch differentiable grid warp & do the gradient descent this way on the NTG loss rather than recoding everything from scratch.
I’m aware CNN based methods should be more suitable to that task, including a recent CVPR 2022 paper but improving all this requires so much time RGB Multispectral matching with deep leargning.
I’m therefore highly open to discussions regarding this excellent topic aswell (spent already much time on it).
Very cool project! Thanks for sharing and keeping it Open Source!
I think developing low cost solutions is definitely essential to get drone technology into my area which is farming. Because NDVI is used massively in this field, this project immediatly caught my attention. Until now I only fly with a Mavic Mini with a RGB sensor as a hobby on our farm to create some biomass maps. Sadly I am afraid that your hardware and workflow setup will be too sensible and prone to error for our stressful farming conditions… But I will definitely follow your developments and share the project with my colleagues at university!
Hi Steffen, thanks for the kind reply & thanks for sharing
Data processing is the burden part, it certainly takes a bit of extra time & extra storage compared to a “dji” visible only operation …
It’s been tested in pretty rough field conditions by the botanists we developped it for (they make maps of terrains to plan wind generators installation to avoid interfering if there are some species to protect - Infrared images already helped them to find interesting things they would have not seen by analyzing they).
I’m interested in what you mean by “stressful farming conditions”? (wind? very little time?) thanks
