New Project - DroneDB (request for thoughts)

Lately I’ve been thinking a lot about what’s inefficient about the current state-of-the-art for publishing/sharing aerial images. I think Google Drive / Dropbox / USB drives and zip files have served us well, but I think there’s room for improvement.

So I started working on https://github.com/uav4geo/DroneDB

I think raw images captured from a drone are currently underutilized in analysis and they are often lost in archives never to be looked up again after they have been processed.

When working with drone data, I (personally) want to be able to:

  • Quickly access my data or a subset of it (without downloading all of it)
  • View where my data was captured, at what time (accurately), what flight pattern was used, what was the image footprint and did I have enough overlap, ground sampling distance metrics
  • Recreate and display the flight
  • Compare different datasets
  • Check for errors, corrupt images
  • Generate reports about my dataset (to publish along with the dataset itself)
  • Share it (locally or via internet)
  • Add notes and labels to my images / folders.
  • Have the option to not use a cloud service, because trust and speed.

Just wondering what people’s thoughts are and if these points are shared by other folks in the drone industry (or if you have your own personal take)?

2 Likes

Piero,

I think that’s a fantastic idea. It’s actually something I had tried to setup In a manner for myself using an existing package but just couldn’t get it to run properly.

I’m assuming that you’re thinking along the lines of a web-ui again? Sort of Lightroom for drone and photogrammetry users.

The report generation would be brilliant. I actually find it quite awkward to go back and find out the parameters I used to capture the images.

In my case it would the sort of thing I’d be happy to run on my new NAS server.

Longer term would you consider including flight logs? I appreciate that parsing many different manufacturer formats could be a pain. It does however link in with the capture pattern, and could be expanded to be a self hosted version of of the likes of DroneLogBook or AirData.

1 Like

Hey @dsandson :hand:

Yes there will be a web component to access the database (along with a desktop application for managing the database also).

When you say “find out the parameters you used to capture”, which specifically do you mean (overlap? flight speed?)

Definitely looking to add support for flight logs (GCPs, orthophotos, elevation models and point clouds are in the plans as well). By chance would you have a copy of a few logs that you could share (I don’t have much experience with flight log formats)? Would be helpful to have a few examples.

Piero,

For flight parameters I’d say at a minimum flight speed, front lap, side lap, and any common image capture parameters (iso, shutter speed, white balance), perhaps with user fillable options to add in basic weather params (cloud cover, wind speed). And perhaps total flight time and batteries used if flight logs are added in.

I’ll dig out some flight logs. I have access to a DJI Phantom 3 Pro and a Yuneec H520 & E90 camera. One issue with logs you will come across is each mapping app for DJI products produces a slightly different log format. And theres quite a few different mapping apps!

@pierotofy

Link is below to as many different log types as I can can find. This includes some of the major DJI flight apps;

  • DJI Go (not for P4P or Mavic 2)
  • DJI GS Pro
  • Litchi
  • AutoPilot
  • Pix4D Capture
  • Map Pilot (MapsMadeEasy)

Also, Ulog & Tlog formats from the Yuneec H520

https://drive.google.com/open?id=1jWozcsacb_52NW3evAoLhHHND8Ov1eTt

1 Like

Fantastic @dsandson. @pierotofy — reach out via email, and we can send logs from sensefly, and probably some pixhawk logs if you need them and maybe mavic.

On what scale or how much of the colourspectrum from a raw does the WebODM use (if it could read raw images)?
Hence JPG/TIF vs Raw colour spectrum.
I know raw from working in lightroom you get much large room to play with wider range of colours, so i just assume a raw file would give more data to play with. But also huge files…

Sorry, i noticed this was not the topic. More kind of metadata

This is gold @dsandson :+1:

DJI flight logs are encrypted, but I like a good challenge… :man_scientist:

1 Like

Ah I downloaded them again from DroneLogBook. Perhaps that’s the issue. I’ll look for originals tomorrow.

Actually I think they are simply encrypted by the DJI apps. But if you find the originals double-check!

Edit: the format is detailed in Documenting the Format of DJI Log Files

Correction, they are “scrambled”, not encrypted.

Piero, thought I’d add that the Yuneec H520 runs the PX4 flight control stack, which is apparently open source.

Found the following python script on the PX4 GitHub page, might be of use;
https://github.com/PX4/pyulog

2 Likes

This sounds like a great project! I have actually been looking for something similar.

Would you have any use of logs from P4RTK, P4P or Sensefly?

Hey @hugo :hand: actually yes! It would help implementing https://github.com/uav4geo/DroneDB/issues/12

1 Like

I think the 3DR Solo has a slightly different log format from the standard ArduPilot logs, so I’ve attached a link to a few log dumps (dataflash and Telemetry Log) for your perusal.
https://1drv.ms/u/s!AvMZEGXuAwQzqNMn5KL8pm7ADImyEw?e=JzdSZy

2 Likes

Sorry for not getting back earlier! Here is a link to P4RTK logs exported from the controller to the SD-card. I am not sure what they contain, so I am not sure they will help.

The transfer will be deleted in 4w.
https://we.tl/t-YFdIgkRBlK pw: odm

1 Like

Hi Piero,
I’m a fan of ODM. I bought the installers and have it running on my pc.
The list of points you shared are very interesting. I am using ODM for agriculture. I’m mapping crops with multispectral, RGB and thermal. I have couple of features I’d love to find on ODM:

  • Tree counting
  • Tree measurement (height / canopy width / volume)
  • Option for selection different types of multispectral cameras (RGNIR / RNIRB)
  • Besides the NDVI map, a simplified version of it, an AREAS map.

I am a user, not a developer, and I’ve not idea of how feasible, easy or difficult it can be to have those features. What do you think?

2 Likes

Hi @pierotofy,

Not sure if this could help but a buddy and I have a project going that allows to parse DJI txt flight logs (such as from DJI Go) and convert those to kml files. Perhaps the parsing could be useful to you for your project?

I’ve been wondering about adding the ability to review flight logs on an offline map and view locations along the flight path where photos were taken to get coordinates, etc.

Hey @MorganCaldbeck it would be useful, yes.

So far I’ve found this implementation http://djilogs.live555.com/, which looks promising.

Other research findings: Documenting the Format of DJI Log Files

What you are describing is definitely part of the goals of the project.

@pierotofy

Yeah those certainly look helpful. This is a link to a repo I found that is set up to parse those files already.

Are you planning on hooking your project into ODM?