Dataset URL (Dropbox, Google Drive, GitHub…):
https://sftp.kcmn.us/public/folder/xfvthwemm0orjfpm6o7hrg/shared-drone-projects
password: 6&t*XY
Number of images:
3000+
Size in MB:
230GB
EXIF GPS (yes/no):
Mostly (not rtk)
GCP (yes/no):
Mostly (ARUCO & MANUAL)
RTK (yes/no):
PPK
Comment/Description:
Hey All,
Wanted to release our first data sets resulting from discussion here: https://community.opendronemap.org/t/new-pilot-purchase-help-me-choose-proposal/19800
Wanted to also get some feedback on how to improve work flow, process, ID some bugs, and figure out where we can improve processing times/crashes. Since we are releasing the data under a AGPLv3 license, I thought this would be the best place for the topic. I’d be happy to post it elsewhere.
# Link (IT may want to take it down after ~1 year; message me if it doesnt work):
https://sftp.kcmn.us/public/folder/xfvthwemm0orjfpm6o7hrg/shared-drone-projects
password: 6&t*XY
# Quick work flow details (see README(s) for more):
- $38,500 investment (including multi-purpose server)
- Multi-purpose server & GPUs; expected min. 10-year lifetime to replace registered surveys during construction & routine volume estimates for compaction
- WHISPR Systems SkyScout
- GPS Ublox ZED-F9P
- Sony ILX-LR1 w/35mm
- Threadripper 5955WX w/16-core/32-thread
- 2x NVIDIA GA102GL [RTX A6000]
- WD BLACK SN850X HS 8000GB SDD drive
- Custom UIs for 4 ODM Nodes & Project management
- Backend using:
-
Docker to manage ODM Nodes, GPUs, custom GPS extraction (PPK)
-
PyODM to submit tasks to nodes
-
Nodered to orchestrate pre-processing & post-processing
-
Jupyter Notebooks to develop processing steps and coordinate data entry
-
Postgresql w/PostGIS to store GCP, PPK positions, ground shots for GCP
- Other software:
-
Python & Javascript
-
Rust Desk on Controller to manage Flight plans
-
SSHd DropBear to backup/sync flight plans
-
Quasar.dev to develop custom UI
-
OpenLayers to view output & custom GCP QAQC/data entry
-
Potree to view output
-
900-3000+ Photos per area
# Conceptual Goals
-
Want a system that did not require paid services, exposure to market forces & unncessary work flow changes.
-
Want to collect up to weekly volume calculations of stock piles, waste placement, and construction quantities. - Didn’t want to spend a lot of time with on-site survey equipment.
-
Want a fully remote control-operable work flow, with automated data intake & processing.
-
Did not want to juggle all the files manually, both for time saving and error control. Whenever you have to grab some files and put them somewhere, it’s easy to miss some, etc. See README for more user error details.
# Why not use WebODM?
We first start with WebODM. The primary reason it did not meet the goals was that the landfill only has a 5 MB connection, and all data is on-site server and thus can’t be uploaded remotely without an intermediary. It’s probably beyond the scope of WebODM, but for it to fit into our remote-first model, it would need some kind of S3 or other web-first connection. We want to avoid external services, as the goal is to have a system unaffected by internet and unaffected by errant “enshittification”.
# Work flow issues
-
Drone flights for various areas (potentially contiguous) at the site. We started with multiple SD cards then switched to a single large [Sony SD card SF-G256T]( ILX-LR1 : SD card Compatibility Information ) as recommended for the camera. The projects being released were put on multiple cards. The camera itself names the photos serially. This creates a problem when trying to interact with NodeODM (potentially WebODM) as it expects a list of unique photo names. So files from 2025-05-22 have three sets of identically named photos. These requires PPK geotxt files run off of just the file names, while the automated process created multiple folders. A new process which uniquely names the files should solve this particular problem. The database backend now tracks files bashed on sha256 hashes to ensure geo.txt generation refers to the correct images.
-
Lesson: filenames are really poor universal keys yet used universally.
-
Can it change: Probably not, even the guts of OpenSfM uses order #s of the filenames for pairs matching
-
Consider: Perceptual hashes might allow for a more robust system of management
-
System started as 4 NodeODM nodes with GPUs. Now only running 2 due to crashes from memory or Docker issues. Also found long running containers lose track of the GPU somehow. Seems like a lot of issues with Docker. Work flow requires inspecting each container for `nvidia-smi` output then restarts if necessary. Issue was opened here: GPU not found inside long running containers · Issue #1940 · OpenDroneMap/ODM · GitHub ; Solution would be to check the no-gpu flag if the GPU isn’t available and to warn the user that it’s unavailable. Another Docker related mystery: FPCFilter - v0.2 - double free or corruption (out) [docker] · Issue #4 · OpenDroneMap/FPCFilter · GitHub
-
Lesson: Docker has a lot of bugs at the higher end of processing for cpu/memory and may be causing errant crashes. It also has trouble managing the gpu over longer term
-
Can it change: Probably not. Docker is the best tech for the type of system orchestration ODM uses.
-
Consider:
-
GPU checks for running or re-running
-
Better documentation for preparing Docker to run GPU & high-memory work loads.
-
Memory logging alongside the benchmark system to notify when
-
More staged processing capabilities in NodeODM as it currently has no way to rerun crashes with modified config.
-
-
The drone has programming issues. It takes a photo after launch but doesnt orientate camera. GPS-less photos cause this issue: pair_selections drops all gps_neighbors, graph_rounds to 0 · Issue #42 · OpenDroneMap/OpenSfM · GitHub We’ve also had some battery charging issues, as the drone requires matched pairs charged within 5%. It writes EXIF data to photos on touch down, and takes several minutes. We’ve recommended they write the associated text files first before going through EXIF. The work flow works fine without any EXIF data for the photos. We also stripped the microusb while trying to trouble shoot issues with photo tagging & gps.
-
Lesson: Quirks are expected
-
Can it change: One of the pros of open hardware is not requiring the manufacturer to exist just for long term management. The downside is open source is more progressive enhancement then polished product.
-
Consider: More feedback and issues management
-
We decided to use ARUCO & Find-GCP and create fixed locations on posts since we’re flying the same project areas multiple times. While Find-GCP does a great job selecting locations, it’s centers are not always correct and require manual adjustment. Also, during construction, we took down a few of them, so we needed a work flow that could use manual GCP. Since we have survey equipment on-site, it’s easy for us to catalog distinct features with ground shots and locate them in photos. Since all phots should have GPS, I created a workflow to select photo candidates; this limits the amount of photos to look through to find GCPs.
-
Lesson: Automation gets us a lot, but manual procedures are still required, even if it’s QAQC
-
Can it change: Find-GCP probably could improve accuracy with imperfect centering; a algorithm that creates a square and aligns the center would seemingly work better than how it’s currently doing it (I did not investigate)
-
Consider: Find & locate controls for redundant GCP curation
-
PPK: The main selling point of the drone, beyond cost, was functional PPK. Why PPK? Well, setting up a base station is labor intensive. [RTKExplorer]( GitHub - rtklibexplorer/RTKLIB: A version of RTKLIB optimized for low cost GNSS receivers, especially u-blox receivers. It is based on RTKLIB 2.4.3. This software is provided “AS IS” without any warranties of any kind so please be careful, especially if using it in any kind of real-time application. Click on the "Releases" label below to see the latest Windows pre-release. ) has done a lot of good work to make PPK work flow, although we’re relying on very limit knowledge of the configuration options for using the MnCORS. It took way more time to troubleshoot the landfill’s local coordinate system.
-
Local coordinate system: Landfill uses a unique/poorly documented local coordinate system. It was setup in the 1960’s and seems to use a unique ellipse, along with plan shifts in XYZ. It’s unlikely that the current on-site Trimble rover & base system [landfill corrections file](equipment/2025-08-08%20landfill-corrections.pdf) was not matching up to the PPK. However, what matters was relative elevations and in that realm, we used WGS84 ellipse elevations to do ppk, then on the back end we’re using [CODEM]( GitHub - NCALM-UH/CODEM ) separately, but it appears to be the same as the beta ```–align``` flag. This allows for custom processing stage after ODM completes.
-
I tried to upgrade or deploy ODM outside of a container and the make files are so complex that I couldn’t debug what the errors were. I assume they’re dependency related. This does not seem like a easy to upgrade/improve project.
# Conclusion
Open Drone Map provides a stellar backend to generate weekly surveys of waste and to track construction progress. Docker appears to have issues when processing high end memory/CPU/GPU. WHISPR provides a great value for open hardware, although a lot of initial bugs required trouble shooting for atleast 6 months. A complex work flow takes us from raw GPS & photos to amazing 3d models. Reliability though, seems lacking and there’s so many layers of functionality that it could use a testing framework for each stage as suggested here: Implement testing infrastructure · Issue #207 · OpenDroneMap/ODM · GitHub
But it’s amazing it functions coherently and the real bugs seem to be upstream and PEBKAC.
Thanks all and let me know what you think.