Large dataset (3300 photos)

ODM
#1

I have a huge dataset, about 3000 pictures. I have no problems running it… but it took 48h on a m4d10xlarge aws instance.

I just need few generated files (odm_orthophoto.original.png odm_orthophoto.png, odm_orthophoto_log.txt, odm_georeferencing_model_geo.txt)

I have already seen some arguments to use, like --fast-orthophoto or --skip-3dmodel)

the fast-orthophoto results in a differente orthomosaic and the skip-3dmodel does not work.

I just need the png files and these two txt files. Can anyone help me to figure it out on how can I proccess it faster?

#2

If you want an ugly process that has you bouncing between checked out versions of ODM in order to use split-merge, I can provide that. Or else wait a month or so for @dkbenjamin to finish split-merge into the ODM-OAM workflow, that will be a cleaner process.

#3

Well, I am not into an ugly process, but I can try… how can I do it ??

Just to execute few tests… and then when @dkbenjamin finish the process I try it again.

Thaaank you for an answer.

#4

@accferronato the two options that come to mind are: --use-hybrid-bundle-adjustment and --use-opensfm-dense. The second will change your point cloud and perhaps the orthophoto, the first one should improve runtime with not much output change.

Please note that your orthophoto will never look the same between two runs, even with the same options, the texturing step is non-deterministic.

Run local bundle adjustment for every image added to
                      the reconstruction and a global adjustment every 100
                      images. Speeds up reconstruction for very large
                      datasets.

If you let us know the runtime by passing these other options we could perhaps make further suggestions.

#5

If the suggestions Piero suggested dont work, the process Stephen is referring to is documented here: http://docs.opendronemap.org/large.html

#6

Yes, but it’s broken on the current master, and has been since July. So to make it work, you need to do the following:

See post below instead (edited)
1 Like
#7

Thank you guys, I’ll try it out!! And I’ll post the results here!

1 Like
#8

Somehow this doesn’t work. I am working on writing a new version. Standby.

#9

Try this instead:

Remove final two lines in the run_all as:

#/usr/bin/env bash

RUNPATH="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"/../..
export PYTHONPATH=$RUNPATH:$RUNPATH/SuperBuild/install/lib/python2.7/dist-packages:$RUNPATH/SuperBuild/src/opensfm:$PYTHONPATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$RUNPATH/SuperBuild/install/lib

set -e

DIR=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )

python $DIR/setup.py "[email protected]"
python $DIR/run_matching.py $1
python $DIR/split.py $1
python $DIR/run_reconstructions.py $1
python $DIR/align.py $1
#python $DIR/run_dense.py $1
#python $DIR/merge.py $1

Now when we run this, it will run the structure from motion portion but not the multi-stereo view.

# Run split-merge 
cd ~/OpenDroneMap/scripts/metadataset
./run_all.sh ~/ODMProjects/projectname/

Now that this is complete, when need to clean up some paths in our list.txt in our mve subdirectory:

find ~/ODMProjects/projectname/submodels/submodel_????/opensfm/mve -name 'list.txt' -exec rpl "../images" "../../images" {} +

Now we can finish running through our dense point clouds:

for i in {0..9}
  do python run.py submodel_000$i --project-path ~/ODMProjects/projectname/submodels
done

for i in {10..99}
  do python run.py submodel_00$i --rerun-from smvs --project-path ~/ODMProjects/projectname/submodels
done

for i in {100..999}
  do python run.py submodel_0$i --rerun-from smvs --project-path ~/ODMProjects/projectname/submodels
done
1 Like
#10

The last step can also be rewritten as:

for i in {0..999}
  do python run.py submodel_0$(printf "%03.f" $i) --project-path ~/ODMProjects/projectname/submodels
done
#11

Haha. Of course. V. nice.

#12

Here’s a simpler solution that I will propose to OpenSfM-

Edit SuperBuild/src/opensfm/opensfm/large/metadataset.py Line 155-156:

                    # dst_relpath = os.path.relpath(dst, submodel_path)
                    txtfile.write(dst + "\n")

Then you can run as normal. This changes to relative paths to absolute in the images text file.

2 Likes
#14

Does split-merge feature creates point cloud and 3d mesh too?..apart from orthophoto?

#15

The aim is for the merged outputs to be merged ortho, dems, and point cloud. If there is a good specification out there for massive hierarchical 3D meshes as well, we can build out a solution for that as well, but in the known timeline (Aprilish) it will be the first 3.

1 Like
#16

I was trying the split-merge feature with above modifications.

I ran the run_all.sh successfully. It created two submodels.

Later I ran below command:
for i in {0…2}
do python run.py submodel_0$(printf “%03.f” $i) --project-path ~/ODMProjects/projectname/submodels
done

But I got this error:

[INFO] Running SMVS Cell
[DEBUG] running /home/garlac/myodm/SuperBuild/src/elibs/mve/apps/makescene/makescene /home/garlac/ODM_input_datsets/Tadepalli_P1_Rsized/submodels/submodel_0002/opensfm/mve /home/garlac/ODM_input_datsets/Tadepalli_P1_Rsized/submodels/submodel_0002/smvs
MVE Makescene (built on Jan 9 2019, 17:23:58)
Info: Detected Noah bundler format.
Reading Bundler file (573 cameras, 552196 features)…
Recognized 573 original images from Noah’s Bundler.
Creating output directories…
Saving bundle file…
Writing bundle (573 cameras, 552196 features): /home/garlac/ODM_input_datsets/Tadepalli_P1_Rsized/submodels/submodel_0002/smvs/synth_0.out…
Processing view view_0000.mve…
Error loading: /home/garlac/ODM_input_datsets/Tadepalli_P1_Rsized/submodels/submodel_0002/opensfm/mve/…/images/vbgh(803).JPG
Traceback (most recent call last):
File “run.py”, line 47, in
plasm.execute(niter=1)
File “/home/garlac/myodm/scripts/smvs.py”, line 70, in process
system.run(’%s %s %s’ % (context.makescene_path, tree.mve_path, tree.smvs))
File “/home/garlac/myodm/opendm/system.py”, line 34, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

Any Help on how to proceed further? Thank you.

#17

This solutions didn’t make it faster… I just need 4 files in the end of the proccess:

odm_orthophoto.original.tif
odm_orthophoto.png
odm_orthophoto_log.txt
odm_georeferencing_model_geo.txt

I don’t need any 3d model ou file. Just these 4 files…

how could I run just 2d stuff and reduce processes in the execution ?

#18

This is a common question. The best solution is to use the --skip-3dmodel flag. It sometimes degrades the quality of the orthophoto but it can speed things up.

1 Like
#19

Ok. I am already using it.

But, is there any step on texturing or anorther thing that I could skip it too ?

#20

–skip-3dmodel

#22

Please open a new topic, do not reply to old ones.