CPU/RAM for processing 2DFull - 2677 Images

General Help
1

Really new to OpenDroneMap. I’m looking for recommendations on the ideal CPU/RAM needed to process 2677 images 2DFull. I have one running now with 96 vCPU/384 GB RAM, the process is still running and at 19 hours so far. I ran a fast-ortho same specs it took 21 hours to complete. With both options I’m looking for ways to get the process down faster. Should I increase CPU or RAM or both?

1 Like
2

Hmm… What speed is the vCPU? Some parts of our pipeline benefit from high clockrate more than core count since they are single-threaded (though possibly spawned in parallel).

1 Like
3

Hi Saijin,
3.3ghz

1 Like
4

That’s pretty quick!

I assume you’re not getting any Exit Code 1 errors, right? So no Out-Of-Memory errors yet?

Maybe try upping the vCPU speed first (if possible, not sure what your host offers), and if not, slightly increasing vCPU count.

5

The run with 96 vCPU/384 GB RAM was my first successful run. But took 30 hours :slight_smile: . I tried a z1d.12xl using the following options too:

–dem-resolution 1.0
–dsm
–dtm
–feature-quality high
–mesh-octree-depth 12
–mesh-resolution 300000
–orthophoto-resolution 1.0
–pc-geometric
–pc-quality high
–use-3dmesh

That took 29:33 to process. A z1d.12xl has 4.0ghz, 48 vCPUs and 384GB of memory. Now I’m thinking if I run in a cluster and split, parallelize if I can Splitting Large Datasets — OpenDroneMap 3.1.7 documentation. Any thoughts here? I’m trying hard to at least get this processing at least under a day. Drone2Map took ~18 with a g4dn.12xl (4GPU, 48 vCPUs, 192GB RAM).

1 Like
6

Split-merge should help at lot up front, but you will have a slowdown at the end when all the work needs to be composited on one machine…

Hmm… Could be worth quantifying for your datasets

7

A couple of points of reference-

2214 images with resizing to 1/3 linear size, took 7h 41m with 96GB RAM, i7 8 core 3.8 GHz Intel processor.
It never uses more than about half the RAM though.

[Nombi - 15/09/2021] 2214 07:41:52

Created on: 20/09/2021, 12:32:48

Processing Node: node-odm-1 (manual)

Options:** dsm: true, dtm: true, optimize-disk-space: true, pc-quality: high, pc-rectify: true

1194 images took 22:47:54, although due to large elevation variations, there were gaps in the ortho due to not enough overlap on higher ground

Processing Node: node-odm-1 (manual)

Options: dem-resolution: 2.0, dsm: true, dtm: true, feature-quality: ultra, min-num-features: 10000, optimize-disk-space: true, orthophoto-resolution: 2.0, pc-quality: high

8

I’m trying to reproduce Drone2Map 2d full template run with the same data set to possibly save cost when running on AWS. My thoughts are to use less expensive, non-GPU intensive instances and get the job done in less time than with Drone2map. The 2D full template with Drone2Map output (per their documentation):

The options I’m currently using are:
–dem-resolution 1.0
–dsm
–dtm
–feature-quality high
–mesh-octree-depth 12
–mesh-resolution 300000
–orthophoto-resolution 1.0
–pc-geometric
–pc-quality high
–use-3dmesh

I may also need to play around with the options to match that of a 2d full with Drone2Map.

1 Like
9

Yesterday started my testing with distributing the dataset using ClusterODM. I was on track with shaving off ~10 - 14 hours of processing time when I hit an error that crashed my task(s).

Two of the submodels did not complete on two of the processing nodes. The logs states the unable to find EXIF file. Wonder what could have caused this behavior.

[WARNING] LRE: submodel_0002 failed with: (f8a03cef-4500-4f6c-bcf1-d0467df83a1a) failed with task output: exif = {image: self.load_exif(image) for image in self.images()}
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 981, in
exif = {image: self.load_exif(image) for image in self.images()}
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 579, in load_exif
with self.io_handler.open_rt(self._exif_file(image)) as fin:
File “/code/SuperBuild/install/bin/opensfm/opensfm/io.py”, line 1349, in open_rt
return cls.open(path, “r”, encoding=“utf-8”)
File “/code/SuperBuild/install/bin/opensfm/opensfm/io.py”, line 1341, in open
return open(*args, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: ‘/var/www/data/f8a03cef-4500-4f6c-bcf1-d0467df83a1a/opensfm/exif/DSC02038.JPG.exif’
Full log saved at /var/www/data/3f857ff8-62f1-4af0-bc25-11a3fcd10b4c/submodels/submodel_0002/error.log

[WARNING] LRE: submodel_0003 failed with: (43cb7889-c022-40a4-b736-d9a4fc70556b) failed with task output: exif = {image: self.load_exif(image) for image in self.images()}
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 981, in
exif = {image: self.load_exif(image) for image in self.images()}
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 579, in load_exif
with self.io_handler.open_rt(self._exif_file(image)) as fin:
File “/code/SuperBuild/install/bin/opensfm/opensfm/io.py”, line 1349, in open_rt
return cls.open(path, “r”, encoding=“utf-8”)
File “/code/SuperBuild/install/bin/opensfm/opensfm/io.py”, line 1341, in open
return open(*args, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: ‘/var/www/data/43cb7889-c022-40a4-b736-d9a4fc70556b/opensfm/exif/DSC02561.JPG.exif’
Full log saved at /var/www/data/3f857ff8-62f1-4af0-bc25-11a3fcd10b4c/submodels/submodel_0003/error.log
[INFO] LRE: Cleaning up remote task (43cb7889-c022-40a4-b736-d9a4fc70556b)… OK
[INFO] LRE: No remote tasks left to cleanup
Traceback (most recent call last):
File “/code/run.py”, line 54, in
retcode = app.execute()
File “/code/stages/odm_app.py”, line 130, in execute
raise e
File “/code/stages/odm_app.py”, line 94, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 340, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 321, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 226, in process
lre.run_toolchain()
File “/code/opendm/remote.py”, line 57, in run_toolchain
self.run(ToolchainTask)
File “/code/opendm/remote.py”, line 251, in run
raise nonloc.error
pyodm.exceptions.TaskFailedError: (43cb7889-c022-40a4-b736-d9a4fc70556b) failed with task output: exif = {image: self.load_exif(image) for image in self.images()}
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 981, in
exif = {image: self.load_exif(image) for image in self.images()}
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 579, in load_exif
with self.io_handler.open_rt(self._exif_file(image)) as fin:
File “/code/SuperBuild/install/bin/opensfm/opensfm/io.py”, line 1349, in open_rt
return cls.open(path, “r”, encoding=“utf-8”)
File “/code/SuperBuild/install/bin/opensfm/opensfm/io.py”, line 1341, in open
return open(*args, **kwargs)
FileNotFoundError: [Errno 2] No such file or directory: ‘/var/www/data/43cb7889-c022-40a4-b736-d9a4fc70556b/opensfm/exif/DSC02561.JPG.exif’
Full log saved at /var/www/data/3f857ff8-62f1-4af0-bc25-11a3fcd10b4c/submodels/submodel_0003/error.log

1 Like
10

Have you pulled the images it flagged to inspect them?

11

I actually hit another error before the error mentioned above.

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying.html
[WARNING] LRE: submodel_0002 failed with: Child returned 1

The https://docs.opendronemap.org/flying.html page no longer exits. I’m going to try some pre-processing using XnConvert next with my last attempt. I’m using r5a.16xlarge (64 vCPUs, 512RAM) instances for my (4) nodeodm processing nodes. Clusterodm running and webodm running too all separate instances.

2677 images

options used:
dem-resolution: 1
dsm: true
dtm: true
feature-quality: high
mesh-octree-depth: 12
mesh-size: 300000
orthophoto-resolution: 1
pc-geometric: true
pc-quality: high
sm-cluster: http://x.x.x.x:3001
split: 700
split-overlap: 120
use-3dmesh: true

Log file can be found here.

1 Like
12

Are you able to pull this log as well?

Full log saved at /var/www/data/9ecef7b2-d43c-4987-bf16-5ef42c6a3fae/submodels/submodel_0003/error.log
13

@Saijin Let me give you the latest and greatest.

Console Log: here
Submodel 0001 error log: here
Submodel 0002 error log: here
Submodel 0002 error log: here

1 Like
14

Can you try without split/merge?

15

Also, try to update your nodes; I think this should be fixed with one of the changes applied here a few days ago: Point cloud, orthophoto boundaries working · OpenDroneMap/[email protected] · GitHub

We were erroneously detecting a GCP where there wasn’t one, causing the error from the submodels log.

2 Likes
16

@Saijn I’ve been trying the without a split/merge on various memory optimized instances. In one case; the 2D full took 30 hours, another 41 hours. The process takes to long, I’m aiming for a less than 18 hours or less turn around. This is what I can do with drone2map with the same dataset using a single g4dn.12xlarge 48 vCPU, 4 GPU, 64 GPU RAM and 192 RAM.

@ Piero I’ll try the latest and greatest update.

Thank you booth

1 Like
17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.