Split-merge issues

Hi all,
I am trying to process a set of 3000 images on a 128Gb RAM server using NodeODM (loaded via docker in a linux server)
When I upload the images, I have tried to split the dataset in batches of 500, as I would like to get small tiles that will not kill the computer when I get to work on them afterwards (So I use merge: pointcloud). So I just go to the “split” setting, type 500, and the split-overlap to different numbers from 3 to 10 meters (leaving it blank does not let me upload any photo to the node). The photos were taken at 10m agl, as I want to get a very high resolution map. I have tried this with different parameters (resolution, ignore-gsd,…) but I get the following output when the create_submodels phase starts:

[INFO] running /code/SuperBuild/src/opensfm/bin/opensfm create_submodels /var/www/data/44a86e01-7ff1-4848-a6b6-711097026c96/opensfm
Traceback (most recent call last):
File “/code/SuperBuild/src/opensfm/bin/opensfm”, line 34, in
File “/code/SuperBuild/src/opensfm/opensfm/commands/create_submodels.py”, line 37, in run
File “/code/SuperBuild/src/opensfm/opensfm/large/metadataset.py”, line 154, in create_submodels
os.symlink(src_relpath, dst)
OSError: [Errno 95] Operation not supported
Traceback (most recent call last):
File “/code/run.py”, line 56, in
File “/code/stages/odm_app.py”, line 93, in execute
File “/code/opendm/types.py”, line 376, in run
File “/code/opendm/types.py”, line 357, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 65, in process
File “/code/opendm/osfm.py”, line 21, in run
(context.opensfm_path, command, self.opensfm_project_path))
File “/code/opendm/system.py”, line 76, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

Can anyone point me in the right direction to get this process to work? I could not find any information on the forums/docs

I have run a test dataset of 50 photos taken at 50m agl. I tried both the WebODM node in my local machine, and the split process works perfectly. When I select the server nodeODM with the same images and settings, it gives the error above.
I guess this is a bug then?
When I check the nodes in WebODM they both have the same API version (1.5.3) and the same engine version (0.8.2). Is there a way to solve the issue?

I’ve had luck processing a several thousand image dataset with --split 400 --split-overlap 120 (on a machine with 32GB RAM). I wonder if your values of 3 to 10 are too small? I don’t know enough to troubleshoot based on your error message, but it might be worth trying to run the task with a much larger overlap distance.


Unless this is a very high resolution dataset, such small overlaps would be too small to be successful with split merge.

1 Like

The split-merge got solved after I updated the NodeODM on the server, now it works fine. However, I am finding that the submodels are not properly aligned to each other, there is a 20-30cm shift between layers. The server is running NodeODM with 128GB RAM and 8 CPUs, and is working like a charm in the processing front!
I am using the odm_orthophoto from each submodel, that should be the one I should use, is that right? (I take the odm_orthophoto.tif from each submodel_XXXX/odm_orthophoto folder, and ignore the rest)

As for the dataset, the images were taken at 10m agl with a Mavic Pro, which should give a GSD of approx. 0.3cm. I used 5m for the overlap, which I thought it should be enough to align the layers. Did I misunderstand how the submodels are aligned? I would rather not use a monster TIFF file, as my computer will not be able to work with it smoothly.
Would increasing the overlap improve the alignment between layers?
@danbjoseph, I first tried to do this with a 16gb RAM machine, --split 100 and --split.overlap 5, but sadly the process dies on the pointcloud merging. It sounds like I could actually make it work according to your parameters… I will try again and see, just for testing purposes.

Hi again,
I think I managed to crack down what is going on. As soon as I start NodeODM on a location that is not the hard drive on the docker disk, it somehow fails to work. Could it be some read/write problem?
If I start NodeODM normally (run -p 3000:3000 opendronemap/nodeodm), the process works fine. However, if I start the process against another drive (docker run -p 3000:3000 -v E:/:/var/www/data opendronemap/nodeodm), the process stops at the spot mentioned above. I can see how the folder is created in E:, so that doesn’t seem to be a problem.
If I don’t split the dataset, all works smoothly. If I split it, it fails under E:, but not with the defaults. Do you @smathermather or @danbjoseph have any idea on what have I done wrong? Or is it a bug?

It’s a bug. Could you please open an issue on https://github.com/OpenDroneMap/ODM/issues referencing this thread?

Thank you!

1 Like

Thanks, @pierotofy!

The issue is now online on https://github.com/OpenDroneMap/ODM/issues/1010

In addition, I found that the docker ODM image has trouble with spaces in the filenames. I could solve this by renaming the files without spaces, but WebODM does support that function already

Could you think of any temporary workaround I could use?

1 Like

WebODM will actually convert spaces to “_” (via NodeODM filename sanitization), so you shoudn’t have the same issue.

1 Like

Thank you. Everything is working fine now.
I added info on the bug, it is apparently due to the file system: symlinks has issues when running on a cifs file system, which requires adding the mfsymlink flag to the mount command, and that is why I was getting the error. Thanks for all the help!