From Docker to Singularity

Hi,
I need to convert my ODM workflow from Docker to Singularity but I have some binding/mounting problems :

This is my Docker command line

docker run -it --rm -v  /path_to_images/:/code/images -v "$(pwd)/odm_georeferencing:/code/odm_georeferencing"     -v "$(pwd)/odm_meshing:/code/odm_meshing"     -v "$(pwd)/odm_dem:/code/odm_dem"     -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto"     -v "$(pwd)/odm_texturing:/code/odm_texturing"     -v "$(pwd)/odm_report:/code/odm_report" opendronemap/odm --orthophoto-png 

I found that Singularity does not bind directories like Docker. First, Singularity complains if mounts do not exists. Docker in this case creates missing directories. So my script creates directories in this case.
This is my script :

images_dir=/path_to_images/
output_dir=/home/me/odm
odm_georeferencing=$output_dir/odm_georeferencing
odm_meshing=$output_dir/odm_meshing
odm_dem=$output_dir/odm_dem
odm_orthophoto=$output_dir/odm_orthophoto
odm_texturing=$output_dir/odm_texturing
odm_report=$output_dir/odm_report

creates directories if they don’t exists

mkdir -p $output_dir $odm_georeferencing $odm_meshing $odm_dem $odm_orthophoto $odm_texturing $odm_report

singularity --debug run --bind $images_dir:/code/images,$odm_georeferencing:/code/odm_georeferencing,$odm_meshing:/code/odm_meshing,$odm_dem:/code/odm_dem,$odm_orthophoto:/code/odm_orthophoto,$odm_texturing:/code/odm_texturing,$odm_report:/code/odm_report …/odm_latest.sif

I can see ODM starting and the I have the following error
[INFO] Loading dataset from: /code/images
Cannot write log.json: [Errno 30] Read-only file system: ‘/code/log.json’
Traceback (most recent call last):
File “/code/run.py”, line 59, in
retcode = app.execute()
File “/code/stages/odm_app.py”, line 130, in execute
raise e
File “/code/stages/odm_app.py”, line 94, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 327, in run
self.process(self.args, outputs)
File “/code/stages/dataset.py”, line 110, in process
with open(tree.dataset_list, ‘w’) as dataset_list:
OSError: [Errno 30] Read-only file system: ‘/code/img_list.txt’

Here is the debug output for the mounts
DEBUG [U=20071,P=1898064]mountGeneric() Mounting /path_to_images/ to /var/singularity/mnt/session/final/code/images
DEBUG [U=20071,P=1898064]mountGeneric() Remounting /var/singularity/mnt/session/final/code/images
DEBUG [U=20071,P=1898064]mountGeneric() Mounting /home/me/odm/odm_georeferencing to /var/singularity/mnt/session/final/code/odm_georeferencing
DEBUG [U=20071,P=1898064]mountGeneric() Remounting /var/singularity/mnt/session/final/code/odm_georeferencing
DEBUG [U=20071,P=1898064]mountGeneric() Mounting /home/me/odm/odm_meshing to /var/singularity/mnt/session/final/code/odm_meshing
DEBUG [U=20071,P=1898064]mountGeneric() Remounting /var/singularity/mnt/session/final/code/odm_meshing
DEBUG [U=20071,P=1898064]mountGeneric() Mounting /home/me/odm/odm_dem to /var/singularity/mnt/session/final/code/odm_dem
DEBUG [U=20071,P=1898064]mountGeneric() Remounting /var/singularity/mnt/session/final/code/odm_dem
DEBUG [U=20071,P=1898064]mountGeneric() Mounting /home/me/odm/odm_orthophoto to /var/singularity/mnt/session/final/code/odm_orthophoto
DEBUG [U=20071,P=1898064]mountGeneric() Remounting /var/singularity/mnt/session/final/code/odm_orthophoto
DEBUG [U=20071,P=1898064]mountGeneric() Mounting /home/me/odm/odm_texturing to /var/singularity/mnt/session/final/code/odm_texturing
DEBUG [U=20071,P=1898064]mountGeneric() Remounting /var/singularity/mnt/session/final/code/odm_texturing
DEBUG [U=20071,P=1898064]mountGeneric() Mounting /home/me/odm/odm_report to /var/singularity/mnt/session/final/code/odm_report
DEBUG [U=20071,P=1898064]mountGeneric() Remounting /var/singularity/mnt/session/final/code/odm_report
DEBUG [U=20071,P=1898064]addCwdMount() Using /home/me/odm as current working directory
VERBOSE [U=20071,P=1898064]addCwdMount() Not mounting CWD, /home/me/odm doesn’t exist within container
VERBOSE [U=20071,P=1898064]addCwdMount() Not mounting CWD, while getting /home/me/odm information: stat /var/singularity/mnt/session/final/home/me/odm: no such file or directory
DEBUG [U=20071,P=1898064]create() Chroot into /var/singularity/mnt/session/final
DEBUG [U=20071,P=1898086]Chroot() Hold reference to host / directory
DEBUG [U=20071,P=1898086]Chroot() Called pivot_root on /var/singularity/mnt/session/final
DEBUG [U=20071,P=1898086]Chroot() Change current directory to host / directory
DEBUG [U=20071,P=1898086]Chroot() Apply slave mount propagation for host / directory
DEBUG [U=20071,P=1898086]Chroot() Called unmount(/, syscall.MNT_DETACH)
DEBUG [U=20071,P=1898086]Chroot() Changing directory to / to avoid getpwd issues
DEBUG [U=20071,P=1898064]create() Chdir into / to avoid errors

I am thinking that maybe singularity wants to write to /code mount that is not writable. But I am not sure.

Thanks for your help

1 Like

Ok I found my solution :slight_smile:
I need to use this option

run --writable-tmpfs

1 Like

This is great! Thanks for sharing this. I have been wondering about deployment for singularity.

It might be nice to write up some brief docs at docs.opendronemap.org or alternatively deployment setup for singularity. Would you be willing to help?

1 Like

I would be happy to share my solution in the doc, but I did not manage to finished my first test !

Now I have the following error :

2022-01-27 09:39:04,715 DEBUG: Found 13658 points in 4.073966026306152s
2022-01-27 09:39:04,717 DEBUG: No segmentation for DSC_0424.JPG, no features masked.
2022-01-27 09:39:04,761 DEBUG: Found 13784 points in 4.054922819137573s
2022-01-27 09:39:04,763 DEBUG: No segmentation for DSC_0435.JPG, no features masked.
2022-01-27 09:39:04,820 DEBUG: Found 13710 points in 4.167374134063721s
2022-01-27 09:39:04,821 DEBUG: No segmentation for DSC_0427.JPG, no features masked.
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
self.close()
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
self.fp.seek(self.start_dir)
ValueError: seek of closed file
Traceback (most recent call last):
File “/usr/local/lib/python3.9/dist-packages/numpy/lib/npyio.py”, line 722, in _savez
format.write_array(fid, val,
File “/usr/local/lib/python3.9/dist-packages/numpy/lib/format.py”, line 696, in write_array
fp.write(chunk.tobytes(‘C’))
File “/usr/lib/python3.9/zipfile.py”, line 1129, in write
self._fileobj.write(data)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 332, in save_words
np.savez_compressed(f, words=words.astype(np.uint16))
File “<array_function internals>”, line 5, in savez_compressed
File “/usr/local/lib/python3.9/dist-packages/numpy/lib/npyio.py”, line 689, in savez_compressed
_savez(file, args, kwds, True)
File “/usr/local/lib/python3.9/dist-packages/numpy/lib/npyio.py”, line 722, in _savez
format.write_array(fid, val,
File “/usr/lib/python3.9/zipfile.py”, line 1168, in close
self._fileobj.seek(self._zinfo.header_offset)
OSError: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py”, line 25, in
commands.command_runner(
File “/code/SuperBuild/install/bin/opensfm/opensfm/commands/command_runner.py”, line 37, in command_runner
command.run(data, args)
File “/code/SuperBuild/install/bin/opensfm/opensfm/commands/command.py”, line 12, in run
self.run_impl(data, args)
File “/code/SuperBuild/install/bin/opensfm/opensfm/commands/detect_features.py”, line 11, in run_impl
detect_features.run_dataset(dataset)
File “/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py”, line 15, in run_dataset
features_processing.run_features_processing(data, data.images(), False)
File “/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py”, line 65, in run_features_processing
parallel_map(process, arguments, processes, 1)
File “/code/SuperBuild/install/bin/opensfm/opensfm/context.py”, line 56, in parallel_map
res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
File “/usr/local/lib/python3.9/dist-packages/joblib/parallel.py”, line 1061, in call
self.retrieve()
File “/usr/local/lib/python3.9/dist-packages/joblib/parallel.py”, line 940, in retrieve
self._output.extend(job.get(timeout=self.timeout))
File “/usr/lib/python3.9/multiprocessing/pool.py”, line 771, in get
raise self._value
File “/usr/lib/python3.9/multiprocessing/pool.py”, line 125, in worker
result = (True, func(*args, **kwds))
File “/usr/local/lib/python3.9/dist-packages/joblib/_parallel_backends.py”, line 595, in call
return self.func(*args, **kwargs)
File “/usr/local/lib/python3.9/dist-packages/joblib/parallel.py”, line 262, in call
return [func(*args, **kwargs)
File “/usr/local/lib/python3.9/dist-packages/joblib/parallel.py”, line 262, in
return [func(*args, **kwargs)
File “/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py”, line 120, in process
run_detection(queue)
File “/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py”, line 155, in run_detection
detect(image, image_array, segmentation_array, instances_array, data, force)
File “/code/SuperBuild/install/bin/opensfm/opensfm/features_processing.py”, line 273, in detect
data.save_words(image, closest_words)
File “/code/SuperBuild/install/bin/opensfm/opensfm/dataset.py”, line 332, in save_words
np.savez_compressed(f, words=words.astype(np.uint16))
OSError: [Errno 28] No space left on device
Exception ignored in: <function ZipFile.del at 0x146dbdd24a60>
Traceback (most recent call last):
File “/usr/lib/python3.9/zipfile.py”, line 1807, in del
File “/usr/lib/python3.9/zipfile.py”, line 1824, in close
ValueError: seek of closed file

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 94, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 346, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 346, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 346, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 327, in run
self.process(self.args, outputs)
File “/code/stages/run_opensfm.py”, line 35, in process
octx.feature_matching(self.rerun())
File “/code/opendm/osfm.py”, line 321, in feature_matching
self.run(‘detect_features’)
File “/code/opendm/osfm.py”, line 34, in run
system.run(’"%s" %s “%s”’ %
File “/code/opendm/system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.7.1 documentation

For some reasons singularity does not have “enough space on device”…

1 Like

@smathermather
Where should I put my singularity script ? Maybe in the Docker page ?
I started using singularity few days ago. So I may only share my working script and not more.
Thanks

1 Like

A pull request to put it into the contrib directory in its own folder would be perfect.

1 Like

Before sharing my script I would like to use the GPU of our server.

With singularity doc (GPU Support (NVIDIA CUDA & AMD ROCm) — Singularity container 3.5 documentation)
I know that my gpu(s) are available with singularity.

But with the latest docker://opendronemap/odm:gpu, ODM does not detect my gpu.

We should use “–gpus all” option for Docker with maybe set a flag for ODM.
For singularity I should use the “–nv” option.

I think that singularity does not cascade the GPU options to odm.
Any ideas ? How does ODM is aware of the gpus ?

Thanks

It looks for the name of the GPU in lspci

And in ODM ? I would like to use ODM with GPU and not WebODM
Thanks

1 Like

It checks for the CUDA runtime DLL on Windows:

And NVIDIA SMI on other platforms (Linux, namely):

Thanks !
On my linux server, I can see my gpu with nvidia-smi command.
Using singularity shell with docker://opendronemap/odm:gpu , I can see that nvidia-smi is available in the opendronemap/odm:gpu singularity image.

But ODM does not see any gpu :frowning:
my odm command line is

singularity  --debug  run  --nv \
 --bind $images_dir:/$output_dir/code/images,\
 --writable-tmpfs ../odm_gpu.sif  \
 --ignore-gsd --feature-quality ultra --max-concurrency 16 \
 --use-hybrid-bundle-adjustment --build-overviews --time --min-num-features 10000 \
 --project-path $output_dir
1 Like

For now ODM does not print something when a gpu is detected. I created a pull request for that

1 Like

Excellent! Thanks for the PR!

Here is my contribution for the Singularity documentation

2 Likes

Wonderful! Thanks so much!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.