Error during mesh reconstruction when running in Singularity (error: invalid project)

ODM
1

I am running into an error when running ODM in Singularity. A basic othro creation runs fine but its failing trying to create the 3D surface layers. The error starts with

13:33:58 [App     ] error: invalid project

and then I get a Python error stack. The folders its looking for for the data/inputs do exist where its looking and i dont really understand the “invalid project” message

[INFO]    Already extracted cameras
[INFO]    Export reconstruction stats
[WARNING] Found existing reconstruction stats /datasets/Flight_1/opensfm/stats/stats.json
[WARNING] Will skip exporting /datasets/Flight_1/opensfm/reconstruction.geocoords.json
[INFO]    Undistorting /datasets/Flight_1/opensfm ...
[WARNING] Already undistorted (nominal)
[WARNING] Found a valid OpenSfM NVM reconstruction file in: /datasets/Flight_1/opensfm/undistorted/reconstruction.nvm
[INFO]    Finished opensfm stage
[INFO]    Running openmvs stage
[WARNING] Found a valid OpenMVS reconstruction file in: /datasets/Flight_1/opensfm/undistorted/openmvs/scene_dense_dense_filtered.ply
[INFO]    Finished openmvs stage
[INFO]    Running odm_filterpoints stage
[WARNING] Found a valid point cloud file in: /datasets/Flight_1/odm_filterpoints/point_cloud.ply
[INFO]    Finished odm_filterpoints stage
[INFO]    Running odm_meshing stage
[INFO]    Writing ODM Mesh file in: /datasets/Flight_1/odm_meshing/odm_mesh.ply
[INFO]    running "/code/SuperBuild/install/bin/PoissonRecon" --in "/datasets/Flight_1/odm_filterpoints/point_cloud.ply" --out "/datasets/Flight_1/odm_meshing/odm_mesh.dirty.ply" --depth 11 --pointWeight 4.0 --samplesPerNode 1.0 --threads 35 --bType 2 --linearFit --verbose
*************************************************************
*************************************************************
** Running Screened Poisson Reconstruction (Version 13.72) **
*************************************************************
*************************************************************
	--bType 2
	--in /datasets/Flight_1/odm_filterpoints/point_cloud.ply
	--depth 11
	--out /datasets/Flight_1/odm_meshing/odm_mesh.dirty.ply
	--verbose
	--samplesPerNode 1.000000
	--pointWeight 4.000000
	--threads 35
	--linearFit
Input Points / Samples: 24781317 / 2945739
# Read input into tree:      31.2 (s),     477.9 (MB) /     477.9 (MB) / 506 (MB)
#   Got kernel density:       0.7 (s),     552.3 (MB) /     552.3 (MB) / 552 (MB)
#     Got normal field:       5.9 (s),    1012.6 (MB) /    1012.6 (MB) / 1012 (MB)
Point weight / Estimated Measure: 1.70602e-08 / 0.422775
#Initialized point interpolation constraints:       1.3 (s),    1071.7 (MB) /    1071.7 (MB) / 1071 (MB)
#       Finalized tree:       5.7 (s),    1738.7 (MB) /    1738.7 (MB) / 1738 (MB)
#  Set FEM constraints:       1.9 (s),    1495.9 (MB) /    1738.7 (MB) / 1738 (MB)
#Set point constraints:       0.1 (s),    1496.1 (MB) /    1738.7 (MB) / 1738 (MB)
Leaf Nodes / Active Nodes / Ghost Nodes / Dirichlet Supported Nodes: 13949048 / 15934800 / 6969 / 0
Memory Usage: 1496.062 MB
Cycle[0] Depth[ 5/11]:	Updated constraints / Got system / Solved in:  0.001 /  0.017 /  0.009	(1496.062 MB)	Nodes: 29791
Cycle[0] Depth[ 6/11]:	Updated constraints / Got system / Solved in:  0.001 /  0.005 /  0.017	(1496.062 MB)	Nodes: 31548
Cycle[0] Depth[ 7/11]:	Updated constraints / Got system / Solved in:  0.002 /  0.009 /  0.025	(1496.758 MB)	Nodes: 101144
Cycle[0] Depth[ 8/11]:	Updated constraints / Got system / Solved in:  0.006 /  0.024 /  0.052	(1515.492 MB)	Nodes: 349072
Cycle[0] Depth[ 9/11]:	Updated constraints / Got system / Solved in:  0.020 /  0.065 /  0.143	(1551.168 MB)	Nodes: 1231224
Cycle[0] Depth[10/11]:	Updated constraints / Got system / Solved in:  0.090 /  0.202 /  0.441	(1625.559 MB)	Nodes: 4089632
Cycle[0] Depth[11/11]:	Updated constraints / Got system / Solved in:  0.001 /  0.518 /  1.021	(1651.855 MB)	Nodes: 10074336
# Linear system solved:       4.4 (s),    1651.9 (MB) /    1738.7 (MB) / 1738 (MB)
Got average:       0.2 (s),    1345.9 (MB) /    1738.7 (MB) / 1738 (MB)
Iso-Value: 4.985763e-01 = 1.23554e+07 / 2.47813e+07
Vertices / Polygons: 2336139 / 4672070
Corners / Vertices / Edges / Surface / Set Table / Copy Finer: 0.3 / 2.8 / 0.5 / 0.5 / 1.4 / 0.1 (s)
#        Got triangles:      15.0 (s),    1867.2 (MB) /    1867.2 (MB) / 1867 (MB)
#          Total Solve:      69.1 (s),    1867.2 (MB)
[INFO]    running "/code/SuperBuild/install/bin/OpenMVS/ReconstructMesh" -i "/datasets/Flight_1/odm_meshing/odm_mesh.dirty.ply" -o "/datasets/Flight_1/odm_meshing/odm_mesh.ply" --remove-spikes 0 --remove-spurious 20 --smooth 0 --target-face-num 400000 -v 0
13:33:58 [App     ] Build date: Jul 15 2022, 14:33:31
13:33:58 [App     ] CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz (36 cores)
13:33:58 [App     ] RAM: 125.42GB Physical Memory 8.00GB Virtual Memory
13:33:58 [App     ] OS: Linux 4.18.0-372.9.1.el8.x86_64 (x86_64)
13:33:58 [App     ] Disk: 15.99MB (16.00MB) space
13:33:58 [App     ] SSE & AVX compatible CPU & OS detected
13:33:58 [App     ] Command line: ReconstructMesh -i /datasets/Flight_1/odm_meshing/odm_mesh.dirty.ply -o /datasets/Flight_1/odm_meshing/odm_mesh.ply --remove-spikes 0 --remove-spurious 20 --smooth 0 --target-face-num 400000 -v 0
13:33:58 [App     ] error: invalid project

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 94, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 347, in run
    self.next_stage.run(outputs)
  [Previous line repeated 3 more times]
  File "/code/opendm/types.py", line 328, in run
    self.process(self.args, outputs)
  File "/code/stages/odm_meshing.py", line 24, in process
    mesh.screened_poisson_reconstruction(tree.filtered_point_cloud,
  File "/code/opendm/mesh.py", line 208, in screened_poisson_reconstruction
    system.run('"{reconstructmesh}" -i "{infile}" '
  File "/code/opendm/system.py", line 106, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow... =====

[ERROR]   Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying/


***************** Cleanup *****************


***************** ODM process complete! *****************

Run completed on:  Tue Jul 19 13:33:58 EDT 2022
Total runtime: 1977.400387863
1 Like
2

Changes might be needed to get ODM to work with Singularity; it’s not officially supported (but we’d welcome pull requests). :+1:

2 Likes
3

Yeah I could create a PR for the docs with my findings. I basically sorted out how to run with Singularity in this thread: ModuleNotFoundError: No module named 'packaging' when running with Singularity - #3 by serbinsh

I used the approach I would have expected, by following a similar approach to docker but instead using the Singularity calls. It seems to work very well on my HPC allowing me to make full memory and CPU/core utilization per node (e.g. running in parallel on 36 cores per task or more). I even have a working draft of a SLURM submission script. I could provide a basic example for that as well.

Overall I think the current Singularity instructions on the docs are perhaps a bit too specific to a single setup and the example in the other thread I provided is far simpler to follow and get started with

2 Likes
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.