Image masks feature

Hi All,

Recently I’m checking the way to bring the image masks features back to the latest ODM version. It’s still working in opensfm stage, but not for dense point cloud and later stages. I’ve made some progress to implement a script to filter the dense point cloud by masks. The approach is the same as the one used in MVE, it iterates through each shot, projects all vertices to that camera, and removes vertices locate inside the mask area. The filtered point clouds from both approaches are similar. I might need to optimize it as it’s implemented in python and could be slow for large datasets.

However, while I move on to check the mesh reconstruction results. I noticed that the new and old version of PoissonRecon provides different results. The old reconstruction is like the first 2 images, the new reconstruction is like the last 2 images. While the old reconstruction fills the filtered holes with smooth and flat faces, the new reconstruction fills with bowl-shaped faces.

I wonder if we can tweak the arguments for PoissonRecon to make it have similar results to the old version? (I’m not familiar with PoissonRecon, hope I can find help here)

Thanks!

old1
old2
new1
new2

1 Like

Really great testing!

Do you happen to know what version of ODM/PoissonRecon you used in each set of screenshots to help us pick through commits?

Yes, the old version is ODM v2.0.0, and the new one is v2.5.7.

1 Like

Have you checked image mask support in OpenMVS? It’s there, we just haven’t tied it in ODM:

I suspect it’s a matter of modifying the opensfm code (from our fork) that exports the OpenMVS scene with the proper path/mask name.

1 Like

The implementation in OpenMVS is filtering the depthmap image by image before merging them as the dense point cloud. This will have different behavior compared to MVE’s implementation I believe.

If there are 10 images contain an object we want to remove, we can provide a mask for one of the images for MVE’s implementation to remove the object from the dense point cloud, but for OpenMVS, we need to provide masks for every image. On the other hand, MVE’s implementation could remove points that are occluded by the object. Like the test I did below, I tried to remove the black earbud box on top of the paper box, the algorithm will also remove points on the other side.
Screenshot from 2021-08-27 10-00-11 Screenshot from 2021-08-27 09-59-03

Not sure which behavior is expected

1 Like

At least historically, the expectation in ODM was that every image needed to be masked (this is needed for the orthophoto generation too, if image A is masked but image B is not, if B contains the masked subject in A, the masked subject will be rendered in the final orthophoto).

1 Like

In ODM 2.0.0, the 3D geometry of the object can still be removed if not all the images containing the target object are masked.

To remove the object in the texture and orthophoto, masks for all images containing the object are needed. In this case, I suppose there will be a hole there as the faces are removed because of no texture. (I’m not sure about 2.5D mesh, I prefer to use the use-3dmesh flag)

Anyway, my work needs to remove the object on the 3d mesh file so I can do some analysis after that. But at this point, the PoissonRecon sometimes doesn’t give flat faces in the filtered area. I might try to post-process the mesh data to remove them instead.

2 Likes