Importance of masking

This is really in some ways a continuation of this topic First attempt at a 3D model - methane flue dataset but it’s locked so this is a new thread.

I’m wondering how important it is to mask areas of the input images that you don’t want to be considered for a 3D model. So for example for the flue that I’ve been using as a test case, the surrounding ground is really not of interest at all – really I only care about the fence and everything it encloses in images such as this which are typical:

AFAIU the main way this would be accomplished is with _mask images which is straightforward in principle but quite a bit of additional effort. So I have a number questions to hopefully save me some rabbit-hole exploration:

  1. Is my assumption that masking is important to remove the extraneous parts of these images?
  2. To what extent does masking improve processing time/memory requirements etc., as opposed to simply omitting areas that aren’t of interest in the result?
  3. How important is it to be consistent with masking the same areas in each image? I.e. what happens if in most of the images something in the background, e.g. a tree or bush, gets masked out but in some others it does not?
  4. I expect masking moving objects is pretty important, but is there some threshold where I shouldn’t really worry about that?
  5. More generally are there particular types of things that are very important to mask out? For example I thought I read somewhere that it’s very important to mask out the horizon and things above?
  6. Are there tools that can be used to help speed up the creation of the mask images? E.g. for successive images probably the mask is similar as the parts to be masked will have moved only slightly. So a manual workflow would be something like having a 3-layer image in GIMP where the top layer is the mask, the bottom is all white and the middle is the image to be masked. You’d update the top layer with black wherever the mask should be, hide the middle layer, export the _mask file, replace the middle layer with the next image and re-show it, and repeat. Are there scripts for GIMP or something like that extant? Or maybe dedicated tools?
  7. What about tools that automatically mask the horizon and above?

One random thing I was thinking was whether having all these _mask files could be avoided, and the workflow simplified, by allowing a key color, something that doesn’t occur in the image (bright magenta perhaps), be considered the mask by the algorithm.

Here’s another specific question that comes to mind: Suppose you have this image

Obviously you don’t want background area behind the flue to be part of the reconstruction, but do you want to mask it very precisely to the edges of the actual flue or is it better to go a bit more inward, i.e.:

4 Likes

If you want to avoid masking, you can use the --boundary option:
https://docs.opendronemap.org/arguments/boundary/

This will clip all the results to the shape of the GeoJSON you pass to the --boundary option.

  1. Yes, fully masked reconstructions will only contain what you left unmasked
  2. Should be significant. Have not carefully profiled yet.
  3. If it is something you want ignored, not always masking it means it might reconstruct or might not, depending upon how many times it is “seen”.
  4. Probably if it isn’t in 3-5 images or more, I wouldn’t really think it’d reconstruct
  5. That’s to keep the reconstruction area from growing massive and crashing/slowing. --auto-boundary and --boundary can accomplish similar.
  6. GitHub - danielgatis/rembg: Rembg is a tool to remove images background.
    this is been recommended and used a number of times here
  7. See above

You can “matte” the images instead of making mask images and accomplish similar to a key color.

As for how close to mask, I would definitely not over-mask. Anything masked out will not be reconstructed, so it is better to have a bit of bleed/slop area, in my opinion. You can always trim/cleanup later, you can’t add back data that was thrown away.

2 Likes

Thanks for the pointers! Trying rembg first as that seems to be the laziest option. Not perfect but it’s doing remarkably well so far, e.g.:

3 Likes

Yeah, rembg isn’t perfect, but it’s good enough often enough, especially if you have extra images.

1 Like

Is there any logical difference between the two in terms of resulting quality or processing time? If I matte the images, is there any particular color that is better, e.g., black or white? Or is all that matters that it be a solid color with no details?

1 Like

I have not profiled matte vs masking, so I can’t say if one is faster or if the results are actually any different.

I used white on my matte, but I’m not entirely sure it matters provided there are no features.

bad advice

If you’re using something like TIFF, you can just use the alpha channel too, I suppose (have not tried yet).

MVS-texturing chokes on alpha channel transparency. Haven’t looked into why, but I’ve tried it with PNGs.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.