Multispectral fails every time with nodeodm:gpu and succeeds with nodeodm

I’ve got a data set of about 300 multispectral photos (and no GCPs). If I process them with default multispectral settings, processing succeeds with nodeodm but fails 100% of the time with nodeodm:gpu.

It fails during or after this:
[INFO] running “/code/SuperBuild/install/bin/opensfm/bin/opensfm” export_visualsfm --points “/var/www/data/0e8df53d-9710-4a03-a78f-9afa7779eaa1/opensfm”

The error message with is always “Processing stopped because of strange values in the reconstruction” but the logs indicate this is where the problem occurs:
ile “/usr/local/lib/python3.8/dist-packages/PIL/Image.py”, line 3023, in open
raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file <_io.BufferedReader name=’/var/www/data/0e8df53d-9710-4a03-a78f-9afa7779eaa1/opensfm/undistorted/images/DJI_0100.JPG.tif’>

I have re-run processing over 20 times, removing the RGB image referenced (and the related multispectral images) from the dataset before the next attempt, and it just fails citing a different image the next time around.
My gut feel is this error is happening due to a race condition with nodeodm:gpu…

System details:
WebODM 2.8.8
Docker 20.10.18
Ubuntu 22.04.1 LTS
Dual Intel Xeon 20-core CPU’s
512GB RAM
Nvidia K80 (dual 12GB GPUs)
1.8TB swap

1 Like

Have you processed just the TIFFs with the GPU node vs the TIFFs/JPGs coincident? Does this change the behavior at all?

1 Like

Interesting. This is the error I get as well when using the GPU and we have a similar system. What do you mean by “race condition”, bus architecture & communication?

2 Likes

@saijin - no I haven’t tested that, I will try it.

@vonnonn - by “race condition” I mean that one software process is dependent on the output of another software process, but one of them is completing their task too quickly so the output it expects to be ready is not yet ready and it reports an error and fails. E.g. I’m wondering if the GPU completes a task much faster than the CPU, but the software was not coded to handle that and hence the error where it cannot find a JPG image.

2 Likes

I often encounter this when using GPU for feature extraction.
It manifests as a .npz file not found. Look through the lines of the error and see if that is in there.

FileNotFoundError: [Errno 2] No such file or directory: ‘E:\WebODM\resources\app\apps\NodeODM\data\7c5af6da-e558-4d6a-a81e-ed8598e11436\opensfm\features\20210516a_397.JPG.features.npz’

1 Like

@saijin - Processing with nodeodm:gpu succeeds when the JPG images are excluded.
What do you think is going on?

1 Like

mmm… Maybe the GPU build of NodeODM hasn’t been synced with changes we’ve made to the CPU version that can process comingled JPG/TIFF datasets.

Can you open an issue, link to here, and possibly include your logs (if not your dataset)?

2 Likes

Will do Saijin

1 Like

Issue #1267 submitted.

2 Likes