Sentera Double 4K Multispectral Processing

Hello all, I recently started working with webODM to process the multispectral images I take using the Sentera Double 4K Multispectral camera. The images usually come in 2 separate folders labeled RGB and NIR, with corresponding images having the same name. The RGB files have the standard R,G,B bands and the NIR images bands are structured like this: RedEdge, Garbage, NIR. After reading through some of the other posts, I added an _rgb and _nir to the respective images and placed them all in the same file. I used exiftool to check the EXIF information of the images. The images seem to have all the correct EXIF data, however the “RigCameraIndex” was just set to a single number. Some other posts stated that RigCameraIndex should have a different number for each band so I changed “RigCameraIndex” so that there is a number corresponding to each band. In my case, red: 1, blue: 2, green: 3, rededge: 4, garbage: 5, nir: 6.

When I run webODM on my dataset, using the Multispectral preset, I get the following errors:

[INFO] Running dense reconstruction. This might take a while.
[INFO] Estimating depthmaps
[INFO] running /code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud “/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/openmvs/scene.mvs” --resolution-level 2 --min-resolution 500 --max-resolution 1752 --max-threads 24 --number-views-fuse 2 -w “/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/openmvs/depthmaps” -v 0
17:36:19 [App ] Build date: Apr 16 2021, 04:01:15
17:36:19 [App ] CPU: AMD Ryzen 9 3900 12-Core Processor (24 cores)
17:36:19 [App ] RAM: 125.90GB Physical Memory 128.00GB Virtual Memory
17:36:19 [App ] OS: Linux 4.15.0-112-generic (x86_64)
17:36:19 [App ] SSE & AVX compatible CPU & OS detected
17:36:19 [App ] Command line: /var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/openmvs/scene.mvs --resolution-level 2 --min-resolution 500 --max-resolution 1752 --max-threads 24 --number-views-fuse 2 -w /var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/openmvs/depthmaps -v 0
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00039_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00084_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00034_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00055_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00009_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00082_rgb.jpg.tif’
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00037_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00079_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00071_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00045_rgb.jpg.tif’
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00045_rgb.jpg.tif’
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00045_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00056_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00042_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00010_rgb.jpg.tif’
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00022_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00074_rgb.jpg.tif’
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00038_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00013_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00073_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00085_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00077_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00040_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00046_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00060_rgb.jpg.tif’
17:36:19 [App ] error: failed loading image header
17:36:19 [App ] error: failed reloading image ‘/var/www/data/85ad4fe9-73c3-44e3-8310-34343b0766af/opensfm/undistorted/images/IMG_00028_rgb.jpg.tif’
17:36:19 [App ] error: preparing images for dense reconstruction failed (errors loading images)
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 2.4.7 documentation
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “/code/opendm/types.py”, line 319, in run
self.process(self.args, outputs)
File “/code/stages/openmvs.py”, line 77, in process
system.run(’%s “%s” %s’ % (context.omvs_densify_path,
File “/code/opendm/system.py”, line 82, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

Here is a link to a folder that contains the images I’m using (rgb_nir_images), a sample EXIF readout for both the RGB and NIR images (sample_exif) and the entire log (console.txt).
https://drive.google.com/drive/folders/1GN6wF8Rxw211l_AcUHRXnzqjn30piJvE?usp=sharing

I’m not sure if there’s an error in the EXIF data of the images or if something else is going on. Many of the other posts stated that if webODM detected multi-camera images, there would be a message along the lines of “Multi-Camera setup detected” and “Starting BOW matching”. Those messages never appeared in my case, so I’m suspecting that webODM is not recognizing my multi-camera images. I’d appreciate any help you all can provide. Thanks in advance!

1 Like

Hi, welcome! :hand:

I think this is the same problem as the one reported on Support of 8bit multi band JPEG + 16bit single band TIFF images · Issue #1238 · OpenDroneMap/ODM · GitHub

In short, you currently cannot mix RGB images (3 channel, 8bit) and multispectral ones (1 channel, >8 bits). You might be able to work around it by normalizing all of your images to have a single channel and to use the same bit depth (extract the JPG bands into individual R/G/B TIFF images).

2 Likes

Hello!! Thanks for your quick reply :slightly_smiling_face:. I checked the file info for both the RGB and NIR images and they seem to be 8bit images. Here is the file info:

RGB

user@Michaelas-MacBook-Air 2021-03-03_18-50-41 % file RGB/IMG_00001.jpg
RGB/IMG_00001.jpg: JPEG image data, Exif standard: [TIFF image data, big-endian, direntries=11, model=21022-01_12MP-ERS-0001], baseline, precision 8, 4000x3000, components 3

NIR

user@Michaelas-MacBook-Air 2021-03-03_18-50-41 % file NIR/IMG_00001.jpg
NIR/IMG_00001.jpg: JPEG image data, Exif standard: [TIFF image data, big-endian, direntries=11, model=21022-01_12MP-ERS-0001], baseline, precision 8, 4000x3000, components 3

I believe the RGB image is a 3 channel 8bit image containing the band data for the R, G, B bands and the NIR image is an identical 3 channel 8bit image containing the band data for the Red Edge, garbage (an extra band that doesn’t hold any usable data), and NIR bands.

Is it possible to use two 3-channel images containing the different bands and have webODM recognize the multi-camera setup?

Oh interesting; I’m not sure I’ve tried this type of images before. It may or may not work, but the first step would be to have the multi-camera setup to be detected to find out.

Re-reading your original post, the issue might be in:

If you have two 3-channel types of images (one for RGB, one for ReGN) the indexes should be just 1 and 2?

The detector logic is written in ODM/types.py at 7a210e2714d73c006ec5370f50ab65742bd39a69 · OpenDroneMap/ODM · GitHub

Along with an index, make sure you have a bandName set; if there’s none, it will default to “RGB” and I suspect it will not work.

2 Likes

Great, thanks for the information. I made some changes to the Rig Camera Index atg (labeling the RGB images as RigCameraIndex=1 and the ReGN as RigCameraIndex=2), but I still got the same errors. The entire log can be found here.

[INFO] running /code/SuperBuild/install/bin/OpenMVS/DensifyPointCloud “/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/openmvs/scene.mvs” --resolution-level 2 --min-resolution 500 --max-resolution 1758 --max-threads 8 --number-views-fuse 2 -w “/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/openmvs/depthmaps” -v 0
19:32:17 [App ] Build date: Apr 16 2021, 04:01:15
19:32:17 [App ] CPU: AMD EPYC Processor (with IBPB) (8 cores)
19:32:17 [App ] RAM: 15.29GB Physical Memory 30.59GB Virtual Memory
19:32:17 [App ] OS: Linux 4.15.0-142-generic (x86_64)
19:32:17 [App ] SSE & AVX compatible CPU & OS detected
19:32:17 [App ] Command line: /var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/openmvs/scene.mvs --resolution-level 2 --min-resolution 500 --max-resolution 1758 --max-threads 8 --number-views-fuse 2 -w /var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/openmvs/depthmaps -v 0
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00083_rgb.jpg.tif’
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00086_rgb.jpg.tif’
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00074_rgb.jpg.tif’
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00074_rgb.jpg.tif’
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00028_rgb.jpg.tif’
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00021_rgb.jpg.tif’
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00060_rgb.jpg.tif’
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00091_rgb.jpg.tif’
19:32:17 [App ] error: failed loading image header
19:32:17 [App ] error: failed reloading image ‘/var/www/data/530dc873-4514-45b8-8467-994a6a8a9d1e/opensfm/undistorted/images/IMG_00077_rgb.jpg.tif’
19:32:17 [App ] error: preparing images for dense reconstruction failed (errors loading images)
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition?
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 338, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “/code/opendm/types.py”, line 319, in run
self.process(self.args, outputs)
File “/code/stages/openmvs.py”, line 77, in process
system.run(’%s “%s” %s’ % (context.omvs_densify_path,
File “/code/opendm/system.py”, line 82, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

The original photos came with the BandName tag set to Red,Green,Blue for the RGB images and RedEdge,Garbage,NIR for the ReGN images. Here’s a sample of the XMP tags for the images:

RGB
image

ReGN
image

Is this the correct format the XMP tags should be in? I checked the documentation for the Sentera 6x Camera since the Sentera 6x is supported. It seems that the format of the JPG metadata XMP tags from the 6X documentation are pretty similar to those on the Double 4K images, however, the 6X saves the JPG RGB images in one folder, and 5 monochrome TIFFs in another. I’ve also included the relevant 6X JPG XMP tags for your reference:
image

(Here’s the link if you’d like to see the full multispectral documentation for the Sentera 6X camera)

Alternatively, I’ve gone through and split the images into single-band JPGs, keeping just the relevant bands. I’ve been struggling to transfer the EXIF data to the single band images. Do you think it would be beneficial to keep pursuing this route?

Are you able to write out to EXIF instead of XMP? I don’t believe we handle XMP at all at this point.

1 Like

Oh yeah, here’s the entire EXIF data from sample RGB and ReGN images.
https://drive.google.com/drive/folders/19zEVdanmnpdaGqjLYYED4l_g9qbCxXZe?usp=sharing

I added the picture above with the XMP:Camera tags since they aren’t separated by type in the file with the entire EXIF information.

I may have misunderstood, but aren’t the tags read by this the XMP tags?

1 Like

Check the images.json file. It will contain the image information as it was parsed by ODM.

3 Likes

Here is the information from images.json:

RGB

“filename”: “IMG_00009.jpg”, “mask”: null, “width”: 4000, “height”: 3000, “camera_make”: “Sentera”, “camera_model”: “21022-01_12MP-ERS-0001”, “latitude”: 36.61954782033163, “longitude”: -91.68687050920224, “altitude”: 238.2871, “band_name”: “RedGreenBlue”, “band_index”: “1”, “capture_uuid”: null, “fnumber”: 2.5, “radiometric_calibration”: null, “black_level”: null, “exposure_time”: 0.0016420361247947454, “iso_speed”: 807, “bits_per_sample”: null, “vignetting_center”: null, “vignetting_polynomial”: null, “spectral_irradiance”: null, “horizontal_irradiance”: null, “irradiance_scale_to_si”: null, “utc_time”: 1614797962693.1301, “sun_sensor”: null, “dls_yaw”: -2.8509, “dls_pitch”: 5.4721, “dls_roll”: 90.0, “gps_xy_stddev”: null, “gps_z_stddev”: null

NIR

“filename”: “IMG_00009.jpg”, “mask”: null, “width”: 4000, “height”: 3000, “camera_make”: “Sentera”, “camera_model”: “21022-01_12MP-ERS-0001”, “latitude”: 36.61954782033163, “longitude”: -91.68687050920224, “altitude”: 238.2871, “band_name”: “RedEdgeGarbageNIR”, “band_index”: “2”, “capture_uuid”: null, “fnumber”: 2.5, “radiometric_calibration”: null, “black_level”: null, “exposure_time”: 0.002188183807439825, “iso_speed”: 1600, “bits_per_sample”: null, “vignetting_center”: null, “vignetting_polynomial”: null, “spectral_irradiance”: null, “horizontal_irradiance”: null, “irradiance_scale_to_si”: null, “utc_time”: 1614797962231.3481, “sun_sensor”: null, “dls_yaw”: -2.8509, “dls_pitch”: 5.4721, “dls_roll”: 90.0, “gps_xy_stddev”: null, “gps_z_stddev”: null

I’m still not sure if that’s the correct BandName format. Do I need to make sure that the band names are read as 3 separate bands (Red, Green, and Blue instead of RedGreenBlue)? The band names are set as Red,Green,Blue currently and the original images came with the xmp-camera:BandName tag set to that as well.

Also, my images are labeled IMG_00001_rgb.jpg for the RGB images and IMG_00001_nir.jpg for the ReGN images. I read in another post that ODM may recognize the different images based on the image names. I’m not sure if there’s a standard naming convention that might help.

1 Like

Everything looks correct.

I examined more in detail the output log and these lines:

[INFO]    Reconstruction will use 93 images from auto band
[WARNING] Unreliable UUID/capture time detected (no primary band match), will use filenames instead

Indicate that the multi-camera setup is detected correctly.

So it must be a problem in the DensifyPointCloud which cannot for some reason read the images generated during the pipeline.

This will require some debugging and time. You can open a bug report on Issues · OpenDroneMap/ODM · GitHub ? :pray:

2 Likes