Multispectral photo getting started question

ODM
#1

Pardon my ignorance as I am new to drone mapping software.

I see that multispectral photos are now supported, but I’m having a hard time understanding exactly how to process my data. I have two JPG image sets - one for RGB and one for NIR. Do I need to manually merge these into a single TIFF with N bands and let ODM figure out the rest? I am attempting to create an orthomosaic (with GCPs) and then compute NDVI.

Thanks in advance!

  • Adam

Edit
After reading through some of the code and downloading one of the demo data sets I realize I need a TIF for each band. I am using a Sentera Inspire camera (https://sentera.com/product/inspire-double-4k-upgrade-crop-health-sensor/) which isn’t officially supported.

If I upload some sample data, is it possible to support Sentera? I would be happy to contribute (code and data) if someone could point me in the right direction.

Edit 2
Well, this is non trivial. Im having trouble understanding why it appears that ODM relies on XMP tags added by Pix4D.

Thanks!

#2

Hey @adamjson :hand: welcome, ideally you should be able to place all images (RGB and NIR) in the same images directory, launch ODM and get a 5-band orthophoto (RGBNA), A being transparency (alpha).

When you start the process, ODM should be able to detect that this is a multi-camera setup and should output within the first lines of the console:

Multi-camera setup, using BOW matching

If you don’t see that, there’s a bug and some changes are needed. https://github.com/OpenDroneMap/ODM/blob/e4d162ec8369bfc7ebdc269de9e6817bbc215524/opendm/types.py#L153

The XMP tags are not specific to Pix4D, ODM attempts to detect the Band names and indexes (which dictate the order of the bands), which are needed to detect a multi-camera setup.

1 Like
#3

Hi
Just to verify… Multispectral data can be JPG right? All discussions here of multispectral came from sets that are TIF.
I have been having problems starting up the Multi-camera setup in the log as in the above when the run starts. I have have both band name and capture ID in the exif but still no go.
Then I thought maybe it’s because all images are JPG and not TIF.
All other RGB data processing I did so far has been great. I just came across this issue in multispectral. Using webODM, Ubuntu 18.04LTS

Thanks much !

Sample exif:

[email protected]:~/odm_tests$ exiftool -sort blue_left_DSC00002.JPG
Band Name : Blue
Bits Per Sample : 8
Capture Id : 1
Capture UUID : 1
Color Components : 3
Directory : .
Encoding Process : Baseline DCT, Huffman coding
Exif Byte Order : Big-endian (Motorola, MM)
ExifTool Version Number : 10.80
File Access Date/Time : 2020:03:31 19:23:04+03:00
File Inode Change Date/Time : 2020:03:31 19:23:04+03:00
File Modification Date/Time : 2020:03:31 19:23:04+03:00
File Name : blue_left_DSC00002.JPG
File Permissions : rw-rw-r–
File Size : 4.1 MB
File Type : JPEG
File Type Extension : jpg
GPS Altitude : 254.19 m
GPS Latitude : 36 deg 8’ 16.20" N
GPS Latitude Ref : North
GPS Longitude : 119 deg 0’ 45.33" W
GPS Longitude Ref : West
GPS Position : 36 deg 8’ 16.20" N, 119 deg 0’ 45.33" W
GPS Version ID : 2.3.0.0
Image Height : 2528
Image Size : 3960x2528
Image Width : 3960
JFIF Version : 1.01
MIME Type : image/jpeg
Megapixels : 10.0
Resolution Unit : None
X Resolution : 1
XMP Toolkit : XMP Core 4.4.0-Exiv2
Y Cb Cr Positioning : Centered
Y Cb Cr Sub Sampling : YCbCr4:2:0 (2 2)
Y Resolution : 1

#4

JPG should be fine even for multispectral.

#5

ok thank you. still having issue to activate multi-camera mode.


I am including a small sample set with:

  1. 30 imgs with 3 single bands from Sony Agrowing (450,550,650).
  2. camera_models.json for Agrowing
  3. Log snip showing the Multi-camera mode did not initialize
  4. 50 imgs from mica data set, with 5 single bands
  5. camera_models.json for the mica
  6. Log snip showing Multi-camera mode did initialize properly

I initiated ODM runs with identical parameters on the two data sets, trying to get multi-spectral/band orthophoto from the Agrowin data.

Would be happy to hear any ideas on what’s the cause. Is there documentation on how the exif needs to be formatted to make it work?

Thank you.

#6

Edit to the above:

It looks like the issue is with the ‘BandName’ tag in the exif. I am using data from Sony Agrowing multi spectral camera. It seems that Mica, Parrot have the same field which gets read and invokes the multi camera mode per the code here.
When looking at exif from Mica, Parrot compared to mine - all show this (Blue in this case):

<XMP-Camera:BandName>Blue</XMP-Camera:BandName>

So it seems the tag is present in my exif but somehow does not get interpreted properly by ODM.

I am using exiv2 to write the band name as:

exiv2 -M"reg Camera https://xmp.camera/" -M"add Xmp.Camera.BandName ‘Blue’" filename.JPG

Is there a problem with writing the band name in this manner so ODM will interpret it properly?

I tried to run the snippet of code I think is relevant link, specifically the get_xmp function. I think this is what is responsible to parse the exifs…?
I am attaching the two outputs from my exif and the mica. They are definitely different. Probably due to the way each exif is getting written.

Would appreciate any help on formatting the exif Band-Name tag.

Tnx !

#7

Band name is not getting recognized for the JPG images attached in the archive. This is improved in https://github.com/OpenDroneMap/ODM/pull/1097

1 Like
#8

Thank you very much for the merge !!

ODM recognizes a multi-camera setup now and appears to continue normally, but only for a bit…

What also helped was to use exiftool to write the XMP_camera tags in stead of exiv2.

However, now the run exits with error code 1 with what appears to be connected to an empty mesh error when running mve dense point cloud. Looked on the forum for this error and didn’t see any threads as well as in github.

Attached a small sample set with 10 images per band x 3 bands including the log from the run.

Below is the relevant part of the log. I tried to run with skip-3dmodel enabled but it did not help as well as running with in --fast-orthophoto

This looks a little deeper than the BandName issue. I am not sure it’s related. This is a bit beyond my knowledge level of ODM at the moment…

Would appreciate any help.

tnx!

[INFO] running /code/SuperBuild/src/elibs/mve/apps/meshclean/meshclean -t0.6 --no-clean --component-size=0 “/datasets/code/mve/mve_dense_point_cloud.ply” “/datasets/code/mve/mve_dense_point_cloud.filtered.ply”
MVE FSSR Mesh Cleaning (built on Mar 19 2020, 15:23:46)
Loading mesh: /datasets/code/mve/mve_dense_point_cloud.ply
PLY Loader: comment Export generated by libmve
Reading PLY: 0 verts… done.
Error: Mesh is empty!
Traceback (most recent call last):
File “/code/run.py”, line 61, in
app.execute()
File “/code/stages/odm_app.py”, line 92, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 344, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 325, in run
self.process(self.args, outputs)
File “/code/stages/mve.py”, line 89, in process
system.run(’%s -t%s --no-clean --component-size=0 “%s” “%s”’ % (context.meshclean_path, min(1.0, args.mve_confidence), tree.mve_model, mve_filtered_model), env_vars={‘OMP_NUM_THREADS’: args.max_concurrency})
File “/code/opendm/system.py”, line 76, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

Attaching the sample set including the log file

#9

I really don’t know, without the entire dataset it’s difficult to tell. Look at the opensfm point cloud for clues as to why it’s not working. Run it with --fast-orthophoto to get a reconstruction.ply file in the opensfm directory which you can open with Meshlab/cloudcompare.

#10

I could use some help or info on how to get going with multispectral on ODM as well. I’m using images from a Sentera Double 4K. Do bands have to be split into single channel (monochrom) jpgs tagged with the BandName metadata? Or can 3 channel jpgs be used too (with metadata)?

Here’s what I’ve tried that seemed to get the farthest:

Images come off the camera in two subfolders: NIR, and RGB. Each jpg inside those folders has a corresponding one in the opposing folder with the same name. I copied the jpgs from each sub folder into a single folder, adding a respective token to the filename to denote if it came from the NIR or RGB source. (e.g. ‘IMG_00001.jpg’ -> ‘IMG_RGB_00001.jpg’).

I then tagged the jps with the exiftool to generate the BandName metadata. (e.g. for RGB jpgs, I used the arguments 'exiftool -xmp:Camera:BandName=“Red, Green, Blue” RGB ’ and ‘exiftool -xmp:Camera:BandName=“NIR,RedEdge,Junk” NIR’ for NIR jpgs). I verified the tags were there with exiftool, seeing: “Band Name : Red,Green,Blue”, etc.

At the end of processing on WebODM with the ‘Multispectral’ preset (using the latest git pull and docker images as of 4 days ago from this post), I see: ‘ERROR 6: PNG driver doesn’t support 8 bands. Must be 1 (grey),
2 (grey+alpha), 3 (rgb) or 4 (rgba) bands.’ repeated hundreds of times on the output log. I can see an ortho map with RGB data in it on the web interface map viewer (with black borders, no alpha opacity to see the basemap below it), but clicking on the ‘Plant Health’ button just yields a ‘“Bad Request”’ message, and nothing happens.

I can upload the images I’m using shortly.

I’d greatly appreciate any help!

#11

Uploads here:

These are the images combined into a single folder, with the aforementioned BandName metadata added:
https://nextcloud.doorcreekorchard.com/index.php/s/6XnHnAm79nScfqC

Here’s the same images, but directly as they come off the camera (camera’s filestructure, no XMP BandName metadata): https://nextcloud.doorcreekorchard.com/index.php/s/EPdxrpCWSafsP3E

Pass for both is “odmsentera4kimgs”, and links will expire the 14th.

1 Like
#12

@pierotofy or anyone else, does anything stand out as something I’m doing wrong for the multispectral workflow? I can extend the expiration time of the links which contain the full dataset.

1 Like
#13

I have yet to process true multispectral data, so I simply can’t be of value here. Just as a matter of course, I would recommend extending the timeout a bit further, just to be sure anyone who could help has a chance before they time out.

#14

Links now expire 4/21

1 Like
#15

Hi @mgriffith,

You should have the bands separated, i.e. red imgs, green imgs etc. The BandName tag should be ‘red’ only for the red, ‘green’ only for the green etc.

You should also have a tag ‘Band Index’ which is just a serial integer that is identical for all the imgs of a given band, i.e. ‘0’ for all the red imgs, ‘1’ for all the green etc.

Put all imgs in the same folder.

Right in the beginning of the run you should see something similar to this in the log:

[INFO]    Increasing matcher neighbors to 24 to accomodate multi-camera setup
[INFO]    Multi-camera setup, using BOW matching

If you do, yo have entered the multi-band (or multi-camera mode) and ODM will try and process it as such.

Hope it works.

By the way I initially used exiv2 to write exif tags and had more success using exiftool. That is also the recommended utility in the “Practical Guide to Drone Mapping…” by @pierotofy.

1 Like
#16

Actually, if you look here

you’ll see that the band index is actually one of three options that were tailored to either MicaSense or Sequoia… so you’ll have to “dress” your band index tag to look like one of them…

I chose for whatever reason the last one:

> 'Camera:RigCameraIndex', # MicaSense Altum

For now it seems ODM understands my exif and enters multi camera mode successfully even though my sensor is Agrowing.(www.agrowing.com)

2 Likes
#17

@gast Ah, I missed the importance of the Band Index tag! I was even looking at that part of photo.py for the BandName logic, but missed that the index bit was necessary.

I’m using exiftool right now too for tagging, so hopefully that won’t be a source of issues.

It would be nice if somewhere down the line, it wasn’t necessary to split multichannel images into single channel ones for multispectral processing in ODM. It also is a bit syntactically strange to have to mimic an XMP tag for a SensorID or RigCamera, as with camera sets like the Sentera Double 4k, there’s only really two sensors with multiple bands coming from each. (Yes, I see that gets transferred into attribute called ‘band_index’ which does make syntactic sense, but it seems the only way to get that info into ODM at the moment is to mimic the XMP tags of other cameras/rigs).

BTW, @pierotofy et. all, I’m happy for anyone who wants to use or rehost my posted datasets if it can help with the development of ODM. I just don’t have a huge amount of monthly transfer on the host they’re on at the moment (I don’t have a personal gdrive/dropbox).

2 Likes
#18

So, just got back to this, and still no luck.

I am seeing

[INFO]    Increasing matcher neighbors to 40 to accomodate multi-camera setup
[INFO]    Multi-camera setup, using BOW matching
[WARNING] Using BOW matching, will use HAHOG feature type, not SIFT

in the log, so it does seem to be entering the multispectral mode, but I still get the errors I was describing before with some variations (more on that below).

To re-iterate, in-case I’m still missing something, here’s the steps I’m taking prior to submission to WebODM:

  1. Split the multichannel jpgs from the sensor into single channel jpgs, a separate set for each band (5 bands for this particular sensor). The names would be all identical, so I add a relevant token to the jpg filename (i.e. _RED)
  2. Combining these sets into a single folder.
  3. Add BandName metadata tags to the jpgs that correspond to their actual represented bands.
  4. Add a metadata tag that will get translated to band_index (I’m using ‘Camera:RigCameraIndex’) which also corresponds to which band the jpg represents. (In my case, 0=red,1=green,2=blue,3=RedEdge,4=NIR).

Then I upload these and submit the task using the ‘Multispectral’ preset.

This time, the error listed at the end is:

ERROR 6: PNG driver doesn't support 7 bands.  Must be 1 (grey),
2 (grey+alpha), 3 (rgb) or 4 (rgba) bands.

So now it appears it’s trying to process 7 bands/channels instead of 8 from earlier tests. I would think given its input, it would be 6 (5 bands + 1 alpha)?

Also, now that I’ve separated the bands into their individual channel jpgs, the results I can see are MUCH worse for registration/alignment in the orthophoto between rgb chanels. (Before when I was initially writing up my problems above, the rgb ortho was coherent). See attached screenshot.

Screenshot from 2020-04-16 17-47-28
Screenshot from 2020-04-16 17-47-281439×823 1.05 MB

Anyone have any idea what I’m doing wrong?
I’d like to fix any mistakes I’m doing as eventually I’d like to test some imagery with 200 bands.

Also uploading the log of the task in case it can help shed some light (I don’t see anything glaringly obvious beyond the errors I’ve already described).

Edit: forgot the log link:
https://gist.githubusercontent.com/chippey/daa679cee22b8f737d68e90c657e454b/raw/e2c86d887925d7229e959d1ac633ee342396e582/console.txt

#19

Hi @mgriffith

From what it look like - you are doing the right thing as far as I can tell.

I have actually been trying to split multi-band to single channels exactly like you did and run in multi camera and so far it always entered the multi-camera with the log messages you mentioned above but I was never able to complete the runs an actually get orthophotos.

I was only able to successfully complete multi-camera runs when the input is multi-band for each camera, i.e. two cameras where each has RGB JPG images. Then the output is a 6-layer tiff.

So going back to your original post - I think you should be able to run multi-camera without splitting to monochrome. It may actually speed up the process because you are running only two JPGs for every location rather than processing 6 monochromes.

Also - are you running with the --orthophoto-png flag? Does it happen when that flag is off?

By the way - if you have other tips on how to get the monochrome multi-camera set to run all the way - I would be happy to hear.

Cheers.

#20

Hi @mgriffith,

My issues with multi-camera were initially different where I could not complete ODM run at all. That happened to be a result of bad EXIF metadata.

However - I am coming now to similar error as you do in terms of the PNG driver. It happened on multiple occasions.

  1. Running with two cameras. Each is an RGB image. So I expect a total of 6 bands + 1 alpha channel. I get indeed a 7-layer TIF as output but the error appears at the very end of the log after everything was already done. I even get

[INFO] OpenDroneMap app finished - Mon Apr 27 11:33:56 2020
Postprocessing: /var/www/data/bdcedf98-3268-4ee5-b3d0-d31eee1b4339

Computing source raster statistics…

It looks like it is can’t exit some loop it is in…as in the below (link to log log_01052020.txt - Google Drive)

ERROR 6: PNG driver doesn’t support 8 bands. Must be 1 (grey),
2 (grey+alpha), 3 (rgb) or 4 (rgba) bands.

ERROR 6: PNG driver doesn’t support 8 bands. Must be 1 (grey),
2 (grey+alpha), 3 (rgb) or 4 (rgba) bands.

ERROR 6: PNG driver doesn’t support 8 bands. Must be 1 (grey),
2 (grey+alpha), 3 (rgb) or 4 (rgba) bands.

the very bottom of the log file is

ERROR 4: orthophoto_tiles/21/355267/1274610.png: No such file or directory

Generating Overview Tiles:
0
Traceback (most recent call last):
File “/var/www/scripts/gdal2tiles.py”, line 2947, in 

main()

File “/var/www/scripts/gdal2tiles.py”, line 2943, in main

multi_threaded_tiling(input_file, output_folder, options)

File “/var/www/scripts/gdal2tiles.py”, line 2927, in multi_threaded_tiling

create_overview_tiles(conf, output_folder, options)

File “/var/www/scripts/gdal2tiles.py”, line 1114, in create_overview_tiles

dsquerytile.ReadRaster(0, 0,

AttributeError: ‘NoneType’ object has no attribute ‘ReadRaster’

Generating Base Tiles:

0
.
.
.
Done!

Is this a bug?? I could not find “PNG driver” anywhere here GitHub - OpenDroneMap/ODM: A command line toolkit to generate maps, point clouds, 3D models and DEMs from drone, balloon or kite images. 📷

Where is the error coming from? Looks like is related to the ortho tiles…

To reiterate, the run ended successfully and all outputs were written successfully but the error still appeared.

  1. This issue also happened when I used 4 cameras. Each with 3-band RGB JPGs. Again the process ended with all outputs written successfully but this time, it simply hung on the PNG error and would not finish. I expected to write 13-layer TIF. However it only wrote a 6-layer TIF.

Any ideas?

Edit
Maybe connected to GDAL as in this case?