Empty file stop process

[daniele_rossii]

I uploaded 900 pictures from drone, and did not notice that on was empty.
After upload, the program stop and exit.
I did check the single picture named in error log, and it was a zero byte, probably an error during recording in the fly.

Deleted the single picture and upload again.

It wolud be a nica to have an instruction to skip unreadible pictures insted of simple abort.
It wold make the program more robust.

Daniele

extract from error log.

[INFO] Loading 924 images
[WARNING] Cannot read /var/www/data/3c6d9526-3963-44b5-9db8-aedaf5f8035a/images/DJI_0199.JPG with PIL, fallback to cv2: cannot identify image file ‘/var/www/data/3c6d9526-3963-44b5-9db8-aedaf5f8035a/images/DJI_0199.JPG’
Traceback (most recent call last):
File “/code/opendm/get_image_size.py”, line 13, in get_image_size
with Image.open(file_path) as img:
File “/usr/local/lib/python3.9/dist-packages/PIL/Image.py”, line 2943, in open
raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file ‘/var/www/data/3c6d9526-3963-44b5-9db8-aedaf5f8035a/images/DJI_0199.JPG’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/code/run.py”, line 53, in
retcode = app.execute()
File “/code/stages/odm_app.py”, line 130, in execute
raise e
File “/code/stages/odm_app.py”, line 94, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 321, in run
self.process(self.args, outputs)
File “/code/stages/dataset.py”, line 112, in process
p = types.ODM_Photo(f)
File “/code/opendm/photo.py”, line 92, in init
self.parse_exif_values(path_file)
File “/code/opendm/photo.py”, line 274, in parse_exif_values
self.width, self.height = get_image_size.get_image_size(_path_file)
File “/code/opendm/get_image_size.py”, line 19, in get_image_size
width = img.shape[1]
AttributeError: ‘NoneType’ object has no attribute ‘shape’

[pierotofy]

Skip corrupted images

[ThorZ]

Sometimes DJI could record a white image, it has shape, has exif data but the pixels are all white. It will cause an error in opensfm stage (due to 0 feature detected), not sure if this case is worth handling as well?

Traceback (most recent call last):
  File "/code/SuperBuild/install/bin/opensfm/bin/opensfm_main.py", line 15, in <module>
    commands.command_runner(
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command_runner.py", line 38, in command_runner
    command.run(data, args)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/command.py", line 12, in run
    self.run_impl(data, args)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/commands/detect_features.py", line 11, in run_impl
    detect_features.run_dataset(dataset)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 68, in run_dataset
    parallel_map(process, arguments, processes, 1)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/context.py", line 52, in parallel_map
    res = Parallel(batch_size=batch_size)(delayed(func)(arg) for arg in args)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 1061, in __call__
    self.retrieve()
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 940, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 771, in get
    raise self._value
  File "/usr/lib/python3.8/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.8/dist-packages/joblib/_parallel_backends.py", line 595, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in __call__
    return [func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/joblib/parallel.py", line 262, in <listcomp>
    return [func(*args, **kwargs)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 137, in process
    run_detection(queue)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 171, in run_detection
    detect(image, image_array, segmentation_array, instances_array, data)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/actions/detect_features.py", line 236, in detect
    p_unmasked, f_unmasked, c_unmasked = features.extract_features(
  File "/code/SuperBuild/install/bin/opensfm/opensfm/features.py", line 560, in extract_features
    points, desc = extract_features_sift(image_gray, config, features_count)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/features.py", line 343, in extract_features_sift
    desc = root_feature(desc)
  File "/code/SuperBuild/install/bin/opensfm/opensfm/features.py", line 212, in root_feature
    s = np.sum(desc, 1)
  File "<__array_function__ internals>", line 5, in sum
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 2241, in sum
    return _wrapreduction(a, np.add, 'sum', axis, dtype, out, keepdims=keepdims,
  File "/usr/local/lib/python3.8/dist-packages/numpy/core/fromnumeric.py", line 87, in _wrapreduction
    return ufunc.reduce(obj, axis, dtype, out, **passkwargs)
numpy.AxisError: axis 1 is out of bounds for array of dimension 0

[pierotofy]

It would be good to handle it too; can you share one such image?

[daniele_rossii]

I try but th system do not load it !
see the error mssage.
I privat mssage the file to you.

D

[ThorZ]

I don’t have a real one in hand, I removed such images so I can move on. I created a fake one by editing all the values to white, which can reproduce the same error.

DJI_0013_0.JPG

[pierotofy]

Thanks @ThorZ. It would be really handy to have the real one though, because it might have special tags we can use to identify it (checking for white is slow and expensive).

[ThorZ]

It might be easier to handle it in opensfm, seems it only happens for ROOT_SIFT features, others are fine