Fault with gcp_list.txt processing

#1

I have a test GCP file that’s failing to process. The error is:

2019-04-02 16:40:50,024 INFO: Starting incremental reconstruction 
Traceback (most recent call last): 
File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module> command.run(args) 
File "/code/SuperBuild/src/opensfm/opensfm/commands/reconstruct.py", line 21, in run report = reconstruction.incremental_reconstruction(data) 
File "/code/SuperBuild/src/opensfm/opensfm/reconstruction.py", line 1158, in incremental_reconstruction gcp = data.load_ground_control_points() 
File "/code/SuperBuild/src/opensfm/opensfm/dataset.py", line 671, in load_ground_control_points fin, self.load_reference_lla(), exif) File "/code/SuperBuild/src/opensfm/opensfm/io.py", line 429, in read_ground_control_points_list for line in lines] 
File "/code/SuperBuild/src/opensfm/opensfm/io.py", line 373, in _read_gcp_list_line d = exif[shot_id] KeyError: 'FF9466AA-545C-11E9-9194-96FE18339E5A.JPG' 
Traceback (most recent call last): File "/code/run.py", line 47, in <module> plasm.execute(niter=1) 
File "/code/scripts/run_opensfm.py", line 136, in process (context.pyopencv_path, context.opensfm_path, tree.opensfm)) 
File "/code/opendm/system.py", line 34, in run raise 
Exception("Child returned {}".format(retcode)) 
Exception: Child returned 1

Here’s the GCP file, which is generated by software. The difference that I see in the below vs the documented example is that “zone=nnX”, while the docs give “zone=nn” as an example [Edit: have since tested with nn and it’s the same]. Also, none of the points are exactly aligned with each other, but I would not expect that to break it.

+proj=utm +zone=60G +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62 365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG
373593.617824043 5453757.40217389 62.1 301 464 159ED656-545D-11E9-AAA2-30FF18339E5A.JPG
373587.882779595 5453776.17031264 62.1 341 329 159ED656-545D-11E9-AAA2-30FF18339E5A.JPG
373573.364856366 5453685.5996143 62 93 443 1604E2C0-545D-11E9-A65C-31FF18339E5A.JPG
373573.345407249 5453685.29933688 62 82 376 1667E76C-545D-11E9-A29C-34FF18339E5A.JPG
373658.220456689 5453605.40830835 61.9 411 468 41DD6138-545D-11E9-B545-C6FF18339E5A.JPG
373645.944898609 5453599.64024765 61.9 480 276 4208F442-545D-11E9-B908-C9FF18339E5A.JPG
373648.791029143 5453597.11159145 61.9 357 178 4890C1F0-545D-11E9-8467-050019339E5A.JPG
373645.675837941 5453598.50240016 62 366 250 48933C96-545D-11E9-B2A0-020019339E5A.JPG
373604.248721612 5453595.09279158 62 99 119 48933C96-545D-11E9-B2A0-020019339E5A.JPG
373649.502037897 5453596.93745791 61.9 340 75 49135BF6-545D-11E9-8B5E-080019339E5A.JPG
373526.827142429 5453565.58490677 61.9 189 277 4B569BF0-5460-11E9-9D44-048918339E5A.JPG
373549.698430207 5453585.96916432 61.9 410 256 4B569BF0-5460-11E9-9D44-048918339E5A.JPG
373526.490157469 5453565.00313902 62 178 173 51E52F90-5460-11E9-9343-388918339E5A.JPG
373550.232046937 5453587.19923722 62 403 161 51E52F90-5460-11E9-9343-388918339E5A.JPG
373590.215335754 5453556.66935243 62 520 471 51E52F90-5460-11E9-9343-388918339E5A.JPG
373549.95076296 5453586.64822875 62 387 56 52E509EC-5460-11E9-8713-408918339E5A.JPG
373526.20112618 5453564.90120102 62 165 70 52E509EC-5460-11E9-8713-408918339E5A.JPG
373590.373228336 5453556.79972759 62 507 372 52E509EC-5460-11E9-8713-408918339E5A.JPG
373775.520014271 5453307.21482524 62 317 396 618E1A60-5460-11E9-A82F-BB8918339E5A.JPG
373775.483168754 5453306.93006469 61.9 320 198 6381A2E2-5460-11E9-8040-D08918339E5A.JPG
373775.427419084 5453306.99402978 61.9 217 295 64F452E6-5460-11E9-8ECE-ED8918339E5A.JPG
373775.384577588 5453307.18002085 61.9 231 201 66E0E4CA-5460-11E9-AFF9-008A18339E5A.JPG
373775.45078121 5453306.97551692 61.9 318 297 67576C9E-5460-11E9-8F54-FE8918339E5A.JPG

This is my first time using GCPs - what am I missing?

0 Likes

#2

I think there’s a typo. Did you mean to write 60S instead of 60G?

0 Likes

#3

No, not unless I’m misunderstanding what it is. 60S is nowhere http://www.dmap.co.uk/utmworld.htm?sa=X&ved=2ahUKEwiZlLH4iLLhAhXIfX0KHYJNAa0Q9QEwAHoECAsQAg

Seems to complain about the file name - KeyError: ‘FF9466AA-545C-11E9-9194-96FE18339E5A.JPG’

0 Likes

#4

Here is all of the ways that I’ve tried so far, with no success on any:

[root@survey files]# head -2 */gcp_list.txt | grep -v '=='
WGS84
+proj=utm +zone=60 +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs

WGS84 UTM 60S
-41.05672687987802 175.49570524999876 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373737.334358417 5453885.45891182 62 550 40 FF9466AA-545C-11E9-9194-96FE18339E5A.JPG

WGS84 UTM 60G
-41.05672687987802 175.49570524999876 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60 +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
-41.05672687987802 175.49570524999876 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +ellps=WGS84 +datum=WGS84 +units=m +no_defs
-41.05672687987802 175.49570524999876 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

-41.05672687987802 175.49570524999876 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG
-41.05689816921059 175.49577691137574 62.1  301 464 159ED656-545D-11E9-AAA2-30FF18339E5A.JPG

+proj=utm +zone=60 +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60S +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

WGS84 UTM 60S
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

WGS84 UTM 60G
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60 +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
-41.05672687987802 175.49570524999876 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60 +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60 +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373587.267613245 5453776.31470008 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60 +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373593.015880765 5453855.64580222 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60 +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373593.015880765 5453855.64580222 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +south +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373593.015880765 5453855.64580222 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60G +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373593.015880765 5453855.64580222 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

+proj=utm +zone=60S +ellps=WGS84 +datum=WGS84 +units=m +no_defs
373593.015880765 5453855.64580222 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

WGS84 UTM 60G
373593.015880765 5453855.64580222 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

WGS84 UTM 60S
373593.015880765 5453855.64580222 62  365 446 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG

Each job takes ~28-32 minutes to process before it dies with the key error above.

0 Likes

#5

Maybe there’s something wrong with FF9466AA-545C-11E9-9194-96FE18339E5A.JPG. What happens if you remove the entries for that particular image?

0 Likes

#6

The images themselves seem OK. You’ll see that I also tried 1517E3F8-545D-11E9-AB00-2DFF18339E5A.JPG which also fails.

I’ve done another 30 or so more tests today; rounding, converting lat/long to UTM differently, changing the headers - you name it. Then it occurred to me that I hadn’t actually tried renaming the files and that the software could in fact be choking on either the length or the minuses (classic unescaped variable issue). I gave them their original names; sure enough, it’s swallowed it all so far.

Before:

    [DEBUG]   running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm reconstruct /www/data/41c105ba-f72d-44a0-90a5-8809313d7bb7/opensfm
    /code/SuperBuild/src/opensfm/opensfm/config.py:141: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    return yaml.load(default_config_yaml)
    /code/SuperBuild/src/opensfm/opensfm/config.py:150: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    new_config = yaml.load(fin)
    2019-04-04 16:05:21,859 INFO: Starting incremental reconstruction
    Traceback (most recent call last):
    File "/code/SuperBuild/src/opensfm/bin/opensfm", line 34, in <module>
    command.run(args)
    File "/code/SuperBuild/src/opensfm/opensfm/commands/reconstruct.py", line 21, in run
    report = reconstruction.incremental_reconstruction(data)
    File "/code/SuperBuild/src/opensfm/opensfm/reconstruction.py", line 1158, in incremental_reconstruction
    gcp = data.load_ground_control_points()
    File "/code/SuperBuild/src/opensfm/opensfm/dataset.py", line 671, in load_ground_control_points
    fin, self.load_reference_lla(), exif)
    File "/code/SuperBuild/src/opensfm/opensfm/io.py", line 429, in read_ground_control_points_list
    for line in lines]
    File "/code/SuperBuild/src/opensfm/opensfm/io.py", line 373, in _read_gcp_list_line
    d = exif[shot_id]
    KeyError: '1604E2C0-545D-11E9-A65C-31FF18339E5A.JPG'
    Traceback (most recent call last):
    File "/code/run.py", line 47, in <module>
    plasm.execute(niter=1)
    File "/code/scripts/run_opensfm.py", line 136, in process
    (context.pyopencv_path, context.opensfm_path, tree.opensfm))
    File "/code/opendm/system.py", line 34, in run
    raise Exception("Child returned {}".format(retcode))
    Exception: Child returned 1

After:

    [DEBUG]   running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm create_tracks /www/data/d0d87236-26d8-48ed-ab11-16b3dcf771b8/opensfm
    /code/SuperBuild/src/opensfm/opensfm/config.py:141: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    return yaml.load(default_config_yaml)
    /code/SuperBuild/src/opensfm/opensfm/config.py:150: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    new_config = yaml.load(fin)
    2019-04-04 17:45:44,833 INFO: reading features
    2019-04-04 17:45:58,470 DEBUG: Merging features onto tracks
    2019-04-04 17:46:12,237 DEBUG: Good tracks: 304085
    [DEBUG]   running PYTHONPATH=/code/SuperBuild/install/lib/python2.7/dist-packages /code/SuperBuild/src/opensfm/bin/opensfm reconstruct /www/data/d0d87236-26d8-48ed-ab11-16b3dcf771b8/opensfm
    /code/SuperBuild/src/opensfm/opensfm/config.py:141: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    return yaml.load(default_config_yaml)
    /code/SuperBuild/src/opensfm/opensfm/config.py:150: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
    new_config = yaml.load(fin)
    2019-04-04 17:47:10,148 INFO: Starting incremental reconstruction
    2019-04-04 17:47:53,857 INFO: Starting reconstruction with DJI_0273.JPG and DJI_0274.JPG
    2019-04-04 17:47:54,646 INFO: Two-view reconstruction inliers: 4545 / 4545
    2019-04-04 17:47:55,308 INFO: Triangulated: 4545
    2019-04-04 17:47:55,408 DEBUG: Ceres Solver Report: Iterations: 1, Initial cost: 2.417579e+01, Final cost: 2.226611e+01, Termination: CONVERGENCE
    2019-04-04 17:47:56,221 DEBUG: Ceres Solver Report: Iterations: 1, Initial cost: 2.268571e+01, Final cost: 2.204476e+01, Termination: CONVERGENCE
    2019-04-04 17:47:58,165 DEBUG: Ceres Solver Report: Iterations: 50, Initial cost: 4.180935e+01, Final cost: 3.710918e+01, Termination: NO_CONVERGENCE
    2019-04-04 17:47:58,168 INFO: Removed outliers: 0
    2019-04-04 17:47:58,329 INFO: -------------------------------------------------------
    2019-04-04 17:47:58,421 INFO: DJI_0272.JPG resection inliers: 1997 / 2034

I’ve got another test running that replaces the minuses with underscores - that should rule out whether they’re the issue or whether it’s the file name length. Will report back shortly.

0 Likes

#7

Confirmed - hyphens break it. Substituting the hyphens to underscores fixes it. This is a bug for sure.

0 Likes

#8
1 Like

#9

@ITWarrior – can you also submit an issue at https://github.com/mapillary/OpenSfM/issues?

0 Likes