Cannot process dataset lightining

Dataset URL (Dropbox, Google Drive, GitHub…): 100MEDIA - Google Drive
Number of images: 196
Size in MB: 1GB
EXIF GPS (yes/no): YES
GCP (yes/no): YES
RTK (yes/no): YES

I configurated the mission using pix4dcapture, with both ovelap 80%

Comment/Description:

Good day, I have a dataset with 196 images, flight height: 55m, drone: phantom 4.

I used my ODM in my pc, but I cannot process the information, I thougth that my memory was the problem 8GB, so I used my free lightining account but I received this error “cannot process dataset”, and also I received this email:

Your task [Calle 4 - 7/11/2021] could not complete. Here’s a copy of the error message:

[WARNING] Really low resolution DEM requested (1, 1) will set floor at 64 pixels. Resolution changed to 1.8868418007462628. The scale of this reconstruction might be off.
[INFO] DEM resolution is (38, 64), max tile size is 4096, will split DEM generation into 1 tiles
[INFO] Generating /var/www/data/47d586ea-a0e5-4764-bddc-6c28ad5a0370/odm_meshing/tmp/mesh_dsm_r2.668397264667838_x0_y0.tif (max, radius: 2.668397264667838, resolution: 1.8868418007462628)
[INFO] running pdal pipeline -i /tmp/tmpk2kc7_ac.json > /dev/null 2>&1
[INFO] running gdalbuildvrt -input_file_list “/var/www/data/47d586ea-a0e5-4764-bddc-6c28ad5a0370/odm_meshing/tmp/tiles_list.txt” “/var/www/data/47d586ea-a0e5-4764-bddc-6c28ad5a0370/odm_meshing/tmp/tiles.vrt”
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=24 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 47.2% “/var/www/data/47d586ea-a0e5-4764-bddc-6c28ad5a0370/odm_meshing/tmp/tiles.vrt” “/var/www/data/47d586ea-a0e5-4764-bddc-6c28ad5a0370/odm_meshing/tmp/tiles.tmp.tif”
Input file size is 1, 1
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=24 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 47.2% -outsize 10% 0 “/var/www/data/47d586ea-a0e5-4764-bddc-6c28ad5a0370/odm_meshing/tmp/tiles.tmp.tif” “/var/www/data/47d586ea-a0e5-4764-bddc-6c28ad5a0370/odm_meshing/tmp/tiles.small.tif”
ERROR 5: Invalid output width: 0.1
Input file size is 1, 1

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 89, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 340, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 340, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 340, in run
self.next_stage.run(outputs)
[Previous line repeated 3 more times]
File “/code/opendm/types.py”, line 321, in run
self.process(self.args, outputs)
File “/code/stages/odm_meshing.py”, line 66, in process
mesh.create_25dmesh(tree.filtered_point_cloud, tree.odm_25dmesh,
File “/code/opendm/mesh.py”, line 26, in create_25dmesh
commands.create_dem(
File “/code/opendm/dem/commands.py”, line 228, in create_dem
run('gdal_translate ’
File “/code/opendm/system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See https://docs.opendronemap.org/flying.html

I checked the images position in QGIS, and they are fine, also I processed 36 images in my ODM and worked that process was done without GCP.

Finally, I processed the dataset (196 pictures) without GCP in ODM lightining , and I had good results, (the forum does not allow me to attach more than one photo) . So the problem is when I use GCP. I attached in the Google drive URL my GCP using GCP ODM interface.

So, please I need some help, because I want process my information using GCP (made with RTK).

Best regards,

Enrique Torres
Bogota
ColombiaIMG-20210713-WA0000

1 Like

Welcome!

Sorry you’re having trouble with processing with your GCP.

At a quick glance, it looks like your gcp_list.txt file is malformed:
image

I don’t think you meant to have all five GCPs exactly co-located in one spot (with really low precision values for RTK [-74.00, 5.01, 2594.479])

Saijin thanks, now I can see the problem, the GCP tool don’t receive the decimal precision (I think that this tool need some revision), so I will change the txt file, because these are my GCP RTK for my 5 GCP

Name Easting Northing Elevation
1 -73.99563011 5.01366258 2597.356
2 -73.99516785 5.01305747 2595.41
3 -73.99584131 5.01264336 2595.921
4 -73.99678861 5.01095004 2595.95
5 -73.99645004 5.01078738 2594.479

Thanks I will try, and I tell you my results.

Best regards.

1 Like

Saijin again ODM Lightning send me an error.

Screenshot from 2021-07-13 10-44-39

I used the new GCP file:

+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs ,
-73.99563011,5.01366258,2594.479,3101.73,569.58,DJI_0043.JPG,5
-73.99563011,5.01366258,2594.479,3140.9,1215.63,DJI_0044.JPG,5
-73.99563011,5.01366258,2594.479,3193.64,1917.99,DJI_0045.JPG,5
-73.99563011,5.01366258,2594.479,3234.73,2568.03,DJI_0046.JPG,5
-73.99563011,5.01366258,2594.479,1979.91,425.94,DJI_0057.JPG,5
-73.99563011,5.01366258,2594.479,2031.81,1060.85,DJI_0058.JPG,5
-73.99563011,5.01366258,2594.479,2086.11,1737.98,DJI_0059.JPG,5
-73.99563011,5.01366258,2594.479,2139.35,2401.87,DJI_0060.JPG,5
-73.99563011,5.01366258,2594.479,571.61,142,DJI_0106.JPG,5
-73.99563011,5.01366258,2594.479,648.4,804.01,DJI_0107.JPG,5
-73.99563011,5.01366258,2594.479,725.83,1485.52,DJI_0108.JPG,5
-73.99563011,5.01366258,2594.479,797.72,2121.44,DJI_0109.JPG,5
-73.99563011,5.01366258,2594.479,874.76,2774.45,DJI_0110.JPG,5
-73.99678861,5.01095004,2595.95,2741.12,224.19,DJI_0107.JPG,4
-73.99678861,5.01095004,2595.95,2821.6,946.14,DJI_0108.JPG,4
-73.99678861,5.01095004,2595.95,2892.44,1617.54,DJI_0109.JPG,4
-73.99678861,5.01095004,2595.95,2967.81,2310.86,DJI_0110.JPG,4
-73.99678861,5.01095004,2595.95,2340.06,513.92,DJI_0122.JPG,4
-73.99678861,5.01095004,2595.95,2425.93,1249.99,DJI_0123.JPG,4
-73.99678861,5.01095004,2595.95,2512.3,1966.89,DJI_0124.JPG,4
-73.99678861,5.01095004,2595.95,2598.95,2706.65,DJI_0125.JPG,4
-73.99678861,5.01095004,2595.95,138.33,500.01,DJI_0174.JPG,4
-73.99678861,5.01095004,2595.95,214.66,1205.05,DJI_0175.JPG,4
-73.99678861,5.01095004,2595.95,301.85,1901.2,DJI_0176.JPG,4
-73.99678861,5.01095004,2595.95,382.67,2612.45,DJI_0177.JPG,4
-73.99584131,5.01264336,2595.921,3531.36,514.32,DJI_0092.JPG,3
-73.99584131,5.01264336,2595.921,3594.55,1204.33,DJI_0093.JPG,3
-73.99584131,5.01264336,2595.921,3663.73,1915.05,DJI_0094.JPG,3
-73.99584131,5.01264336,2595.921,3729.01,2598.88,DJI_0095.JPG,3
-73.99584131,5.01264336,2595.921,1570.82,297.48,DJI_0136.JPG,3
-73.99584131,5.01264336,2595.921,1650.99,1006.48,DJI_0137.JPG,3
-73.99584131,5.01264336,2595.921,1734.09,1700.74,DJI_0138.JPG,3
-73.99584131,5.01264336,2595.921,1806.09,2424.2,DJI_0139.JPG,3
-73.99584131,5.01264336,2595.921,822.72,36.45,DJI_0158.JPG,3
-73.99584131,5.01264336,2595.921,906.19,751.98,DJI_0159.JPG,3
-73.99584131,5.01264336,2595.921,983.08,1455.37,DJI_0160.JPG,3
-73.99584131,5.01264336,2595.921,1142.5,2854.95,DJI_0161.JPG,3
-73.99516785,5.01305747,2595.41,1051.75,339.4,DJI_0076.JPG,2
-73.99516785,5.01305747,2595.41,1086.97,970.31,DJI_0077.JPG,2
-73.99516785,5.01305747,2595.41,1130.69,1692.98,DJI_0078.JPG,2
-73.99516785,5.01305747,2595.41,1172.81,2399.59,DJI_0079.JPG,2
-73.99516785,5.01305747,2595.41,1404.36,102.2,DJI_0086.JPG,2
-73.99516785,5.01305747,2595.41,1480.14,832.28,DJI_0087.JPG,2
-73.99516785,5.01305747,2595.41,1550.26,1474.6,DJI_0088.JPG,2
-73.99516785,5.01305747,2595.41,1689.25,2832.37,DJI_0089.JPG,2
-73.99563011,5.01366258,2597.356,3173,610.53,DJI_0151.JPG,1
-73.99563011,5.01366258,2597.356,3283.56,1451.37,DJI_0152.JPG,1
-73.99563011,5.01366258,2597.356,3367.5,2211.12,DJI_0153.JPG,1
-73.99563011,5.01366258,2597.356,3458.17,2928.59,DJI_0154.JPG,1
-73.99563011,5.01366258,2597.356,2015.36,713.95,DJI_0210.JPG,1

And I received this mail, but I can not find the error:

Your task [Calle 4 - 7/11/2021] could not complete. Here’s a copy of the error message:

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,3663.73,1915.05,DJI_0094.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,3729.01,2598.88,DJI_0095.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,1570.82,297.48,DJI_0136.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,1650.99,1006.48,DJI_0137.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,1734.09,1700.74,DJI_0138.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,1806.09,2424.2,DJI_0139.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,822.72,36.45,DJI_0158.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,906.19,751.98,DJI_0159.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,983.08,1455.37,DJI_0160.JPG,3

[WARNING] Malformed GCP line: -73.99584131,5.01264336,2595.921,1142.5,2854.95,DJI_0161.JPG,3

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1051.75,339.4,DJI_0076.JPG,2

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1086.97,970.31,DJI_0077.JPG,2

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1130.69,1692.98,DJI_0078.JPG,2

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1172.81,2399.59,DJI_0079.JPG,2

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1404.36,102.2,DJI_0086.JPG,2

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1480.14,832.28,DJI_0087.JPG,2

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1550.26,1474.6,DJI_0088.JPG,2

[WARNING] Malformed GCP line: -73.99516785,5.01305747,2595.41,1689.25,2832.37,DJI_0089.JPG,2

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,3173,610.53,DJI_0151.JPG,1

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,3283.56,1451.37,DJI_0152.JPG,1

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,3367.5,2211.12,DJI_0153.JPG,1

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,3458.17,2928.59,DJI_0154.JPG,1

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,2015.36,713.95,DJI_0210.JPG,1

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,2090.25,1540.06,DJI_0211.JPG,1

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,2157.14,2304.2,DJI_0212.JPG,1

[WARNING] Malformed GCP line: -73.99563011,5.01366258,2597.356,2211.25,2920.62,DJI_0213.JPG,1

Traceback (most recent call last):

File “/code/run.py”, line 54, in

retcode = app.execute()

File “/code/stages/odm_app.py”, line 125, in execute

raise e

File “/code/stages/odm_app.py”, line 89, in execute

self.first_stage.run()

File “/code/opendm/types.py”, line 321, in run

self.process(self.args, outputs)

File “/code/stages/dataset.py”, line 144, in process

reconstruction.georeference_with_gcp(tree.odm_georeferencing_gcp,

File “/code/opendm/types.py”, line 79, in georeference_with_gcp

raise RuntimeError(“This GCP file does not have any entries. Are the entries entered in the proper format?”)

RuntimeError: This GCP file does not have any entries. Are the entries entered in the proper format?

1 Like

This is what I got when I loaded the GCP file text you pasted above:
image
Looks to be malformed again.

Here’s an example to work from:

+proj=utm +zone=46 +ellps=WGS84 +datum=WGS84 +units=m +no_defs
230649.94 2631114.55 4 2816.67 1477.78 DJI_0067.JPG gcp01

Here’s the Format Specification:

  1. The first row must be a valid projection string (PROJ) or EPSG code
  2. After the header your gcp_list.txt must contain at least 6 columns separated by spaces or tabs:

geo_x geo_y geo_z im_x im_y image_name [gcp_name] [extra1] [extra2] …

  1. The first 3 colums are the geo coordinates in the selected coordinate system units: (latitude, longitude, elevation) or (northing, easting, elevation)
  2. Elevation values can be set to “NaN” to indicate no value
  3. The 4th and 5th columns contain the GCP pixel coordinates in the image
  4. The 6th column contains the image file name
  5. The 7th column (optional) contains the name of the GCP. If this information is not provided, the names of the GCPs will be autogenerated
  6. Extra columns are allowed

Saijin thanks a lot of, the last problem was about the delimiter character, I was using comma, and ODM use space o tab.

Finally, I have my processing with GCP.

Thanks.

Best regards.

Enrique Torres

2 Likes

Sweet! Show us a screencap when it’s done (if you can/if you’d like). We all love seeing new cool data!

1 Like

Saijin good day, yesterday I run the process, and I saw that one GCP was wrong, after that check in detail my GCP and again I started a new process, but after 9 hours the process don’t stop.
Sorry, I need your help, could you process my pictures? because I need an expert opinion… I can not see what happen.
My pictures are totally open, if any person want to use, feel free.
My dataset and GCP are in 100MEDIA - Google Drive

This folder is totally open.

Please, I need another opinion about what is my error.

Best regards.

Enrique Torres 9 hours processing dataset

2 Likes

Looks to be malformed again.
image

It seems like you were using double-spaces as column separators which is not to spec. Single-space only.

  

After doing a Find/Replace in Notepad, the GCP file seems to be well-formed:
image

Also, I see in your google drive the gcp file is not named properly. It needs to be named gcp_list.txt to be used properly.

Looks really good! Exceptionally well-done survey/GCP:
image

image


Options: debug: true, dsm: true, verbose: true

Enrique_Torres_Moya_8283_Report.pdf (8.0 MB)

1 Like

Thanks Saijin at the momment I fixed the file name and the double espace, and I am processing, so when the software finishes I will be reporting the result.

But great to see the results…

Another question, what is the program that you use to check the well-formed format??? It is possible to use, open??

Best regards

Enrique Torres M

1 Like

Saijin finally the software stopped, something strange happened, I received the email with the results, but my computer was still processing.

I share my results.

Thanks a lot.

Best regards.

Enrique Torres MEnrique_results_17jul2021

1 Like