Processing failed (4294967295)

Ibrahim_Ulin · 13 December 2021 22:27

I’ve been trying to process 612 images using i7-10510U and 16gb RAM.
Processing Node: node-odm-1 (manual)
Options: fast-orthophoto: true

After about an hour processing it failed as noticed in logs below.

[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_visualsfm --points “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\opensfm”
[INFO] running C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\opensfm\bin\opensfm export_ply --no-cameras --point-num-views “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\opensfm”
[INFO] Finished opensfm stage
[INFO] Running odm_filterpoints stage
[INFO] Filtering C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\opensfm\reconstruction.ply (statistical, meanK 16, standard deviation 2.5)
[INFO] running pdal pipeline -i “C:\Users\ibrah\AppData\Local\Temp\tmp0pnk3xdo.json”
[INFO] Finished odm_filterpoints stage
[INFO] Running odm_meshing stage
[INFO] Writing ODM 2.5D Mesh file in: C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\odm_25dmesh.ply
[WARNING] Maximum resolution set to 1.0 * (GSD - 10.0%) (12.84 cm / pixel, requested resolution was 5.00 cm / pixel)
[INFO] ODM 2.5D DSM resolution: 1.8324155329263385
[INFO] Created temporary directory: C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp
[INFO] Creating DSM for 2.5D mesh
[INFO] running pdal info “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_filterpoints\point_cloud.ply” > “C:\Users\ibrah\AppData\Local\Temp\tmp25uhmunp.json”
[INFO] Point cloud bounds are [minx: -763.8071289, maxx: 820.5772095] [miny: -757.6486816, maxy: 731.184021]
[INFO] DEM resolution is (813, 865), max tile size is 4096, will split DEM generation into 1 tiles
[INFO] Generating C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\mesh_dsm_r5.182853797135102_x0_y0.tif (max, radius: 5.182853797135102, resolution: 1.8324155329263385)
[INFO] running pdal pipeline -i “C:\Users\ibrah\AppData\Local\Temp\tmp1l9e6527.json”
[INFO] running gdalbuildvrt -input_file_list “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles_list.txt” “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.vrt”
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=8 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 30.7% “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.vrt” “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.tmp.tif”
Input file size is 865, 813
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=8 -co BIGTIFF=IF_SAFER --config GDAL_CACHEMAX 30.7% -outsize 10% 0 “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.tmp.tif” “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.small.tif”
0
…
10…20…
30
…
.40
.
…
50
.
.
.
60
.
.
.
70
.
.
.
80
.
.
.
90
.
.
.
Input file size is 865, 813
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdalbuildvrt -resolution highest -r bilinear “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\merged.vrt” “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.small_filled.tif” “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.tmp.tif”
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] running gdal_translate -co NUM_THREADS=8 -co TILED=YES -co BIGTIFF=IF_SAFER -co COMPRESS=DEFLATE --config GDAL_CACHEMAX 30.7% “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\merged.vrt” “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\tiles.tif”
Input file size is 865, 813
0…10…20…30…40…50…60…70…80…90…100 - done.
[INFO] Completed mesh_dsm.tif in 0:00:03.496680
[INFO] Sampling points from DSM: C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\mesh_dsm.tif
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\dem2points” -inputFile “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\mesh_dsm.tif” -outputFile “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\dsm_points.ply” -skirtHeightThreshold 1.5 -skirtIncrements 0.2 -skirtHeightCap 100
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\PoissonRecon” --in “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\tmp\dsm_points.ply” --out “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\odm_25dmesh.dirty.ply” --depth 11 --pointWeight 4 --samplesPerNode 1.0 --threads 7 --maxMemory 7 --bType 2 --linearFit
[WARNING] D:\a\ODM\ODM\SuperBuild\src\PoissonRecon\Src\FEMTree.IsoSurface.specialized.inl (Line 1896)
IsoSurfaceExtractor<3,float,struct VectorTypeUnion<float,struct Point<float,3>,struct EmptyVectorType > >::Extract
bad average roots: 16
[INFO] running “C:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\ReconstructMesh” -i “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\odm_25dmesh.dirty.ply” -o “C:\WebODM\resources\app\apps\NodeODM\data\7ccf4a40-f2e5-4520-a5a8-c1907e0185a7\odm_meshing\odm_25dmesh.ply” --remove-spikes 0 --remove-spurious 20 --smooth 0 --target-face-num 400000
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 3221225477
Traceback (most recent call last):
File “C:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
[Previous line repeated 2 more times]
File “C:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 327, in run
self.process(self.args, outputs)
File “C:\WebODM\resources\app\apps\ODM\stages\odm_meshing.py”, line 66, in process
mesh.create_25dmesh(tree.filtered_point_cloud, tree.odm_25dmesh,
File “C:\WebODM\resources\app\apps\ODM\opendm\mesh.py”, line 43, in create_25dmesh
mesh = screened_poisson_reconstruction(dsm_points, outMesh, depth=depth,
File “C:\WebODM\resources\app\apps\ODM\opendm\mesh.py”, line 207, in screened_poisson_reconstruction
system.run('“{reconstructmesh}” -i “{infile}” ’
File “C:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 3221225477

===== Done, human-readable information to follow… =====

[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. This is often a sign that the input data has some issues or the software cannot deal with it. Have you followed best practices for data acquisition? See Flying Tips — OpenDroneMap 3.1.7 documentation
100 - done.

I’ve processed these images using Agisoft and they turn out fine.
Images are available in 20211213 Rencana Huntara RAW - Google Drive
Any help or explanation may greatly help me. Thanks

smathermather · 14 December 2021 00:53

16GB RAM with 16-32 GB swap will let you process roughly 500 images.

We need some docs on memory usage, with this markdown as the starting place:

github.com

OpenDroneMap/ClusterODM/blob/master/docs/digitalocean.md

# Provider Configuration for DigitalOcean

Example configuration file:

```json
{
    "provider": "digitalocean",
    "accessToken": "CHANGEME",
    "s3": {
        "accessKey": "CHANGEME",
        "secretKey": "CHANGEME",
        "endpoint" :"sfo2.digitaloceanspaces.com",
        "bucket": "CHANGEME",
        "ignoreSSL": false
    },

    "createRetries": 10,
    "maxRuntime": -1,
    "maxUploadTime": -1,
    "dropletsLimit": 30,

This file has been truncated. show original

Saijin_Naib · 14 December 2021 13:05

Welcome, Ibrahim!

Sorry for the trouble.

Like Stephen noted, it is likely insufficient RAM+pagefile/swap.

Can you please provide some details about your Operating System, Processing Parameters, pagefile/swap, and WebODM version and install method?

Ibrahim_Ulin · 15 December 2021 02:31

Hi, thank you for all of your replies. I’ve found a workaround by updating my WebODM Native Windows version 1.9.11 build 45 to build 47. Somehow when I process the dataset using the same Processing Parameters and it works flawlessly.

Saijin_Naib · 15 December 2021 17:55

Then you hit upon a little bug we had where the last(ish) step would fail without a proper NVIDIA GPU supporting CUDA 11.x or above.

Glad you’re processing smoothly!

Show off some stuff when you get a chance!

system · 14 January 2022 17:55

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.