Hung on filterpoints for 46 hours

I am trying to process a 3d model of a commercial building. I had one reasonably successful result however I was looking for more detail so I added several additional oblique perimeter images. I did two things that I have never done before, I stepped up the gap fill to 5 and I also decided to try out using the boundary tool and created a .geojson shape to save time trimming in metashape.
I started processing almost exactly 48 hours ago. We seem to be stuck at Running odm_filterpoints stage. It has been sitting with no additional log changes since about 2 hours after processing started, so about 46 hours. There seems to be disk activity and processes are using resources.

There are only 840 images so it is not a huge project. I did restart the computer about 12 hours ago. When I opened WebODM, the processing restarted without any input from me. Does this amount of time on this stage sound right? I hate the idea of cancelling since there has been a big time investment. At the risk of knee jerking, can I restart, change options to NOT use the geojson and retain all the other work? I have to get this project completed and submitted.

In case this helps, (excuse the long insertion):

[INFO] Initializing ODM 2.8.8 - Sat Sep 10 23:40:57 2022
[INFO] ==============
[INFO] 3d_tiles: False
[INFO] auto_boundary: False
[INFO] auto_boundary_distance: 0
[INFO] boundary: {‘type’: ‘FeatureCollection’, ‘name’: ‘XXXX East Shape for Boundary JSON’, ‘crs’: {‘type’: ‘name’, ‘properties’: {‘name’: ‘urn:ogc:def:crs:OGC:1.3:CRS84’}}, ‘features’: [{‘type’: ‘Feature’, ‘properties’: {}, ‘geometry’: {‘type’: ‘Polygon’, ‘coordinates’: [[[-104.53599664194854, 50.44473878261318], [-104.5333371769824, 50.44475069955851], [-104.53330341230397, 50.44423330551543], [-104.53347620801125, 50.44422635396399], [-104.53344343641159, 50.443557018867956], [-104.53546037940869, 50.443551060395286], [-104.53597082190034, 50.44383607400443], [-104.53599664194854, 50.44473878261318]]]}}]}
[INFO] build_overviews: False
[INFO] camera_lens: auto
[INFO] cameras: {}
[INFO] cog: True
[INFO] copy_to: None
[INFO] crop: 3
[INFO] debug: False
[INFO] dem_decimation: 1
[INFO] dem_euclidean_map: False
[INFO] dem_gapfill_steps: 5
[INFO] dem_resolution: 5
[INFO] depthmap_resolution: 640
[INFO] dsm: False
[INFO] dtm: False
[INFO] end_with: odm_postprocess
[INFO] fast_orthophoto: False
[INFO] feature_quality: high
[INFO] feature_type: sift
[INFO] force_gps: False
[INFO] gcp: None
[INFO] geo: None
[INFO] gps_accuracy: 10
[INFO] ignore_gsd: False
[INFO] matcher_neighbors: 0
[INFO] matcher_type: flann
[INFO] max_concurrency: 12
[INFO] merge: all
[INFO] mesh_octree_depth: 11
[INFO] mesh_size: 300000
[INFO] min_num_features: 10000
[INFO] name: dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4
[INFO] no_gpu: False
[INFO] optimize_disk_space: False
[INFO] orthophoto_compression: DEFLATE
[INFO] orthophoto_cutline: False
[INFO] orthophoto_kmz: False
[INFO] orthophoto_no_tiled: False
[INFO] orthophoto_png: False
[INFO] orthophoto_resolution: 5
[INFO] pc_classify: False
[INFO] pc_copc: False
[INFO] pc_csv: False
[INFO] pc_ept: True
[INFO] pc_filter: 2.5
[INFO] pc_geometric: False
[INFO] pc_las: False
[INFO] pc_quality: high
[INFO] pc_rectify: False
[INFO] pc_sample: 0
[INFO] pc_tile: True
[INFO] primary_band: auto
[INFO] project_path: D:\WebODM\resources\app\apps\NodeODM\data
[INFO] radiometric_calibration: none
[INFO] rerun: None
[INFO] rerun_all: False
[INFO] rerun_from: [‘odm_postprocess’]
[INFO] resize_to: 2048
[INFO] rolling_shutter: False
[INFO] rolling_shutter_readout: 0
[INFO] sfm_algorithm: incremental
[INFO] skip_3dmodel: False
[INFO] skip_band_alignment: False
[INFO] skip_orthophoto: False
[INFO] skip_report: False
[INFO] sky_removal: False
[INFO] sm_cluster: None
[INFO] smrf_scalar: 1.25
[INFO] smrf_slope: 0.15
[INFO] smrf_threshold: 0.5
[INFO] smrf_window: 18.0
[INFO] split: 999999
[INFO] split_image_groups: None
[INFO] split_overlap: 150
[INFO] texturing_data_term: gmi
[INFO] texturing_keep_unseen_faces: False
[INFO] texturing_outlier_removal_type: gauss_clamping
[INFO] texturing_skip_global_seam_leveling: False
[INFO] texturing_skip_local_seam_leveling: False
[INFO] texturing_tone_mapping: none
[INFO] tiles: False
[INFO] time: False
[INFO] use_3dmesh: True
[INFO] use_exif: False
[INFO] use_fixed_camera_params: False
[INFO] use_hybrid_bundle_adjustment: False
[INFO] verbose: False
[INFO] ==============
[INFO] Running dataset stage
[INFO] Loading dataset from: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\images
[INFO] Loading images database: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\images.json
[INFO] Found 840 usable images
[INFO] Coordinates file already exist: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\odm_georeferencing\coords.txt
[INFO] Model geo file already exist: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\odm_georeferencing\odm_georeferencing_model_geo.txt
[INFO] Parsing SRS header: WGS84 UTM 13N
[INFO] Finished dataset stage
[INFO] Running split stage
[INFO] Normal dataset, will process all at once.
[INFO] Finished split stage
[INFO] Running merge stage
[INFO] Normal dataset, nothing to merge.
[INFO] Finished merge stage
[INFO] Running opensfm stage
[WARNING] D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\image_list.txt already exists, not rerunning OpenSfM setup
[WARNING] D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\exif already exists, not rerunning photo to metadata
[WARNING] Detect features already done: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\features exists
[WARNING] Match features already done: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\matches exists
[WARNING] Found a valid OpenSfM tracks file in: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\tracks.csv
[WARNING] Found a valid OpenSfM reconstruction file in: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\reconstruction.json
[INFO] Already extracted cameras
[INFO] Export reconstruction stats
[WARNING] Found existing reconstruction stats D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\stats\stats.json
[WARNING] Will skip exporting D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\reconstruction.geocoords.json
[WARNING] Already undistorted (nominal)
[WARNING] Found a valid OpenSfM NVM reconstruction file in: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\undistorted/reconstruction.nvm
[INFO] Finished opensfm stage
[INFO] Running openmvs stage
[WARNING] Found a valid OpenMVS reconstruction file in: D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\undistorted\openmvs\scene_dense_dense_filtered.ply
[INFO] Finished openmvs stage
[INFO] Running odm_filterpoints stage
[INFO] Filtering D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\undistorted\openmvs\scene_dense_dense_filtered.ply (statistical, meanK 16, standard deviation 2.5)
[INFO] Boundary [(-92.07411367236637, 53.55509557016194), (96.74846738902852, 56.06262409221381), (99.50717938668095, -1.4526216695085168), (87.24282889254391, -2.3026212891563773), (90.03701416833792, -76.71300192084163), (-53.172104792902246, -78.27273582108319), (-89.61402751423884, -46.80782552715391), (-92.07411367236637, 53.55509557016194)]
[INFO] running “D:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\FPCFilter” --input “D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\opensfm\undistorted\openmvs\scene_dense_dense_filtered.ply” --output “D:\WebODM\resources\app\apps\NodeODM\data\dc357c35-ee55-4b19-8dc2-3d50cb3f0ee4\odm_filterpoints\point_cloud.ply” --concurrency 12 --meank 16 --std 2.5 --boundary “C:\Users\LA_LEO~1\AppData\Local\Temp\tmpil3jjh6j.boundary.json”

Strange, it could be a problem with the GeoJSON. You should be able to remove it as an option and restart from filter points.

I really do apreciate the fast response. I chose cancel, then deleted the geojason from options and restarted from filter points, I will let you know how this story unfolds.

I should also say that I am being fussy and pushing ODM on this model. The first version was quite good. I spent quite a bit of time smoothing in meshlab but in the end decided that I could not live with the big gap (which is the store entrance). Quite a lot is riding on this model, so I need it to be as good as possible. First round I did not include a couple hundred oblique shots. Even if this one fails, I think I will get positive results with the first version.

1 Like

I made the suggested changes and let it run for another 5 hours. Processing progress did not change. Unfortunately because of time, I had to abort.