Post processing after ODM

General Help
16

Hello again!
Thanks @smathermather for reopening the post! :pray:

Adventures since last post

So, I am back after different adventures:

  1. Unfortunate feature file corruption after 5 days of OpenSFM step. :see_no_evil: After some debugging and thanks to the help of an OpenSFM contibutor, I could fix the file and continue.

  2. Crash at the meshing step because each thread takes 25go of RAM/swap and I “only” have 160go of it. :sweat_smile: Easily fixed by setting max-concurrency to 6 for this step only.

    I took a look to the source code (permalink). Maybe this could be avoided by launching a single thread, tracking its memory usage and adapt max_workers before calling parallel_map. What do you think?

  3. Being a little bit greedy, I set the pc-quality argument as high at the same time as I increased min-num-features and featrue-quality. I maxed out my 2 TB hard drive. :sweat_smile:

Current issue

So, I started from the top with these parameters:

  • min-num-features: 13 000
  • feature-quality: ultra
  • gps-accuracy: 10
  • mesh-size: 1 000 000 (Am I still too greedy here?)

Now I a am facing a new issue. I investigated the source. I have an idea that may help but I would like your opinion about it.

Description

The process is crashing at the meshing step when it opens tiles.tif with rasterio in median_smoothing.

Here is the complete stack trace:

Stack trace 
Traceback (most recent call last):
   File "/code/run.py", line 59, in <module>
     retcode = app.execute()
   File "/code/stages/odm_app.py", line 117, in execute
     raise e
   File "/code/stages/odm_app.py", line 81, in execute
     self.first_stage.run()
   File "/code/opendm/types.py", line 398, in run
     self.next_stage.run(outputs)
   File "/code/opendm/types.py", line 398, in run
     self.next_stage.run(outputs)
   File "/code/opendm/types.py", line 398, in run
     self.next_stage.run(outputs)
   [Previous line repeated 3 more times]
   File "/code/opendm/types.py", line 377, in run
     self.process(self.args, outputs)
   File "/code/stages/odm_meshing.py", line 54, in process
     mesh.create_25dmesh(tree.filtered_point_cloud, tree.odm_25dmesh,
   File "/code/opendm/mesh.py", line 25, in create_25dmesh
     commands.create_dem(
   File "/code/opendm/dem/commands.py", line 258, in create_dem
     median_smoothing(geotiff_path, output_path, num_workers=max_workers)
   File "/code/opendm/dem/commands.py", line 320, in median_smoothing
     arr = img.read()[0]
   File "rasterio/_io.pyx", line 336, in rasterio._io.DatasetReaderBase.read
numpy.core._exceptions.MemoryError: Unable to allocate 352. GiB for an array with shape (1, 257021, 367258) and data type float32

files.tmp.tif is 352 GiB. Gdal has no problem manage the size in gdal_translate and gdalbuildvrt. But then, trying to open tiles.tif (which is only 7.2 GB but has the same shape and type as files.tmp.tif) all at once with rasterio breaks.

Fix idea

I see that windowing is used in median_smoothing. Rasterio provides windowed reading and writing.
I think using these features should help. What do you think?

I could try it and make a PR if it helps. :smile:

2 Likes