MemoryError: Unable to allocate 91.9 GiB for an array with shape

Hi all

I’m using an aws ec2 with the following details:

os: linux 2
server type: r5ad.4xlarge
volume size: 200

total size of input images: 1.2 GiB, total 101 images

docker command (using python subprocess.run):

docker run -it --rm -v /home/ec2-user/proj:/datasets/code opendronemap/odm
–project-path /datasets --split-overlap 40 --dsm
–pc-quality lowest --fast-orthophoto --use-hybrid-bundle-adjustment
–split 100 --orthophoto-compression LZW --texturing-outlier-removal-type none
–ignore-gsd --skip-3dmodel --orthophoto-resolution 0.2 --dem-resolution 10 --dtm
–pc-rectify --orthophoto-png

I’m getting this error:

Cleaning up temporary files…
[INFO] Generated cutline file: /datasets/code/submodels/submodel_0000/odm_orthophoto/grass_cutline_tmpdir/cutline.gpkg → /datasets/code/submodels/submodel_0000/odm_orthophoto/cutline.gpkg
[INFO] Computing mask raster: /datasets/code/submodels/submodel_0000/odm_orthophoto/odm_orthophoto_cut.tif
2021-02-23 11:08:30,407 DEBUG: Entering env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,407 DEBUG: Starting outermost env
2021-02-23 11:08:30,407 DEBUG: No GDAL environment exists
2021-02-23 11:08:30,408 DEBUG: New GDAL environment <fiona._env.GDALEnv object at 0x7f848dbc9a00> created
2021-02-23 11:08:30,408 DEBUG: Logging error handler pushed.
2021-02-23 11:08:30,408 DEBUG: All drivers registered.
2021-02-23 11:08:30,408 DEBUG: GDAL_DATA found in environment: ‘/usr/local/lib/python3.8/dist-packages/rasterio/gdal_data’.
2021-02-23 11:08:30,409 DEBUG: PROJ data files are available at built-in paths
2021-02-23 11:08:30,409 DEBUG: Started GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9a00>.
2021-02-23 11:08:30,409 DEBUG: Updated existing <fiona._env.GDALEnv object at 0x7f848dbc9a00> with options {}
2021-02-23 11:08:30,409 DEBUG: Entered env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,409 DEBUG: Got a copy of environment <fiona._env.GDALEnv object at 0x7f848dbc9a00> options
2021-02-23 11:08:30,409 DEBUG: Credentialized: {‘CHECK_WITH_INVERT_PROJ’: True, ‘GTIFF_IMPLICIT_JPEG_OVR’: False, ‘FIONA_ENV’: True, ‘GDAL_DATA’: ‘/usr/local/lib/python3.8/dist-packages/rasterio/gdal_data’}
2021-02-23 11:08:30,428 DEBUG: Exiting env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,428 DEBUG: Cleared existing <fiona._env.GDALEnv object at 0x7f848dbc9a00> options
2021-02-23 11:08:30,428 DEBUG: Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9a00>.
2021-02-23 11:08:30,428 DEBUG: Error handler popped.
2021-02-23 11:08:30,428 DEBUG: Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9a00>.
2021-02-23 11:08:30,428 DEBUG: Exiting outermost env
2021-02-23 11:08:30,428 DEBUG: Exited env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,428 DEBUG: Entering env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:08:30,428 DEBUG: Starting outermost env
2021-02-23 11:08:30,428 DEBUG: No GDAL environment exists
2021-02-23 11:08:30,428 DEBUG: New GDAL environment <fiona._env.GDALEnv object at 0x7f848dbc9610> created
2021-02-23 11:08:30,428 DEBUG: Logging error handler pushed.
2021-02-23 11:08:30,429 DEBUG: All drivers registered.
2021-02-23 11:08:30,429 DEBUG: GDAL_DATA found in environment: ‘/usr/local/lib/python3.8/dist-packages/rasterio/gdal_data’.
2021-02-23 11:08:30,429 DEBUG: PROJ data files are available at built-in paths
2021-02-23 11:08:30,429 DEBUG: Started GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9610>.
2021-02-23 11:08:30,429 DEBUG: Updated existing <fiona._env.GDALEnv object at 0x7f848dbc9610> with options {}
2021-02-23 11:08:30,429 DEBUG: Entered env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:08:30,430 DEBUG: OLC_FASTSETNEXTBYINDEX: 0
2021-02-23 11:08:30,431 DEBUG: OLC_FASTFEATURECOUNT: 1
2021-02-23 11:08:30,431 DEBUG: Next index: 0
2021-02-23 11:08:30,431 DEBUG: Next index: 1
2021-02-23 11:08:30,442 DEBUG: Next index: 2
2021-02-23 11:08:30,443 DEBUG: Next index: 3
2021-02-23 11:08:30,443 DEBUG: Next index: 4
2021-02-23 11:08:30,443 DEBUG: Next index: 5
2021-02-23 11:08:30,443 DEBUG: Next index: 6
2021-02-23 11:32:22,111 DEBUG: Exiting env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:32:22,111 DEBUG: Cleared existing <fiona._env.GDALEnv object at 0x7f848dbc9610> options
2021-02-23 11:32:22,112 DEBUG: Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9610>.
2021-02-23 11:32:22,112 DEBUG: Error handler popped.
2021-02-23 11:32:22,112 DEBUG: Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9610>.
2021-02-23 11:32:22,112 DEBUG: Exiting outermost env
2021-02-23 11:32:22,112 DEBUG: Exited env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:32:22,112 DEBUG: Flushed buffer
2021-02-23 11:32:22,114 DEBUG: Stopped session
Traceback (most recent call last):
File “/code/run.py”, line 69, in
app.execute()
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
[Previous line repeated 6 more times]
File “/code/opendm/types.py”, line 341, in run
self.process(self.args, outputs)
File “/code/stages/odm_orthophoto.py”, line 148, in process
orthophoto.compute_mask_raster(tree.odm_orthophoto_tif, cutline_file,
File “/code/opendm/orthophoto.py”, line 100, in compute_mask_raster
out_image, out_transform = mask(rast, shapes, nodata=0)
File “/usr/local/lib/python3.8/dist-packages/rasterio/mask.py”, line 193, in mask
out_image = dataset.read(
File “rasterio/_io.pyx”, line 369, in rasterio._io.DatasetReaderBase.read
MemoryError: Unable to allocate 91.9 GiB for an array with shape (4, 157096, 157096) and data type bool
Traceback (most recent call last):
File “/code/run.py”, line 69, in
app.execute()
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 341, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 222, in process
system.run(" ".join(map(quote, map(str, argv))), env_vars=os.environ.copy())
File “/code/opendm/system.py”, line 79, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

However, the output of cat /proc/meminfo is:

MemTotal:       392780064 kB
MemFree:        336846548 kB
MemAvailable:   389663444 kB

So the server does have RAM and space. Please help me understand the issue.

Thanks :slight_smile:

1 Like

Welcome! Can you try running that dataset with all default parameters to see if it will process to completion?

Testing, would update regarding the outcome.

1 Like

This is asking for an orthophoto at 2mm resolution (millimeters). I would expect this to fail unless you have a lot of memory (do you really need this resolution?).

1 Like

Hi @pierotofy yes this resolution is needed in this case. As mentioned, the server have 200 GB memory, 128 GB RAM.

You probably need more memory then. The 91.9GB estimate is just for that particular step, memory might be held for other parts of the processing. It’s rasterio that’s failing, which means a lot of copying of data between native code and Python is happening. You probably need at least double that.

1 Like

I’m testing on a new server, 400 GB disk, 384 RAM

2 Likes

Hi @pierotofy and @Saijin_Naib thanks for your time and effoert :slight_smile:

The good news: running the dataset of the original question with a 200 GB server using the default docker params worked, so the problem is narrowed down.

the bad news: The 400 GB server docker throw the following error log:

{
“log”: “\u001b[39m[INFO] running /code/build/bin/odm_cleanmesh -inputFile /datasets/code/submodels/submodel_0000/odm_meshing/odm_25dmesh.dirty.ply -outputFile /datasets/code/submodels/submodel_0000/odm_meshing/odm_25dmesh.ply -removeIslands -decimateMesh 200000 \u001b[0m\n”,
“stream”: “stdout”,
“time”: “2021-03-16T04:40:17.900287099Z”
}{
“log”: “ERROR: In /build/vtk6-dZa2pz/vtk6-6.3.0+dfsg2/IO/PLY/vtkPLYWriter.cxx, line 111\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:18.033581571Z”
}{
“log”: “vtkPLYWriter (0x55ae96091c50): No data to write!\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:18.033623431Z”
}{
“log”: “\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:18.033627361Z”
}{
“log”: “\u001b[39m[INFO] Finished odm_meshing stage\u001b[0m\n”,
“stream”: “stdout”,
“time”: “2021-03-16T04:40:26.587409432Z”
}{
“log”: “\u001b[39m[INFO] Running mvs_texturing stage\u001b[0m\n”,
“stream”: “stdout”,
“time”: “2021-03-16T04:40:26.587449733Z”
}{
“log”: “\u001b[39m[INFO] Writing MVS Textured file in: /datasets/code/submodels/submodel_0000/odm_texturing_25d/odm_textured_model_geo.obj\u001b[0m\n”,
“stream”: “stdout”,
“time”: “2021-03-16T04:40:26.587743215Z”
}{
“log”: “\u001b[39m[INFO] running /code/SuperBuild/install/bin/texrecon /datasets/code/submodels/submodel_0000/opensfm/undistorted/reconstruction.nvm /datasets/code/submodels/submodel_0000/odm_meshing/odm_25dmesh.ply /datasets/code/submodels/submodel_0000/odm_texturing_25d/odm_textured_model_geo -d gmi -o none -t none --no_intermediate_results --nadir_mode \u001b[0m\n”,
“stream”: “stdout”,
“time”: “2021-03-16T04:40:26.587756445Z”
}{
“log”: “/code/SuperBuild/install/bin/texrecon (built on Mar 7 2021, 18:51:56)\n”,
“stream”: “stdout”,
“time”: “2021-03-16T04:40:26.705316856Z”
}{
“log”: “Load and prepare mesh: \n”,
“stream”: “stdout”,
“time”: “2021-03-16T04:40:26.705950712Z”
}{
“log”: “\u0009Could not load mesh: No such file or directory\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.705998683Z”
}{
“log”: “Traceback (most recent call last):\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707334355Z”
}{
“log”: " File “/code/run.py”, line 68, in \u003cmodule\u003e\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707343145Z”
}{
“log”: " app.execute()\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707348115Z”
}{
“log”: " File “/code/stages/odm_app.py”, line 81, in execute\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707351655Z”
}{
“log”: " self.first_stage.run()\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707354515Z”
}{
“log”: " File “/code/opendm/types.py”, line 338, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707357105Z”
}{
“log”: " self.next_stage.run(outputs)\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707359905Z”
}{
“log”: " File “/code/opendm/types.py”, line 338, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707362565Z”
}{
“log”: " self.next_stage.run(outputs)\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707365415Z”
}{
“log”: " File “/code/opendm/types.py”, line 338, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707367875Z”
}{
“log”: " self.next_stage.run(outputs)\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707370475Z”
}{
“log”: " [Previous line repeated 3 more times]\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707372856Z”
}{
“log”: " File “/code/opendm/types.py”, line 319, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707375316Z”
}{
“log”: " self.process(self.args, outputs)\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707377866Z”
}{
“log”: " File “/code/stages/mvstex.py”, line 104, in process\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707380346Z”
}{
“log”: " system.run(’{bin} {nvm_file} {model} {out_dir} '\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707382996Z”
}{
“log”: " File “/code/opendm/system.py”, line 79, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707385556Z”
}{
“log”: " raise Exception(“Child returned {}”.format(retcode))\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707388085Z”
}{
“log”: “Exception: Child returned 1\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:26.707390755Z”
}{
“log”: “Traceback (most recent call last):\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144474873Z”
}{
“log”: " File “/code/run.py”, line 68, in \u003cmodule\u003e\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144485384Z”
}{
“log”: " app.execute()\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144489634Z”
}{
“log”: " File “/code/stages/odm_app.py”, line 81, in execute\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144492584Z”
}{
“log”: " self.first_stage.run()\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144495514Z”
}{
“log”: " File “/code/opendm/types.py”, line 338, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144498144Z”
}{
“log”: " self.next_stage.run(outputs)\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144500784Z”
}{
“log”: " File “/code/opendm/types.py”, line 319, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144503234Z”
}{
“log”: " self.process(self.args, outputs)\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144505834Z”
}{
“log”: " File “/code/stages/splitmerge.py”, line 222, in process\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144508284Z”
}{
“log”: " system.run(" “.join(map(quote, map(str, argv))), env_vars=os.environ.copy())\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144510824Z”
}{
“log”: " File “/code/opendm/system.py”, line 79, in run\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144513494Z”
}{
“log”: " raise Exception(“Child returned {}”.format(retcode))\n",
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144516094Z”
}{
“log”: “Exception: Child returned 1\n”,
“stream”: “stderr”,
“time”: “2021-03-16T04:40:27.144518714Z”
}

What might it indicate?

1 Like

The new server throw this error:

[INFO] running /code/build/bin/odm_cleanmesh -inputFile /datasets/code/submodels/submodel_0000/odm_meshing/odm_25dmesh.dirty.ply -outputFile /datasets/code/submodels/submodel_0000/odm_meshing/odm_25dmesh.ply -removeIslands -decimateMesh 200000
ERROR: In /build/vtk6-dZa2pz/vtk6-6.3.0+dfsg2/IO/PLY/vtkPLYWriter.cxx, line 111
vtkPLYWriter (0x562e5d4ccc50): No data to write!

[INFO] Finished odm_meshing stage
[INFO] Running mvs_texturing stage
[INFO] Writing MVS Textured file in: /datasets/code/submodels/submodel_0000/odm_texturing_25d/odm_textured_model_geo.obj
[INFO] running /code/SuperBuild/install/bin/texrecon /datasets/code/submodels/submodel_0000/opensfm/undistorted/reconstruction.nvm /datasets/code/submodels/submodel_0000/odm_meshing/odm_25dmesh.ply /datasets/code/submodels/submodel_0000/odm_texturing_25d/odm_textured_model_geo -d gmi -o none -t none --no_intermediate_results --nadir_mode
/code/SuperBuild/install/bin/texrecon (built on Mar 7 2021, 18:51:56)
Load and prepare mesh:
Could not load mesh: No such file or directory

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.