MemoryError: Unable to allocate 91.9 GiB for an array with shape

Hi all

I’m using an aws ec2 with the following details:

os: linux 2
server type: r5ad.4xlarge
volume size: 200

total size of input images: 1.2 GiB, total 101 images

docker command (using python subprocess.run):

docker run -it --rm -v /home/ec2-user/proj:/datasets/code opendronemap/odm
–project-path /datasets --split-overlap 40 --dsm
–pc-quality lowest --fast-orthophoto --use-hybrid-bundle-adjustment
–split 100 --orthophoto-compression LZW --texturing-outlier-removal-type none
–ignore-gsd --skip-3dmodel --orthophoto-resolution 0.2 --dem-resolution 10 --dtm
–pc-rectify --orthophoto-png

I’m getting this error:

Cleaning up temporary files…
[INFO] Generated cutline file: /datasets/code/submodels/submodel_0000/odm_orthophoto/grass_cutline_tmpdir/cutline.gpkg → /datasets/code/submodels/submodel_0000/odm_orthophoto/cutline.gpkg
[INFO] Computing mask raster: /datasets/code/submodels/submodel_0000/odm_orthophoto/odm_orthophoto_cut.tif
2021-02-23 11:08:30,407 DEBUG: Entering env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,407 DEBUG: Starting outermost env
2021-02-23 11:08:30,407 DEBUG: No GDAL environment exists
2021-02-23 11:08:30,408 DEBUG: New GDAL environment <fiona._env.GDALEnv object at 0x7f848dbc9a00> created
2021-02-23 11:08:30,408 DEBUG: Logging error handler pushed.
2021-02-23 11:08:30,408 DEBUG: All drivers registered.
2021-02-23 11:08:30,408 DEBUG: GDAL_DATA found in environment: ‘/usr/local/lib/python3.8/dist-packages/rasterio/gdal_data’.
2021-02-23 11:08:30,409 DEBUG: PROJ data files are available at built-in paths
2021-02-23 11:08:30,409 DEBUG: Started GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9a00>.
2021-02-23 11:08:30,409 DEBUG: Updated existing <fiona._env.GDALEnv object at 0x7f848dbc9a00> with options {}
2021-02-23 11:08:30,409 DEBUG: Entered env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,409 DEBUG: Got a copy of environment <fiona._env.GDALEnv object at 0x7f848dbc9a00> options
2021-02-23 11:08:30,409 DEBUG: Credentialized: {‘CHECK_WITH_INVERT_PROJ’: True, ‘GTIFF_IMPLICIT_JPEG_OVR’: False, ‘FIONA_ENV’: True, ‘GDAL_DATA’: ‘/usr/local/lib/python3.8/dist-packages/rasterio/gdal_data’}
2021-02-23 11:08:30,428 DEBUG: Exiting env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,428 DEBUG: Cleared existing <fiona._env.GDALEnv object at 0x7f848dbc9a00> options
2021-02-23 11:08:30,428 DEBUG: Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9a00>.
2021-02-23 11:08:30,428 DEBUG: Error handler popped.
2021-02-23 11:08:30,428 DEBUG: Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9a00>.
2021-02-23 11:08:30,428 DEBUG: Exiting outermost env
2021-02-23 11:08:30,428 DEBUG: Exited env context: <fiona.env.Env object at 0x7f848dbc9700>
2021-02-23 11:08:30,428 DEBUG: Entering env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:08:30,428 DEBUG: Starting outermost env
2021-02-23 11:08:30,428 DEBUG: No GDAL environment exists
2021-02-23 11:08:30,428 DEBUG: New GDAL environment <fiona._env.GDALEnv object at 0x7f848dbc9610> created
2021-02-23 11:08:30,428 DEBUG: Logging error handler pushed.
2021-02-23 11:08:30,429 DEBUG: All drivers registered.
2021-02-23 11:08:30,429 DEBUG: GDAL_DATA found in environment: ‘/usr/local/lib/python3.8/dist-packages/rasterio/gdal_data’.
2021-02-23 11:08:30,429 DEBUG: PROJ data files are available at built-in paths
2021-02-23 11:08:30,429 DEBUG: Started GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9610>.
2021-02-23 11:08:30,429 DEBUG: Updated existing <fiona._env.GDALEnv object at 0x7f848dbc9610> with options {}
2021-02-23 11:08:30,429 DEBUG: Entered env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:08:30,430 DEBUG: OLC_FASTSETNEXTBYINDEX: 0
2021-02-23 11:08:30,431 DEBUG: OLC_FASTFEATURECOUNT: 1
2021-02-23 11:08:30,431 DEBUG: Next index: 0
2021-02-23 11:08:30,431 DEBUG: Next index: 1
2021-02-23 11:08:30,442 DEBUG: Next index: 2
2021-02-23 11:08:30,443 DEBUG: Next index: 3
2021-02-23 11:08:30,443 DEBUG: Next index: 4
2021-02-23 11:08:30,443 DEBUG: Next index: 5
2021-02-23 11:08:30,443 DEBUG: Next index: 6
2021-02-23 11:32:22,111 DEBUG: Exiting env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:32:22,111 DEBUG: Cleared existing <fiona._env.GDALEnv object at 0x7f848dbc9610> options
2021-02-23 11:32:22,112 DEBUG: Stopping GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9610>.
2021-02-23 11:32:22,112 DEBUG: Error handler popped.
2021-02-23 11:32:22,112 DEBUG: Stopped GDALEnv <fiona._env.GDALEnv object at 0x7f848dbc9610>.
2021-02-23 11:32:22,112 DEBUG: Exiting outermost env
2021-02-23 11:32:22,112 DEBUG: Exited env context: <fiona.env.Env object at 0x7f848dbc9b20>
2021-02-23 11:32:22,112 DEBUG: Flushed buffer
2021-02-23 11:32:22,114 DEBUG: Stopped session
Traceback (most recent call last):
File “/code/run.py”, line 69, in
app.execute()
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
[Previous line repeated 6 more times]
File “/code/opendm/types.py”, line 341, in run
self.process(self.args, outputs)
File “/code/stages/odm_orthophoto.py”, line 148, in process
orthophoto.compute_mask_raster(tree.odm_orthophoto_tif, cutline_file,
File “/code/opendm/orthophoto.py”, line 100, in compute_mask_raster
out_image, out_transform = mask(rast, shapes, nodata=0)
File “/usr/local/lib/python3.8/dist-packages/rasterio/mask.py”, line 193, in mask
out_image = dataset.read(
File “rasterio/_io.pyx”, line 369, in rasterio._io.DatasetReaderBase.read
MemoryError: Unable to allocate 91.9 GiB for an array with shape (4, 157096, 157096) and data type bool
Traceback (most recent call last):
File “/code/run.py”, line 69, in
app.execute()
File “/code/stages/odm_app.py”, line 83, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 360, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 341, in run
self.process(self.args, outputs)
File “/code/stages/splitmerge.py”, line 222, in process
system.run(" ".join(map(quote, map(str, argv))), env_vars=os.environ.copy())
File “/code/opendm/system.py”, line 79, in run
raise Exception(“Child returned {}”.format(retcode))
Exception: Child returned 1

However, the output of cat /proc/meminfo is:

MemTotal:       392780064 kB
MemFree:        336846548 kB
MemAvailable:   389663444 kB

So the server does have RAM and space. Please help me understand the issue.

Thanks :slight_smile:

1 Like