WebApp redis connection error 104 while writing to socket. Connection reset by peer

Hi guys,
I’ve got a job running (3265 photos and 23 GCPs) and although it still seems to be processing, the WebODM CLI log started showing a lot of errors like the example below.
Can anyone tell me what this error means?

Example error message:
broker | 1:M 23 Mar 2022 02:10:19.358 # Client id=1811 addr=172.18.0.5:43118 laddr=172.18.0.2:6379 fd=19 name= age=62 idle=2 flags=P db=0 sub=107875 psub=0 multi=-1 qbuf=0 qbuf-free=0 argv-mem=0 obl=0 oll=435 omem=8919240 tot-mem=8939744 events=rw cmd=unsubscribe user=default redir=-1 closed for overcoming of output buffer limits.
webapp | Exception ignored in: <function AsyncResult.del at 0x7f3a98df3af0>
webapp | Traceback (most recent call last):
webapp | File “/usr/local/lib/python3.9/dist-packages/celery/result.py”, line 389, in del
webapp | self.backend.remove_pending_result(self)
webapp | File “/usr/local/lib/python3.9/dist-packages/celery/backends/asynchronous.py”, line 180, in remove_pending_result
webapp | self.on_result_fulfilled(result)
webapp | File “/usr/local/lib/python3.9/dist-packages/celery/backends/asynchronous.py”, line 188, in on_result_fulfilled
webapp | self.result_consumer.cancel_for(result.id)
webapp | File “/usr/local/lib/python3.9/dist-packages/celery/backends/redis.py”, line 148, in cancel_for
webapp | self._pubsub.unsubscribe(key)
webapp | File “/usr/local/lib/python3.9/dist-packages/redis/client.py”, line 3112, in unsubscribe
webapp | retval = self.execute_command(‘UNSUBSCRIBE’, *args)
webapp | File “/usr/local/lib/python3.9/dist-packages/redis/client.py”, line 3009, in execute_command
webapp | self._execute(connection, connection.send_command, *args)
webapp | File “/usr/local/lib/python3.9/dist-packages/redis/client.py”, line 3013, in _execute
webapp | return command(*args)
webapp | File “/usr/local/lib/python3.9/dist-packages/redis/connection.py”, line 620, in send_command
webapp | self.send_packed_command(self.pack_command(*args))
webapp | File “/usr/local/lib/python3.9/dist-packages/redis/connection.py”, line 612, in send_packed_command
webapp | raise ConnectionError(“Error %s while writing to socket. %s.” %
webapp | redis.exceptions.ConnectionError: Error 104 while writing to socket. Connection reset by peer.

Occasionally there will be one of these messages:
broker | 1:M 23 Mar 2022 02:07:15.032 * 100 changes in 300 seconds. Saving…
broker | 1:M 23 Mar 2022 02:07:15.035 * Background saving started by pid 1448
broker | 1448:C 23 Mar 2022 02:07:15.254 * DB saved on disk
broker | 1448:C 23 Mar 2022 02:07:15.257 * RDB: 1 MB of memory used by copy-on-write
broker | 1:M 23 Mar 2022 02:07:15.337 * Background saving terminated with success

1 Like

Do you have any CPU threads not processing? You might be getting timeouts…

Also, how is your free RAM/SWAP?


image

[email protected]:~$ free -h
total used free shared buff/cache available
Mem: 62Gi 36Gi 864Mi 98Mi 24Gi 25Gi
Swap: 127Gi 1.9Gi 126Gi

The machine doesn’t seem to be being pushed hard while these errors are happening.
It’s only using 500GB on its 2TB SSD.
The NodeODM logs say it’s currently doing “undistorting image” tasks.

1 Like

It’s using NodeODM:GPU and these settings:
auto-boundary: true, dsm: true, feature-quality: ultra, matcher-neighbors: 40, mesh-octree-depth: 12, mesh-size: 300000, min-num-features: 64000, orthophoto-resolution: 1, pc-geometric: true, pc-quality: ultra, rerun-from: dataset, resize-to: -1, split: 1000

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.