Node went offline but ODM still uses resources

WebODM
#1

Hello, While processing images I got the “Processing node went offline. This could be due to insufficient memory or a network error.” and I suspected that I fell into out of memory issue and the task has been halted yet from htop I can see that all cores still run at full speed and 44 / 48 GB of RAM is still used + over 11 GB in SWAP.

Question: Is it possible that if I leave the task for some more hours the underlying ODM will eventually finish or maybe it does not make any sense because WebODM already lost connection and will not be able to get the results?

I have already encountered the same error with the same set of images but that time it happened after 80 hours of runtime instead of current 50 after the restart.

#2

Hello @Etua

Maybe this can help you (I haven’t used it, but I have it referenced). It’s WebODM, but it could work.

Regards

1 Like
#3

Yes, docker ps should show you what is running, you can also use the ids from that command to do a docker logs <id> to see what’s going on. If you are fairly sure things have just punked out, then you can just do a full restart of webodm, and I usually like to pull the latest when I do this as follows:

./webodm.sh down && ./webodm.sh update && ./webodm.sh restart