We are running 3 servers, a WebODM frontend with two processing nodes. The WebODM isn’t running a node.
For some reason, two tasks running on one of the nodes has stopped updating the frontend. The progress hasn’t changed, nor have the detailed logs.
I am closely monitoring the node and the tasks are definitely continuing without issue. The node has plenty of hardware headroom - it has only used up to 50% of it’s 500GB available RAM, and there is plenty of nvme disk space available.
I’ve restarted WebODM twice. I can’t find any errors logged, it’s just very strange. At this point it seems like the node has stopped sending logs/updates to WebODM. I will let the tasks complete and manually collect the assets if I need to.
Since the nodes are running as docker containers, does anyone know how I can check to see why it may have stopped sending updates to WebODM?