WebODM stops reporting status from node

General Help
#1

We are running 3 servers, a WebODM frontend with two processing nodes. The WebODM isn’t running a node.

For some reason, two tasks running on one of the nodes has stopped updating the frontend. The progress hasn’t changed, nor have the detailed logs.

I am closely monitoring the node and the tasks are definitely continuing without issue. The node has plenty of hardware headroom - it has only used up to 50% of it’s 500GB available RAM, and there is plenty of nvme disk space available.

I’ve restarted WebODM twice. I can’t find any errors logged, it’s just very strange. At this point it seems like the node has stopped sending logs/updates to WebODM. I will let the tasks complete and manually collect the assets if I need to.

Since the nodes are running as docker containers, does anyone know how I can check to see why it may have stopped sending updates to WebODM?

#2

You can check a task status by manually accessing:

http://ipofnodeodm:3000/task/:taskid:/info
http://ipofnodeodm:3000/task/:taskid:/output

#3

Thanks @pierotofy, those do work and give correct status. Any idea why it stopped updating, or is there some info I can get you to open a ticket?

–al

#4

Perhaps WebODM ran out of memory at some point and the worker process was killed before it could unlock a task.

You can run this Python script (unlock_all_tasks.py) to make sure all tasks have been unlocked.

#5

Try updating later this afternoon (when docker images have been updated), the new changes in https://github.com/OpenDroneMap/WebODM/pull/673 should take care of this problem.

#6

@pierotofy you rock my friend, I will do that!