Delayed task limit reached

Hi all,
I’m using WebODM to manage my ODM execution. I’ve configured three nodes. Each of them are EC2 instances (c6i.32xlarge).

WebODM is running on the primary.

I have a set of 561 45MP images from my Zenmuse P1, and they take about 5 hours to process on a single node using the “High Quality” preset with 1cm GSD.

I’m trying to split this up with a --split 200 and --split-overlap 80

I keep getting the following in the logs, and it seems to just be stuck in this loop forever:

[WARNING] LRE: submodel_0002 (1528a2e1-77ce-4306-bfe2-80ba9b39abbe) turned from RUNNING to QUEUED. Re-adding to back of the queue.
[WARNING] LRE: submodel_0002 failed with: Delayed task limit reached

Any ideas?

1 Like

Welcome!

Sorry for the trouble.

Is the primary machine of sufficient size that it could proceess the task on its own?

Hi! Yes it is. It takes about 5 hours though. Was hoping to speed it up by clustering.

1 Like

Does this thread provide any further insight to help you tune your instance(s)?

Thanks, Sajin,
This pointed me down the right path.

My AWS security group policy was blocking UDP traffic between the nodes. I fixed that and now all of the nodes are humming along.

Thanks!

3 Likes

Sweet! Glad you’re up and running!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.