I’ve been struggling for 6 hours now to start a job to process 950 images. Some of these images are pretty big at nearly 30MB - all JPGs. I was originally triggering the job via WebODM API but when that didn’t happened I used the WebODM interface directly to see what was wrong. The upload seems to get to about 80% each time and dies. I’m on a local network, so speeds are as fast as the disks will go (not exactly blazing with Ceph, but 60MB/s+). I’ve changed the order that the files are uploaded by switching the list order.
Sometimes during upload, the upload resets itself to 0 and starts again.
It’s not clear if this is a timeout issue or something else. I’ve timed it and it’s dying at about 7min 20 seconds, which would be a really weird timeout value. Nothing in the nginx error log, or in syslog. Native server install.
Anybody else seen this issue before?
Error from WebODM:
950 files cannot be uploaded. As a reminder, only images (.jpg, .png) and GCP files (.txt) can be uploaded. Try again.
They’re all JPGs, though some are .jpg and some are .JPG - not entirely sure why, supplied to me that way. I’ve md5sum’d them all and even manually looked through them all - there doesn’t seem to be any corruption. Though if there was, I’d expect it to die at exactly the same point every time.
CPU load is low, RAM is fine on both the server and the client. This server doesn’t do any of the processing itself, it’s just the broker, so it’s hardly used:
root@survey:~# free -m total used free shared buff/cache available Mem: 88509 804 80516 28 7188 87161 Swap: 975 0 975 root@survey:~#