Lightning processing node hangs on Compress all.zip

Hi Everyone,

I’m processing a project on Lightning and it is hung up on the compress all.zip, which means I’m unable to view the results. How do I get the results so I can view the ortho and model, etc.? I hope below is enough to help understand and resolve the issue.

The project processed 450 images (found here: image set ) with the following settings:
Processing Node: Lightning (manual)
Options: dsm: true, depthmap-resolution: 1000, dem-resolution: 2.0, orthophoto-resolution: 2.0

The task output from the WebODM project can be viewed here: (console(lightning).txt)

The Docker resources:

Couldn’t find a solution from others reporting this issue…

I did find the command :smiley: to get the worker log from docker…
From the command line:
docker logs worker
Postgres is up - executing command
wait-for-it.sh: waiting for broker:6379 without a timeout
wait-for-it.sh: broker:6379 is available after 0 seconds
wait-for-it.sh: waiting for webapp:8000 without a timeout
wait-for-it.sh: webapp:8000 is available after 7 seconds
Checking for celery… OK
Starting worker using broker at redis://broker
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/celery/platforms.py:800: RuntimeWarning: You’re running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
INFO Processing… Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
[2020-10-21 18:55:48,646: INFO/ForkPoolWorker-7] worker.tasks.process_task[374f2ecc-8ce4-48d7-84b1-a7b8977cd05b]: Processing… Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
INFO Processing status: 40 for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
[2020-10-21 19:04:07,991: INFO/ForkPoolWorker-9] worker.tasks.process_task[75aa2efc-ad12-4af2-98c0-1166ff315ef7]: Processing status: 40 for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
INFO Downloading all.zip for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
[2020-10-21 19:04:07,992: INFO/ForkPoolWorker-9] worker.tasks.process_task[75aa2efc-ad12-4af2-98c0-1166ff315ef7]: Downloading all.zip for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
INFO Extracting all.zip for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
[2020-10-21 19:04:12,008: INFO/ForkPoolWorker-9] worker.tasks.process_task[75aa2efc-ad12-4af2-98c0-1166ff315ef7]: Extracting all.zip for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
INFO Extracted all.zip for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
[2020-10-21 19:04:12,553: INFO/ForkPoolWorker-9] worker.tasks.process_task[75aa2efc-ad12-4af2-98c0-1166ff315ef7]: Extracted all.zip for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
INFO Populated extent field with /webodm/app/media/project/5/task/454e954c-d06b-423b-803a-c4cd3ba21d60/assets/odm_orthophoto/odm_orthophoto.tif for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
[2020-10-21 19:04:12,710: INFO/ForkPoolWorker-9] worker.tasks.process_task[75aa2efc-ad12-4af2-98c0-1166ff315ef7]: Populated extent field with /webodm/app/media/project/5/task/454e954c-d06b-423b-803a-c4cd3ba21d60/assets/odm_orthophoto/odm_orthophoto.tif for Task [Task of 2020-10-21T18:55:24.539Z] (454e954c-d06b-423b-803a-c4cd3ba21d60)
Postgres is up - executing command
wait-for-it.sh: waiting for broker:6379 without a timeout
wait-for-it.sh: broker:6379 is available after 0 seconds
wait-for-it.sh: waiting for webapp:8000 without a timeout
wait-for-it.sh: webapp:8000 is available after 5 seconds
Checking for celery… OK
Starting worker using broker at redis://broker
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/celery/platforms.py:800: RuntimeWarning: You’re running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
INFO Processing… Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 16:47:48,998: INFO/ForkPoolWorker-12] worker.tasks.process_task[5c007b4a-631b-4624-bb9f-6af02c0758b6]: Processing… Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Processing status: 30 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 19:02:26,369: INFO/ForkPoolWorker-26] worker.tasks.process_task[f15f1adc-eeb8-45a8-bd93-fd8cbe8cea81]: Processing status: 30 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)

worker: Warm shutdown (MainProcess)
psql: could not connect to server: Connection refused
Is the server running on host “db” (172.19.0.6) and accepting
TCP/IP connections on port 5432?
Postgres is unavailable - sleeping
Postgres is up - executing command
wait-for-it.sh: waiting for broker:6379 without a timeout
wait-for-it.sh: broker:6379 is available after 0 seconds
wait-for-it.sh: waiting for webapp:8000 without a timeout
wait-for-it.sh: webapp:8000 is available after 5 seconds
Checking for celery… OK
Starting worker using broker at redis://broker
INFO Initializing GRASS engine using /usr/bin/grass78
/usr/local/lib/python3.8/dist-packages/celery/platforms.py:800: RuntimeWarning: You’re running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

warnings.warn(RuntimeWarning(ROOT_DISCOURAGED.format(
INFO Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 19:24:47,708: INFO/ForkPoolWorker-3] worker.tasks.process_task[353eedad-2ea5-414b-b223-014ddeb892b8]: Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Processing status: 30 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 21:48:55,696: INFO/ForkPoolWorker-51] worker.tasks.process_task[446dd46a-db92-4b45-bb66-37ea28987759]: Processing status: 30 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 21:53:53,887: INFO/ForkPoolWorker-51] worker.tasks.process_task[b36de34f-9d74-4243-a8c5-b9ca9a865086]: Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Processing status: 40 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 22:08:46,389: INFO/ForkPoolWorker-52] worker.tasks.process_task[5501414f-a049-4d5b-b866-9939440da15a]: Processing status: 40 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Downloading all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 22:08:46,435: INFO/ForkPoolWorker-52] worker.tasks.process_task[5501414f-a049-4d5b-b866-9939440da15a]: Downloading all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Extracting all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 22:08:57,539: INFO/ForkPoolWorker-52] worker.tasks.process_task[5501414f-a049-4d5b-b866-9939440da15a]: Extracting all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Extracted all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 22:08:59,083: INFO/ForkPoolWorker-52] worker.tasks.process_task[5501414f-a049-4d5b-b866-9939440da15a]: Extracted all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Optimizing /webodm/app/media/project/6/task/80b03ec7-78ec-44a2-ab2c-9fab82889513/assets/odm_orthophoto/odm_orthophoto.tif as Cloud Optimized GeoTIFF
[2020-10-25 22:08:59,459: INFO/ForkPoolWorker-52] worker.tasks.process_task[5501414f-a049-4d5b-b866-9939440da15a]: Optimizing /webodm/app/media/project/6/task/80b03ec7-78ec-44a2-ab2c-9fab82889513/assets/odm_orthophoto/odm_orthophoto.tif as Cloud Optimized GeoTIFF
INFO Populated extent field with /webodm/app/media/project/6/task/80b03ec7-78ec-44a2-ab2c-9fab82889513/assets/odm_orthophoto/odm_orthophoto.tif for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 22:09:14,619: INFO/ForkPoolWorker-52] worker.tasks.process_task[5501414f-a049-4d5b-b866-9939440da15a]: Populated extent field with /webodm/app/media/project/6/task/80b03ec7-78ec-44a2-ab2c-9fab82889513/assets/odm_orthophoto/odm_orthophoto.tif for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 22:11:30,974: INFO/ForkPoolWorker-53] worker.tasks.process_task[91da300a-a4e4-46f5-8ea7-8af6a3b537cf]: Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-25 23:21:48,511: ERROR/MainProcess] Timed out waiting for UP message from <ForkProcess(ForkPoolWorker-57, started daemon)>
[2020-10-25 23:21:49,590: ERROR/MainProcess] Process ‘ForkPoolWorker-57’ pid:2842 exited with ‘signal 9 (SIGKILL)’
INFO Canceling Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-26 14:56:53,492: INFO/ForkPoolWorker-129] worker.tasks.process_task[8b7ab9bf-8bba-4b2d-a860-d460ca5d68a4]: Canceling Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-26 14:57:55,893: INFO/ForkPoolWorker-129] worker.tasks.process_task[16460204-5e4d-4c52-b27d-cb9eb6ba24ea]: Restarting Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513) needs to be reprocessed
[2020-10-26 14:57:55,893: INFO/ForkPoolWorker-129] worker.tasks.process_task[16460204-5e4d-4c52-b27d-cb9eb6ba24ea]: Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513) needs to be reprocessed
INFO Processing… Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-26 14:57:59,553: INFO/ForkPoolWorker-130] worker.tasks.process_task[2c70cbde-583f-4734-984c-42ea307678b9]: Processing… Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Processing status: 40 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-26 22:41:01,984: INFO/ForkPoolWorker-279] worker.tasks.process_task[9866803c-8e72-4742-9b46-2a89fc26b691]: Processing status: 40 for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Removing old assets directory: /webodm/app/media/project/6/task/80b03ec7-78ec-44a2-ab2c-9fab82889513/assets/ for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-26 22:41:01,987: INFO/ForkPoolWorker-279] worker.tasks.process_task[9866803c-8e72-4742-9b46-2a89fc26b691]: Removing old assets directory: /webodm/app/media/project/6/task/80b03ec7-78ec-44a2-ab2c-9fab82889513/assets/ for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
INFO Downloading all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)
[2020-10-26 22:41:02,028: INFO/ForkPoolWorker-279] worker.tasks.process_task[9866803c-8e72-4742-9b46-2a89fc26b691]: Downloading all.zip for Task [Grid Flight for 3D] (80b03ec7-78ec-44a2-ab2c-9fab82889513)This text will be hidden`

browser: Chrome
operating system: MacOS
software installed via WebODM Manager

Regards,
CJ

1 Like

Strange…

signal 9 (SIGKILL)

If you download the results from https://webodm.net/dashboard and import them with the “Import” button do you get the same result?

2 Likes

zip file imported and am able to view assets.
thanks,
CJ

1 Like

Anyway to find out what the

signal 9 (SIGKILL)

means?

also…
Should I cancel the project that is hung up?

1 Like

It means the process was terminated (probably due to low memory).

Yes you can cancel it.

2 Likes

It happened to me. Occurred only with an specific project and when using the GCP txt file.
I downloaded from Lightning and import back to WebODM. I think it was a problem with the GCP file.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.