Node went offline WebODM lightning

Hi,

Good afternoon, I was running a process in WebODM lightning. The process was stopped and I see the following error in the UI " Processing node went offline. This could be due to insufficient memory or a network error.". Please help to fix this problema.

image

Regards

1.Nro images 795
2. Did you fine tune any parameters? Which ones? depthmap-resolution: 1000, dtm: true, dsm: true, dem-resolution: 2.0, orthophoto-resolution: 2.0

Last visua log

Estimated depth-maps 288 (36.23%, 21m28s, ETA 37m)...
Estimated depth-maps 289 (36.35%, 21m33s, ETA 37m)...
Estimated depth-maps 290 (36.48%, 21m39s, ETA 37m)...
Estimated depth-maps 291 (36.60%, 21m43s, ETA 37m)...
Estimated depth-maps 292 (36.73%, 21m47s, ETA 37m)...
Estimated depth-maps 293 (36.86%, 21m52s, ETA 37m)...
1 Like

Hmm…

Could you drop --depthmap-resolution as it has been superceded by --pc-quality and set it to high?

Are the images at nadir, or do they potentially include sky or other far away features? If so, adding --auto-boundary might help it along.

Hi Saijin,

Today happened something strange, I did nothing with the error showed above and after 3 or 4 hours I have received the email notification with the results of the process attached. The resuls looks fine.

Regards

1 Like

Ah, in that case, the node didn’t go offline, but instead your connection dropped.

Are you using the latest build? Can you attach a screenshot of the version/build info (or text)? Piero has made a number of changes that should prevent the node and WebODM from losing communication.

Hi,

Good morning the versions I have installed are the following

image

image

image

Regards

1 Like

Alright, so we still have some more work to do to stabilize that…

Could you please provide the results of the DSLReports Speed Test so we know more about your internet connection quality?

And if it happens again, if you could upload the program logs, that’d be super helpful.

Hi,

Now that I have a subscription I feel I have more problems.

Now I have received the following notification by email:

Your task [Los Laureles del Maullin - 25-11-2021] could not complete. Here’s a copy of the error message:

Fused depth-maps 483 (37.07%, 20m42s, ETA 35m)…
Fused depth-maps 484 (37.15%, 20m45s, ETA 35m)…
Fused depth-maps 485 (37.22%, 20m45s, ETA 35m)…
Fused depth-maps 486 (37.30%, 20m47s, ETA 34m)…
Fused depth-maps 487 (37.38%, 20m51s, ETA 34m)…
Fused depth-maps 488 (37.45%, 20m54s, ETA 34m)…
Fused depth-maps 489 (37.53%, 20m57s, ETA 34m)…
Fused depth-maps 490 (37.61%, 21m4s, ETA 34m)…
Fused depth-maps 491 (37.68%, 21m4s, ETA 34m)…
Fused depth-maps 492 (37.76%, 21m5s, ETA 34m)…
Fused depth-maps 493 (37.84%, 21m6s, ETA 34m)…
Fused depth-maps 494 (37.91%, 21m6s, ETA 34m)…
Fused depth-maps 495 (37.99%, 21m15s, ETA 34m)…
Fused depth-maps 496 (38.07%, 21m17s, ETA 34m)…
Fused depth-maps 497 (38.14%, 21m21s, ETA 34m)…
Fused depth-maps 498 (38.22%, 21m24s, ETA 34m)…Killed

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 137
Traceback (most recent call last):
File “/code/stages/odm_app.py”, line 94, in execute
self.first_stage.run()
File “/code/opendm/types.py”, line 346, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 346, in run
self.next_stage.run(outputs)
File “/code/opendm/types.py”, line 346, in run
self.next_stage.run(outputs)
[Previous line repeated 1 more time]
File “/code/opendm/types.py”, line 327, in run
self.process(self.args, outputs)
File “/code/stages/openmvs.py”, line 85, in process
system.run(’%s “%s” %s’ % (context.omvs_densify_path,
File “/code/opendm/system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 137

===== Done, human-readable information to follow… =====

[ERROR] Whoops! You ran out of memory! Add more RAM to your computer, if you’re using docker configure it to use more memory, for WSL2 make use of .wslconfig

See below the speed test report results:

image

Thanks for your help

1 Like

What are you processing options for this job?

Hello,

Good morning the options were the following:

dem-resolution: 2.0, dsm: true, dtm: true, orthophoto-resolution: 2.0, pc-quality: high

The other thing I found is that I have a lot used space but I have only one completed Job.

image
image

Thanks for your help.

Regards

Try adding auto-boundary to your jobs just in case.

The space graph is for your entire system, not just the space we use.

You might need to look at other things that may be taking space.

Thanks Saijin, I’ll add that option to my process now.

Didn’t notice that the graph space was related to my HD. :sweat_smile:

Regards

2 Likes

Hello,

Good morning, sent a new Job with 1307 images with the following options :sweat_smile:

auto-boundary: true, dem-resolution: 2.0, dsm: true, dtm: true, orthophoto-resolution: 2.0, pc-quality: high

The process failed finishing with the following error:

Output could not be retrieved: HTTPSConnectionPool(host=‘spark1.webodm.net’, port=443): Read timed out. (read timeout=5)

1 Like

Alright! We had a disconnect, but that doesn’t mean the job failed.

Have you received a Task Failed or Task Completed email yet? If not, wait it out a bit more.

Could you attach the logs from here:
Tools → View Logs

Hi,

Yes I have received the mail with subject Task Failed.

Please find attached the log file.

Log File

Thanks

1 Like

Thanks for that, but that seems to just be the PostgreSQL database server log. There should also be an app log in that folder.

In addition, can you send/attach the task processing log? It is the download button next to the fullscreen button below the Processing Log panel:
Screenshot_2021-11-29_13-03-39

Hello,

Attached here console will you find the other log file.

Regards

1 Like

Unfortunately, the log is truncated at 16675 lines and is looking good/normal thus far. It doesn’t tell me anything problematic.

Mm, this looks like a potential problem on our end… try to set the --pc-tile option if you get this error again, how large are your images?

1 Like

Hi

Thanks for your felp I have added pc-tile and deleted the DTM option, so the options selected now are the following,

auto-boundary: true, dem-resolution: 2.0, dsm: true, orthophoto-resolution: 2.0, pc-quality: high, pc-tile: true

The size of the images is:

image

Regards

1 Like

Hi,

The server went offline again, the following is the log file of the last process I have executed:

I don’t know what happen, with the environment, It’s impossible ti finish a task.

Thanks for your help.

1 Like