This is!
E:\WebODM\resources\app\data\db\base\16384
This directory has over 500GB of files, some current and clearly in use for the current task, but many of the big files date back to well before I started this particular task.
They generally have names like 51926.493 or 51928.2 etc, and are of size 1048576KB, ie ~1GB, and there are 500 of the big files in total.
What are they, and are the ones not associated with the current task required?
Need to know as HD space is again running short with just WebODM and a single large task on this 1TB drive.
These should have been maintained by PgSQL, but might have been unmanaged after crashes.
I don’t know of a safe way to clean these… Reasonable safety would be dropping all the number, number.number and _fsm _vm entries that have not been modified in the past week, but still risky. If it goes sideways, you’ll likely need to reinstall and will lose your project list.
The 1TB drive is down to 32GB free space and slowly dropping with the task in progress, so I might have to live dangerously and start removing some of the big numbered files if free space drops too much lower.
Well that is really weird, HD free space down to 19GB, so I deleted 50 of those 1GB files xxxxx.1 to xxxxx.30 from Nov 30th, but there was no increase in space on E:\ according to file explorer! I deleted some permanently and some via Recycler, then removed, but no change in free disk space, now down to 17GB. Closing and opening File Explorer made no difference.
I’ve never deleted 50GB of files and seen no change in free space before!
That is something. Has your system locked up fully requiring forced reboots/shutdowns? The filesystem might be inconsistent, which can potentially lead to files not actually being deleted and space recovered, though this should be very difficult a state to get into with modern Windows and NTFS…
You emptied the Recycle Bin and it didn’t warn you that the Recycle Bin for the E: volume was corrupted?
Still up and running*, no warnings re corruption or anything else.
I deleted another 25GB of files form Dec 1st, still no change in E:\ free space, but \16384 showed a reduction in size.
very weird.
*I thought I’d try diagnostics to see what it thought free space was, and it also showed no reduction, but doing back to dashboard, not so good -
Could not load projects list: error
Now down to 2GB free space, so it’s going to fail very soon, even though I cant see what is going on any more, so another 30 hours of processing down the drain with no result
I guess I’ll have to delete everything and start from scratch, is there a link to the latest version somewhere?
EDIT, I just exited WebODM, and suddenly I have 161GB free.
Can’t start WebODM now, so need to reformat the disc and reinstall I think.
And another 30GB free space has appeared!
“Booting WebODM 1.9.11 build 45…”
“Trying to connect to NodeODM localhost:29301…”
“nodeodm: info: Authentication using SimpleToken\r\n”
“nodeodm: info: Listening on 0.0.0.0:6367 UDP for progress updates\r\n”
“nodeodm: info: Initialized 1 tasks\r\n”
“nodeodm: info: Checking for orphaned directories to be removed…\r\n”
“nodeodm: info: About to run: E:\WebODM\resources\app\apps\ODM\run.bat --min-num-features 15000 --feature-type orb --matcher-neighbors 20 --matcher-distance 60 --skip-3dmodel --mesh-size 300000 --mesh-octree-depth 12 --fast-orthophoto --auto-boundary --dtm --gps-accuracy 5 --optimize-disk-space --pc-rectify --pc-ept --cog --project-path E:\WebODM\resources\app\apps\NodeODM\data 19f822a7-0e20-47ea-bbcb-e76a45dec7e8\r\n”
“nodeodm: info: Server has started on port 29301\r\n”
“Connected to NodeODM 2.2.0”
“redis-server: . \n .-__ ''-._ \n _.-. . ‘’-._ Redis 3.2.100 (00000000/0) 64 bit\n .- .-```. ```\\/ _.,_ ''-._ \n ( ' , .-` | `, ) Running in standalone mode\n |`-._`-...-` __...-.-.|'_.-'| Port: 29379\n | -.._ / _.-' | PID: 12296\n -._ -._ -./ .-’ .-’ \n |-._-.-.__.-' _.-'_.-'| \n | -.-._ _.-'_.-' | http://redis.io \n -._ -._-..-'.-’ .-’ \n |-._-.-.__.-' _.-'_.-'| \n | -.-._ _.-'_.-' | \n -._ -._-..-‘_.-’ _.-’ \n -._ -..-’ _.-’ \n -._ _.-' \n -..-’ \n\n[12296] 05 Dec 14:36:46.239 # Server started, Redis version 3.2.100\n”
“redis-server: [12296] 05 Dec 14:36:46.318 * DB loaded from disk: 0.079 seconds\n[12296] 05 Dec 14:36:46.318 * The server is now ready to accept connections on port 29379\n”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/pg_ctl: server starting\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 0 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 1 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 2 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 3 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 4 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 5 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 6 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 7 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 8 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“Attempt 9 failed, retrying…”
“Trying to connect to postgres localhost:29543…”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: psql: FATAL: the database system is starting up\r\n”
“E:\WebODM\resources\app\apps\pgsql/bin/psql: exited with code 2”
{}
Done…
In other weirdness, Samsung Magician (the SSD management software) reckons there is 461GB free!
File Manager now says 258 GB free
And now WebODM will start, and it appears to be doing something, 100%CPU and 88% RAM in use.
27 hours after the last console log entry it is still apparently churning away, producing more and more of the 1GB files in E:\WebODM\resources\app\data\db\base\16384.
239GB, (160GB compressed) of them since the console updated! Oddly 96GB of those monster files dated Nov 30th are there now too, even though I deleted all the big files from that date yesterday.
I’ve deleted all the resized JPG files (47GB) from E:\WebODM\resources\app\data\media\project\2\task\aa8cc016-5dac-4e0a-9c4e-3ccb8a81733f\ to make more disk space after free space dropped to only 1GB, but am again down to only 12GB free space.
It is getting a bit ridiculous!
so I have not aborted the process yet, living in hope that a result will emerge from fast ortho + DTM, but sooner or later I’ll reformat the disk and re-install, and hope it doesn’t fill up with all these 1GB files again.
Lots more of those 1GB files were being created yesterday evening, and despite removing another over 300GB of them, it apparently didn’t immediately free up any disk space, which is very weird.
Console log also still not updated for 36 hours, but it seems disk space ran out sometime just after 5am, although there was 25GB free when I checked at 4am, when I removed another batch of the 1GB files. Processor and memory usage dropped right off then, so no result again after another 3 days of processing.
I did discover 9061 undistorted images, the last one created at 5:02am, indicating something was still happening, despite no console log updating.
Also there’s a 200GB file, dsm_points.ply in E:\WebODM\resources\app\apps\NodeODM\data\19f822a7-0e20-47ea-bbcb-e76a45dec7e8\odm_meshing\tmp
which uses up a rather large proportion of the 1TB SSD.
I’m 68 hours into attempting it again, with pc-sample 0.05, with no console updates for 28 hours since ‘Undistorting’ finished and this process started
Once again lots of the 1GB xxxxx.xxx files are being produced, the 133rd is in progress right now, so I hope the 285GB of free space remaining is enough!
81 hours into this attempt and these 1GB files continue to increase, now up to 163 of them, and down to 223GB of free space.
Is this normal? The console is not updating, so how do I know python hasn’t gotten itself into some sort of loop, uncontrollably producing these files?
Here is PerfMon commit and CPU % usage for the past 3 and a bit days this has been going.
It was part way into the 171st 1GB file, but sadly it ended with another failure, at 83 hours
Console log ending with this -
[INFO] Completed mesh_dsm.tif in 0:00:31.290585
[INFO] Sampling points from DSM: E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\tmp\mesh_dsm.tif
[INFO] running “E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\dem2points” -inputFile “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\tmp\mesh_dsm.tif” -outputFile “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\tmp\dsm_points.ply” -skirtHeightThreshold 1.5 -skirtIncrements 0.2 -skirtHeightCap 100
[INFO] running “E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\PoissonRecon” --in “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\tmp\dsm_points.ply” --out “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\odm_25dmesh.dirty.ply” --depth 12 --pointWeight 4 --samplesPerNode 1.0 --threads 15 --bType 2 --linearFit
[WARNING] D:\a\ODM\ODM\SuperBuild\src\PoissonRecon\Src\FEMTree.IsoSurface.specialized.inl (Line 1896)
IsoSurfaceExtractor<3,float,struct VectorTypeUnion<float,struct Point<float,3>,struct EmptyVectorType > >::Extract
bad average roots: 300
[INFO] running “E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\OpenMVS\ReconstructMesh” -i “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\odm_25dmesh.dirty.ply” -o “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\odm_25dmesh.ply” --remove-spikes 0 --remove-spurious 20 --smooth 0 --target-face-num 600000 -v 0
[INFO] Finished odm_meshing stage
[INFO] Running mvs_texturing stage
[INFO] Writing MVS Textured file in: E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_texturing_25d\odm_textured_model_geo.obj
[INFO] running “E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\texrecon” “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\opensfm\undistorted/reconstruction.nvm” “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_meshing\odm_25dmesh.ply” “E:\WebODM\resources\app\apps\NodeODM\data\491f05c3-2f58-416d-a8b7-2ebc2976a750\odm_texturing_25d\odm_textured_model_geo” -d gmi -o gauss_clamping -t none --no_intermediate_results --nadir_mode --max_texture_size=8192
E:\WebODM\resources\app\apps\ODM\SuperBuild\install\bin\texrecon (built on Nov 24 2021, 19:27:40)
Load and prepare mesh:
Reading PLY: 256475 verts… 498354 faces… done.
Generating texture views:
No proper input scene descriptor given.
A input descriptor can be:
BUNDLE_FILE - a bundle file (currently onle .nvm files are supported)
SCENE_FOLDER - a folder containing images and .cam files
MVE_SCENE::EMBEDDING - a mve scene and embedding
===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
File “E:\WebODM\resources\app\apps\ODM\stages\odm_app.py”, line 94, in execute
self.first_stage.run()
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 346, in run
self.next_stage.run(outputs)
[Previous line repeated 3 more times]
File “E:\WebODM\resources\app\apps\ODM\opendm\types.py”, line 327, in run
self.process(self.args, outputs)
File “E:\WebODM\resources\app\apps\ODM\stages\mvstex.py”, line 117, in process
system.run('“{bin}” “{nvm_file}” “{model}” “{out_dir}” ’
File “E:\WebODM\resources\app\apps\ODM\opendm\system.py”, line 106, in run
raise SubprocessException(“Child returned {}”.format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1
===== Done, human-readable information to follow… =====
[ERROR] Uh oh! Processing stopped because of strange values in the reconstruction. etc
I’m not sure I’ll ever get this dataset processed.
