WebODM won't start: Postgres is unavailable - sleeping [solved]

[Aeret]

Dear all,

I tried to install a WebODM instance on a Ubuntu 22.04 server, 1TB and 32GB RAM in a Docker. This is my second server (with exact the same specs) I would like to install WebODM. First time was half a year ago and worked like a charm.

However this time I get an error while starting the WebODM script.

I follow the tutorial at Installation and Getting Started — OpenDroneMap 3.5.6 documentation and everything seems to work.

However, when I start WebODM (./webodm.sh start) I get the recurring message:

worker | psql: error: could not connect to server: No route to host
worker | Is the server running on host “db” (172.19.0.4) and accepting
worker | TCP/IP connections on port 5432?
worker | Postgres is unavailable - sleeping
webapp | psql: error: could not connect to server: No route to host
webapp | Is the server running on host “db” (172.19.0.4) and accepting
webapp | TCP/IP connections on port 5432?
webapp | Postgres is unavailable - sleeping

Anybody has a suggestion? Thanks!

See the log

Using the following environment:

Host: localhost
Port: 8000
Media directory: appmedia
Postgres DB directory: dbdata
SSL: NO
SSL key:
SSL certificate:
SSL insecure port redirect: 80
Celery Broker: redis://broker
Default Nodes: 1

Make sure to issue a ./webodm.sh down if you decide to change the environment.

docker-compose -f docker-compose.yml -f docker-compose.nodeodm.yml up --scale no de-odm=1
Creating network “webodm_default” with the default driver
Creating broker … done
Creating webodm_node-odm_1 … done
Creating db … done
Creating worker … done
Creating webapp … done
Attaching to broker, webodm_node-odm_1, db, worker, webapp
broker | 1:C 13 Apr 2023 12:36:18.413 # oO0OoO0OoO0Oo Redis is starting oO0 OoO0OoO0Oo
broker | 1:C 13 Apr 2023 12:36:18.413 # Redis version=7.0.10, bits=64, comm it=00000000, modified=0, pid=1, just started
broker | 1:C 13 Apr 2023 12:36:18.413 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /pa th/to/redis.conf
broker | 1:M 13 Apr 2023 12:36:18.413 * monotonic clock: POSIX clock_gettim e
broker | 1:M 13 Apr 2023 12:36:18.414 * Running mode=standalone, port=6379.
broker | 1:M 13 Apr 2023 12:36:18.414 # Server initialized
broker | 1:M 13 Apr 2023 12:36:18.414 # WARNING Memory overcommit must be e nabled! Without it, a background save or replication may fail under low memory c ondition. Being disabled, it can can also cause failures without low memory cond ition, see vm.max_map_count growing steadily when vm.overcommit_memory is 2 · Issue #1328 · jemalloc/jemalloc · GitHub. To fix this issue a dd ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the com mand ‘sysctl vm.overcommit_memory=1’ for this to take effect.
broker | 1:M 13 Apr 2023 12:36:18.414 * Ready to accept connections
node-odm_1 | (node:1) NOTE: We are formalizing our plans to enter AWS SDK for J avaScript (v2) into maintenance mode in 2023.
node-odm_1 |
node-odm_1 | Please migrate your code to use AWS SDK for JavaScript (v3).
node-odm_1 | For more information, check the migration guide at https://a.co/7P zMCcy
node-odm_1 | (Use node --trace-warnings ... to show where the warning was cre ated)
db |
db | PostgreSQL Database directory appears to contain a database; Skipp ing initialization
db |
node-odm_1 | info: Authentication using NoTokenRequired
db | LOG: database system was shut down at 2023-04-13 12:30:38 UTC
db | LOG: MultiXact member wraparound protections are now enabled
db | LOG: database system is ready to accept connections
node-odm_1 | info: Listening on 0.0.0.0:6367 UDP for progress updates
db | LOG: autovacuum launcher started
node-odm_1 | info: No tasks dump found
node-odm_1 | info: Checking for orphaned directories to be removed…
node-odm_1 | info: Server has started on port 3000
worker | psql: error: could not connect to server: No route to host
worker | Is the server running on host “db” (172.19.0.4) and accepting
worker | TCP/IP connections on port 5432?
worker | Postgres is unavailable - sleeping
webapp | psql: error: could not connect to server: No route to host
webapp | Is the server running on host “db” (172.19.0.4) and accepting
webapp | TCP/IP connections on port 5432?
webapp | Postgres is unavailable - sleeping
worker | psql: error: could not connect to server: No route to host
worker | Is the server running on host “db” (172.19.0.4) and accepting
worker | TCP/IP connections on port 5432?
worker | Postgres is unavailable - sleeping
webapp | psql: error: could not connect to server: No route to host
webapp | Is the server running on host “db” (172.19.0.4) and accepting
webapp | TCP/IP connections on port 5432?
webapp | Postgres is unavailable - sleeping

[Aeret]

I tried the rebuild option (./webodm.sh rebuild) and got the following message:

root@ubuntu:~/WebODM# ./webodm.sh rebuild
Checking for docker… OK
Checking for docker-compose… OK
Rebuilding WebODM…
docker-compose down --remove-orphans
Removing orphan container “webodm_node-odm_1”
Removing webapp … done
Removing worker … done
Removing db … done
Removing broker … done
Removing network webodm_default
rm -fr node_modules/ || sudo rm -fr node_modules/
rm -fr nodeodm/external/NodeODM || sudo rm -fr nodeodm/external/NodeODM
docker-compose -f docker-compose.yml -f docker-compose.build.yml build --no-cache
broker uses an image, skipping
worker uses an image, skipping
Building db
[+] Building 151.5s (11/11) FINISHED
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 4.88kB 0.0s
=> [internal] load metadata for Docker Hub Container Image Library | App Containerization 1.2s
=> [1/7] FROM Docker Hub Container Image Library | App Containerization 1.4s
=> => resolve Docker Hub Container Image Library | App Containerization 0.0s
=> => sha256:06d39c85623a79eb68b6cac39f4c4aba8834873ee5fc27be9d9d7655e6313c26 27.50MB / 27.50MB 0.8s
=> => sha256:24a0df437301598d1a4b62ddf59fa0ed2969150d70d748c84225e6501e9c36b9 1.13kB / 1.13kB 0.0s
=> => sha256:b39db7fc56971aac21dee02187e898db759c4f26b9b27b1d80b6ad32ff330c76 424B / 424B 0.0s
=> => sha256:1c5c8d0b973ab9d6d7742cf050beefa8ba2efb7779e09bfb1200d6192180766e 2.30kB / 2.30kB 0.0s
=> => extracting sha256:06d39c85623a79eb68b6cac39f4c4aba8834873ee5fc27be9d9d7655e6313c26 0.5s
=> [internal] load build context 0.0s
=> => transferring context: 11.71kB 0.0s
=> [2/7] RUN mkdir /docker-entrypoint-initdb.d 0.4s
=> [3/7] COPY init.sql /docker-entrypoint-initdb.d/init-db.sql 0.0s
=> [4/7] RUN chmod 644 /docker-entrypoint-initdb.d/init-db.sql 0.4s
=> [5/7] COPY docker-entrypoint.sh /usr/local/bin/ 0.0s
=> [6/7] RUN ln -s /usr/local/bin/docker-entrypoint.sh / # backwards compat 0.3s
=> ERROR [7/7] RUN apt-get update; set -eux; groupadd -r postgres --gid=999; useradd -r -g postgres --uid=999 --home-dir= 147.9s

[7/7] RUN apt-get update; set -eux; groupadd -r postgres --gid=999; useradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres; mkdir -p /var/lib/postgresql; chown -R postgres:postgres /var/lib/postgresql; apt-get update; apt-get install -y --no-install-recommends wget; rm -rf /var/lib/apt/lists/; dpkgArch=“$(dpkg --print-architecture | awk -F- ‘{ print $NF }’)”; wget --no-check-certificate -O /usr/local/bin/gosu “https://github.com/tianon/gosu/releases/download/1.12/gosu-$dpkgArch”; chmod +x /usr/local/bin/gosu; gosu --version; gosu nobody true; apt-get update; apt-get install -y --no-install-recommends locales; localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8; mkdir /staging; apt-get update; apt-get install -y --no-install-recommends wget gcc build-essential libproj-dev libgeos-dev libxml2-dev zlib1g-dev libreadline-dev; cd /staging; wget --no-check-certificate -q https://github.com/OpenDroneMap/WebODM/releases/download/v1.9.2/postgresql-9.5.25.tar.gz; cd /staging; tar -zxf postgresql-9.5.25.tar.gz; cd postgresql-9.5.25; ./configure; make -j$(nproc); make install; postgres --version; sed -ri “s/#autovacuum_max_workers = 3/autovacuum_max_workers = 6/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_naptime = 1min/autovacuum_naptime = 15s/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_vacuum_threshold = 50/autovacuum_vacuum_threshold = 25/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_vacuum_scale_factor = 0.2/autovacuum_vacuum_scale_factor = 0.1/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_analyze_threshold = 50/autovacuum_analyze_threshold = 10/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_analyze_scale_factor = 0.1/autovacuum_analyze_scale_factor = 0.05/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_analyze_threshold = 50/autovacuum_analyze_threshold = 10/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_vacuum_cost_delay = 20ms/autovacuum_vacuum_cost_delay = 10ms/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri “s/#autovacuum_vacuum_cost_limit = -1/autovacuum_vacuum_cost_limit = 1000/” /usr/local/pgsql/share/postgresql.conf.sample; sed -ri "s!^#?(listen_addresses)\s=\s*\S+.!\1 = ''!" /usr/local/pgsql/share/postgresql.conf.sample; grep -F "listen_addresses = ‘'" /usr/local/pgsql/share/postgresql.conf.sample; mkdir -p /var/run/postgresql && chown -R postgres:postgres /var/run/postgresql && chmod 2777 /var/run/postgresql; mkdir -p “/var/lib/postgresql/data” && chown -R postgres:postgres “/var/lib/postgresql/data” && chmod 777 “/var/lib/postgresql/data”; apt-get update; apt-get install -y --no-install-recommends libgdal-dev libjson-c-dev; cd /staging; wget --no-check-certificate -q https://github.com/OpenDroneMap/WebODM/releases/download/v1.9.2/postgis-2.3.2.tar.gz; wget --no-check-certificate -q -O /usr/include/json-c/json_object_private.h https://raw.githubusercontent.com/json-c/json-c/json-c-0.13/json_object_private.h; tar -zxf postgis-2.3.2.tar.gz; sed -i 's/#error./#define ACCEPT_USE_OF_DEPRECATED_PROJ_API_H 1/’ /usr/include/proj_api.h; cd /staging/postgis-2.3.2; ./configure --with-pgconfig=/usr/local/pgsql/bin/pg_config; make; make install; sed -i ‘1d’ /usr/local/pgsql/share/extension/postgis–2.3.2.sql; apt-get remove -y gcc build-essential wget; apt-get autoremove -y; apt-get clean; rm -fr /var/lib/apt/lists/* /staging /usr/include;:
#0 24.91 Err:1 Index of /ubuntu focal InRelease
#0 24.91 Temporary failure resolving ‘archive.ubuntu.com’
#0 24.91 Err:2 Index of /ubuntu focal-security InRelease
#0 24.91 Temporary failure resolving ‘security.ubuntu.com’
#0 49.49 Err:3 Index of /ubuntu focal-updates InRelease
#0 49.49 Temporary failure resolving ‘archive.ubuntu.com’
#0 74.07 Err:4 Index of /ubuntu focal-backports InRelease
#0 74.07 Temporary failure resolving ‘archive.ubuntu.com’
#0 74.07 Reading package lists…
#0 74.08 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Temporary failure resolving ‘archive.ubuntu.com’
#0 74.08 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Temporary failure resolving ‘archive.ubuntu.com’
#0 74.08 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Temporary failure resolving ‘archive.ubuntu.com’
#0 74.08 W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Temporary failure resolving ‘security.ubuntu.com’
#0 74.08 W: Some index files failed to download. They have been ignored, or old ones used instead.
#0 74.08 + groupadd -r postgres --gid=999
#0 74.08 + useradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres
#0 74.09 + mkdir -p /var/lib/postgresql
#0 74.09 + chown -R postgres:postgres /var/lib/postgresql
#0 74.10 + apt-get update
#0 98.67 Err:1 Index of /ubuntu focal InRelease
#0 98.67 Temporary failure resolving ‘archive.ubuntu.com’
#0 98.67 Err:2 Index of /ubuntu focal-security InRelease
#0 98.67 Temporary failure resolving ‘security.ubuntu.com’
#0 123.3 Err:3 Index of /ubuntu focal-updates InRelease
#0 123.3 Temporary failure resolving ‘archive.ubuntu.com’
#0 147.8 Err:4 Index of /ubuntu focal-backports InRelease
#0 147.8 Temporary failure resolving ‘archive.ubuntu.com’
#0 147.8 Reading package lists…
#0 147.8 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal/InRelease Temporary failure resolving ‘archive.ubuntu.com’
#0 147.8 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease Temporary failure resolving ‘archive.ubuntu.com’
#0 147.8 W: Failed to fetch http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease Temporary failure resolving ‘archive.ubuntu.com’
#0 147.8 W: Failed to fetch http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease Temporary failure resolving ‘security.ubuntu.com’
#0 147.8 W: Some index files failed to download. They have been ignored, or old ones used instead.
#0 147.8 + apt-get install -y --no-install-recommends wget
#0 147.8 Reading package lists…
#0 147.8 Building dependency tree…
#0 147.8 Reading state information…
#0 147.8 E: Unable to locate package wget

---

Dockerfile:24

23 | # Setup system
24 | >>> RUN apt-get update;
25 | >>> set -eux;
26 | >>> groupadd -r postgres --gid=999;
27 | >>> useradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres;
28 | >>> mkdir -p /var/lib/postgresql;
29 | >>> chown -R postgres:postgres /var/lib/postgresql;
30 | >>> # grab gosu for easy step-down from root
31 | >>> # Releases · tianon/gosu · GitHub
32 | >>> apt-get update;
33 | >>> apt-get install -y --no-install-recommends wget;
34 | >>> rm -rf /var/lib/apt/lists/;
35 | >>> dpkgArch=“$(dpkg --print-architecture | awk -F- ‘{ print $NF }’)”;
36 | >>> wget --no-check-certificate -O /usr/local/bin/gosu “https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$dpkgArch”;
37 | >>> chmod +x /usr/local/bin/gosu;
38 | >>> gosu --version;
39 | >>> gosu nobody true;
40 | >>> # make the “en_US.UTF-8” locale so postgres will be utf-8 enabled by default
41 | >>> apt-get update;
42 | >>> apt-get install -y --no-install-recommends locales;
43 | >>> localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8;
44 | >>> # Build Postgres from source
45 | >>> mkdir /staging;
46 | >>> apt-get update;
47 | >>> apt-get install -y --no-install-recommends wget gcc build-essential libproj-dev libgeos-dev libxml2-dev zlib1g-dev libreadline-dev;
48 | >>> cd /staging;
49 | >>> wget --no-check-certificate -q https://github.com/OpenDroneMap/WebODM/releases/download/v1.9.2/postgresql-$PG_VERSION.tar.gz;
50 | >>> cd /staging;
51 | >>> tar -zxf postgresql-$PG_VERSION.tar.gz;
52 | >>> cd postgresql-$PG_VERSION;
53 | >>> ./configure;
54 | >>> make -j$(nproc);
55 | >>> make install;
56 | >>> postgres --version;
57 | >>> sed -ri “s/#autovacuum_max_workers = 3/autovacuum_max_workers = 6/” /usr/local/pgsql/share/postgresql.conf.sample;
58 | >>> sed -ri “s/#autovacuum_naptime = 1min/autovacuum_naptime = 15s/” /usr/local/pgsql/share/postgresql.conf.sample;
59 | >>> sed -ri “s/#autovacuum_vacuum_threshold = 50/autovacuum_vacuum_threshold = 25/” /usr/local/pgsql/share/postgresql.conf.sample;
60 | >>> sed -ri “s/#autovacuum_vacuum_scale_factor = 0.2/autovacuum_vacuum_scale_factor = 0.1/” /usr/local/pgsql/share/postgresql.conf.sample;
61 | >>> sed -ri “s/#autovacuum_analyze_threshold = 50/autovacuum_analyze_threshold = 10/” /usr/local/pgsql/share/postgresql.conf.sample;
62 | >>> sed -ri “s/#autovacuum_analyze_scale_factor = 0.1/autovacuum_analyze_scale_factor = 0.05/” /usr/local/pgsql/share/postgresql.conf.sample;
63 | >>> sed -ri “s/#autovacuum_analyze_threshold = 50/autovacuum_analyze_threshold = 10/” /usr/local/pgsql/share/postgresql.conf.sample;
64 | >>> sed -ri “s/#autovacuum_vacuum_cost_delay = 20ms/autovacuum_vacuum_cost_delay = 10ms/” /usr/local/pgsql/share/postgresql.conf.sample;
65 | >>> sed -ri “s/#autovacuum_vacuum_cost_limit = -1/autovacuum_vacuum_cost_limit = 1000/” /usr/local/pgsql/share/postgresql.conf.sample;
66 | >>> sed -ri "s!^#?(listen_addresses)\s=\s*\S+.!\1 = ''!" /usr/local/pgsql/share/postgresql.conf.sample;
67 | >>> grep -F "listen_addresses = ‘'" /usr/local/pgsql/share/postgresql.conf.sample;
68 | >>> mkdir -p /var/run/postgresql && chown -R postgres:postgres /var/run/postgresql && chmod 2777 /var/run/postgresql;
69 | >>> # this 777 will be replaced by 700 at runtime (allows semi-arbitrary “–user” values)
70 | >>> mkdir -p “$PGDATA” && chown -R postgres:postgres “$PGDATA” && chmod 777 “$PGDATA”;
71 | >>> # Build PostGIS from source
72 | >>> apt-get update;
73 | >>> apt-get install -y --no-install-recommends libgdal-dev libjson-c-dev;
74 | >>> cd /staging;
75 | >>> wget --no-check-certificate -q https://github.com/OpenDroneMap/WebODM/releases/download/v1.9.2/postgis-$POSTGIS_VERSION.tar.gz;
76 | >>> wget --no-check-certificate -q -O /usr/include/json-c/json_object_private.h https://raw.githubusercontent.com/json-c/json-c/json-c-0.13/json_object_private.h;
77 | >>> tar -zxf postgis-$POSTGIS_VERSION.tar.gz;
78 | >>> sed -i 's/#error./#define ACCEPT_USE_OF_DEPRECATED_PROJ_API_H 1/’ /usr/include/proj_api.h;
79 | >>> cd /staging/postgis-$POSTGIS_VERSION;
80 | >>> ./configure --with-pgconfig=/usr/local/pgsql/bin/pg_config;
81 | >>> make;
82 | >>> make install;
83 | >>> sed -i ‘1d’ /usr/local/pgsql/share/extension/postgis–$POSTGIS_VERSION.sql;
84 | >>> apt-get remove -y gcc build-essential wget;
85 | >>> apt-get autoremove -y;
86 | >>> apt-get clean;
87 | >>> rm -fr /var/lib/apt/lists/* /staging /usr/include;

88
ERROR: failed to solve: process "/bin/sh -c apt-get update; set -eux; \tgroupadd -r postgres --gid=999; \tuseradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres; \tmkdir -p /var/lib/postgresql; \tchown -R postgres:postgres /var/lib/postgresql; \tapt-get update; \tapt-get install -y --no-install-recommends wget; \trm -rf /var/lib/apt/lists/*; \tdpkgArch="$(dpkg --print-architecture
ERROR: Service ‘db’ failed to build : Build failed

[Saijin_Naib]

Hmm…

What’s the output of:

id $USER

[Aeret]

Thanks for replying.

You mean the server? I can acces the server as a root user.
As far as Postgres concerned, I don’t know what the (generated) user is when running the install-script. Do you know how I can check that?

Edit:

Do you mean this?

Mark

[Saijin_Naib]

What user is running your webodm.sh commands?

[Aeret]

The root user.

[Saijin_Naib]

Hmm…

Not sure what the behavior is when running the entire thing as root.

I run a standard user in the docker group.

Do you have a standard user you can try with?

[Aeret]

Thanks for all the support Saijin, I will give that a try.

[Aeret]

In the end I reinstalled the server with Debian (instead of Ubuntu) and woohoo, within a couple of minutes WebODM was running.

The strange thing is, I got the same message (postgres is sleeping) during the installation but after (one warning) the installation continued.