HPC Scheduler (e.g. Slurm) integration for ClusterODM

Feedback
1

Hi,

Firstly - fantastic project, well done to all involved.

What I’d be interested in is for ClusterODM to support HPC job schedulers, i.e. Slurm, in the same way it supports cloud services for auto scaling of tasks. I can’t seem to find any reference to this, so I’m adding in as a feature request unless anyone can tell me different!

Currently, I have WebODM Docker container running on a VM, handing off to ClusterODM (running as a Singularity container) running on a HPC gateway node which then passes tasks to NodeODM which I run manually as a Singularity container via Slurm. It would be great if ClusterODM could automatically submit Slurm jobs on the HPC that start the containers for processing creating an auto scaling system.

If anyone has any questions on this or wants clarifications, then just let me know!

Thanks,

Chris

1 Like
Fail To Deploy WebODM on Rahti (PaaS) Kubernetes and OpenShift OKD
2

Hey @ChrisCollins :hand: definitely sounds like a feature request. We would welcome contributions to implement this one.

3

Hello,

Could you please provide some info on how you managed to run the nodeodm container as a singularity container? I had success with the ODM container, but not the nodeodm version, I keep getting the “Invalid configuration file config-default.json.”

Thank you very much in advance,
Yavor :slight_smile:

4

Hi,

I’m by no means a Singularity expert - but the following recipe file is functional for me at least!. Should just need to change the path, user and group in []. In ‘production’ I’ve also included sed commands to change ports and parallel queues.

Let me know how you get on.

Cheers,

Chris

Bootstrap: docker
From: opendronemap/odm:latest

%setup
mkdir -p ${SINGULARITY_ROOTFS}/var/www

%files
[PATH]/NodeODM/* /var/www/

%post
apt update
apt install -y curl telnet git
curl --silent --location https://deb.nodesource.com/setup_10.x | bash -
apt-get install -y nodejs python-gdal && npm install -g nodemon && ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine

    cd /var/www
    npm install
    chown -R [USER:GROUP] /var/www
    chown -R [USER:GROUP] /code

%runscript
/bin/bash <<EOF
cd /var/www
node index.js

1 Like
6

Hi ,
would you please share how to run ODM as singularity container

7

Hi,

I’ve not really looked at this for about a year - lockdown has changed priorities etc but hopefully I’ll get chance to look at this again at some point in the future. Anyway, from what I can remember, this is what I did the following. The reason I used Singularity was to run containers on hosts as unprivileged users. The hope was that we could have a permanent WebODM portal and clusterODM, and then fire up instances of NodeODM on demand (that’s where the request for scheduler support comes in):

WebODM - basically run as the standard Docker container. Was too complicated to convert to singularity, but also was running on a host I was happy to run it as administrator on.

ClusterODM - Run with singularity with a 4GB overlay, i.e.

singularity run --overlay clusterodm.overlay clusterodm-ubuntu.img

Where the container recipe is:

Bootstrap: docker
From: node:lts

%post
apt update
apt install -y curl telnet git
git clone GitHub - OpenDroneMap/ClusterODM: A NodeODM API compatible autoscalable load balancer and task tracker for easy horizontal scaling ♆
cd ClusterODM
npm install
mv config-default.json config-default.json.org
cat config-default.json.org | sed “s/"admin-pass": "",/"admin-pass": "somepasswd123!",/g” > config-default.json
chgrp -R 7100 /ClusterODM

%runscript
/bin/bash <<EOF
cd /ClusterODM
node index.js
EOF

NodeODM - Run with singularity and again with an overlay file. This time I used a 32GB overlay, created on /dev/shm for speed (I’m trying to make use of HPC systems, so we have plenty of RAM available on these nodes!)

Bootstrap: docker
From: opendronemap/odm:latest

%setup
mkdir -p ${SINGULARITY_ROOTFS}/var/www

%files
/home/524070/ODM/NodeODM/* /var/www/

%post
apt update
apt install -y curl telnet git
curl --silent --location https://deb.nodesource.com/setup_10.x | bash -
apt-get install -y nodejs python-gdal && npm install -g nodemon && ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine

cd /var/www
pwd
npm install
chown -R bob:7100 /var/www
chown -R bob:7100 /code

%runscript
/bin/bash <<EOF
cd /var/www
node index.js

The way I was running it, with clusterODM working, I needed to telnet localhost 8080 and run login then node add whatever host was running the NodeODM container on.

Hope this helps somewhat…

3 Likes
8

Hi Chris,

I am very interested by ClusterODM with SLURM. Can you tell me if I can use it with my configuration ? WebODM will be installed on a dedicated server. To use our HPC, we need to ssh to a master server from which we can start SLURM jobs through Singularity. Data needs to be on this master server. My questions are :

  • How Webodm will be able to remotly start a slurm JOB through Singularity ?
  • What about the input DATA ? Could WebODM is able to remotely store data ?

I am not allowed to install webODM on the master server

Thanks

1 Like