HPC Scheduler (e.g. Slurm) integration for ClusterODM

Hi,

Firstly - fantastic project, well done to all involved.

What I’d be interested in is for ClusterODM to support HPC job schedulers, i.e. Slurm, in the same way it supports cloud services for auto scaling of tasks. I can’t seem to find any reference to this, so I’m adding in as a feature request unless anyone can tell me different!

Currently, I have WebODM Docker container running on a VM, handing off to ClusterODM (running as a Singularity container) running on a HPC gateway node which then passes tasks to NodeODM which I run manually as a Singularity container via Slurm. It would be great if ClusterODM could automatically submit Slurm jobs on the HPC that start the containers for processing creating an auto scaling system.

If anyone has any questions on this or wants clarifications, then just let me know!

Thanks,

Chris

1 Like

Hey @ChrisCollins :hand: definitely sounds like a feature request. We would welcome contributions to implement this one.

Hello,

Could you please provide some info on how you managed to run the nodeodm container as a singularity container? I had success with the ODM container, but not the nodeodm version, I keep getting the “Invalid configuration file config-default.json.”

Thank you very much in advance,
Yavor :slight_smile:

Hi,

I’m by no means a Singularity expert - but the following recipe file is functional for me at least!. Should just need to change the path, user and group in []. In ‘production’ I’ve also included sed commands to change ports and parallel queues.

Let me know how you get on.

Cheers,

Chris

Bootstrap: docker
From: opendronemap/odm:latest

%setup
mkdir -p ${SINGULARITY_ROOTFS}/var/www

%files
[PATH]/NodeODM/* /var/www/

%post
apt update
apt install -y curl telnet git
curl --silent --location https://deb.nodesource.com/setup_10.x | bash -
apt-get install -y nodejs python-gdal && npm install -g nodemon && ln -s /code/SuperBuild/install/bin/entwine /usr/bin/entwine

    cd /var/www
    npm install
    chown -R [USER:GROUP] /var/www
    chown -R [USER:GROUP] /code

%runscript
/bin/bash <<EOF
cd /var/www
node index.js