Gunicorn 100% CPU, finished 2D results take a long time to display in browser

ITWarrior · 31 January 2020 21:11

There’s an issue present in WebODM 1.3.0 whereby it takes a long time to retrieve a pre-1.3.0 2D image from the server. I would assume that this has to do with the recent vegetation indexing that was added. After the image has loaded, the results are not cached and appear to be regenerated every time. Here’s an example of a tiny survey (162 images) that takes a long time to load:

http://survey.nwk1.com/public/task/8d813b45-6f10-49a3-b3c2-32ebc74c6e17/map/

At the same time, gunicorn loads up a single core to 100% on the CPU, so it looks like whatever is generating this data is single threaded:

/webodm/python3-venv/bin/python3 /webodm/python3-venv/bin/gunicorn webodm.wsgi --bind unix:/webodm/gunicorn.sock --workers 8 --threads 2 --timeout 600000 --max-requests 250 --preload

The issue seems to be worse for pre-1.3.0 results that new surveys generated with 1.3.0, though I haven’t proven that beyond reasonable doubt.

These are being run and displayed on a cluster of big servers (24 cores, 256GB/512GB RAM, enterprise SSDs on Ceph, 10G network) so I don’t believe that hardware is the limiting factor.

pierotofy · 31 January 2020 23:06

Mm, do the TIFFs for the survey in question have overviews (are they Cloud Optimized GeoTIFFs)? They should have been converted during upgrade, but if the upgrade failed they might have been skipped.

github.com

OpenDroneMap/WebODM/blob/master/app/cogeo.py#L21




def valid_cogeo(src_path):
    """
    Validate a Cloud Optimized GeoTIFF
    :param src_path: path to GeoTIFF
    :return: true if the GeoTIFF is a cogeo, false otherwise
    """
    return cog_validate(src_path, strict=True)




def assure_cogeo(src_path):
    """
    Guarantee that the .tif passed as an argument is a Cloud Optimized GeoTIFF (cogeo)
    If the path is not a cogeo, it is destructively converted into a cogeo.
    If the file cannot be converted, the function does not change the file
    :param src_path: path to GeoTIFF (cogeo or not)
    :return: None
    """


    if not os.path.isfile(src_path):
        logger.warning("Cannot validate cogeo: %s (file does not exist)" % src_path)

ITWarrior · 1 February 2020 00:14

Yes, the working ones have Cloud Optimized GeoTIFFs whilst the non-working ones are just single layer TIFFs.

The upgrade didn’t fail as far as I could tell. We tested the upgrade in dev first and it went fine there.

Is there a way that I can force the conversion? Should I just re-run the upgrade? I’m using the purchased Server Install script.

ITWarrior · 1 February 2020 00:45

This?

/code/modules/odm_georef/src/modifiedPclFunctions.cpp: In function ‘int saveOBJFile(const string&, const pcl::TextureMesh&, unsigned int)’:
/code/modules/odm_georef/src/modifiedPclFunctions.cpp:61:50: warning: conversion to ‘int’ from ‘std::vector<unsigned char>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
   int point_size = tex_mesh.cloud.data.size () / nr_points;
                                                  ^
/code/modules/odm_georef/src/modifiedPclFunctions.cpp:64:47: warning: conversion to ‘int’ from ‘std::vector<std::vector<pcl::Vertices> >::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
   int nr_meshes = tex_mesh.tex_polygons.size ();
                                               ^
/code/modules/odm_georef/src/modifiedPclFunctions.cpp:68:14: warning: conversion to ‘int’ from ‘std::vector<pcl::Vertices>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
     nr_faces += tex_mesh.tex_polygons[m].size ();
              ^
/code/modules/odm_georef/src/modifiedPclFunctions.cpp:183:13: warning: conversion to ‘int’ from ‘std::vector<pcl::Vertices>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
       f_idx += tex_mesh.tex_polygons[m-1].size ();
             ^
In file included from /code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply_parser.h:66:0,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/io/ply_io.h:45,
                 from /code/modules/odm_georef/src/Georef.hpp:11,
                 from /code/modules/odm_georef/src/Georef.cpp:14:
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:86:51: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(int8, int16, "int8", "char");
                                                   ^
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:87:54: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(int16, int16, "int16", "short");
                                                      ^
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:88:52: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(int32, int32, "int32", "int");
                                                    ^
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:89:55: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(uint8, uint16, "uint8", "uchar");
                                                       ^
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:90:58: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(uint16, uint16, "uint16", "ushort");
                                                          ^
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:91:56: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(uint32, uint32, "uint32", "uint");
                                                        ^
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:92:60: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(float32, float32, "float32", "float");
                                                            ^
/code/SuperBuild/install/include/pcl-1.8/pcl/io/ply/ply.h:93:61: warning: extra ‘;’ [-Wpedantic]
       PLY_TYPE_TRAITS(float64, float64, "float64", "double");
                                                             ^
In file included from /code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_base.h:46:0,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_pointcloud.h:42,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_search.h:44,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/search/octree.h:43,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/search/pcl_search.h:45,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/surface/reconstruction.h:45,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/surface/texture_mapping.h:43,
                 from /code/modules/odm_georef/src/modifiedPclFunctions.hpp:9,
                 from /code/modules/odm_georef/src/Georef.hpp:13,
                 from /code/modules/odm_georef/src/Georef.cpp:14:
/code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_key.h:145:9: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
         };
         ^
In file included from /code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_base.h:46:0,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_pointcloud.h:42,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_search.h:44,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/search/octree.h:43,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/search/pcl_search.h:45,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/surface/reconstruction.h:45,
                 from /code/SuperBuild/install/include/pcl-1.8/pcl/surface/texture_mapping.h:43,
                 from /code/modules/odm_georef/src/modifiedPclFunctions.hpp:9,
                 from /code/modules/odm_georef/src/Georef.hpp:13,
                 from /code/modules/odm_georef/src/main.cpp:1:
/code/SuperBuild/install/include/pcl-1.8/pcl/octree/octree_key.h:145:9: warning: ISO C++ prohibits anonymous structs [-Wpedantic]
         };
         ^
/code/modules/odm_georef/src/Georef.cpp: In member function ‘void GeorefCamera::extractCamera(std::ifstream&)’:
/code/modules/odm_georef/src/Georef.cpp:140:15: warning: conversion to ‘Eigen::Transform<float, 3, 2>::Scalar {aka float}’ from ‘double’ may alter its value [-Wfloat-conversion]
     pose(0,2) = -1.0*pose(0,2);
               ^
/code/modules/odm_georef/src/Georef.cpp:141:15: warning: conversion to ‘Eigen::Transform<float, 3, 2>::Scalar {aka float}’ from ‘double’ may alter its value [-Wfloat-conversion]
     pose(1,2) = -1.0*pose(1,2);
               ^
/code/modules/odm_georef/src/Georef.cpp:142:15: warning: conversion to ‘Eigen::Transform<float, 3, 2>::Scalar {aka float}’ from ‘double’ may alter its value [-Wfloat-conversion]
     pose(2,2) = -1.0*pose(2,2);
               ^
/code/modules/odm_georef/src/Georef.cpp:144:15: warning: conversion to ‘Eigen::Transform<float, 3, 2>::Scalar {aka float}’ from ‘double’ may alter its value [-Wfloat-conversion]
     pose(0,1) = -1.0*pose(0,1);
               ^
/code/modules/odm_georef/src/Georef.cpp:145:15: warning: conversion to ‘Eigen::Transform<float, 3, 2>::Scalar {aka float}’ from ‘double’ may alter its value [-Wfloat-conversion]
     pose(1,1) = -1.0*pose(1,1);
               ^
/code/modules/odm_georef/src/Georef.cpp:146:15: warning: conversion to ‘Eigen::Transform<float, 3, 2>::Scalar {aka float}’ from ‘double’ may alter its value [-Wfloat-conversion]
     pose(2,1) = -1.0*pose(2,1);
               ^
/code/modules/odm_georef/src/Georef.cpp: In member function ‘pcl::PointXYZ Georef::barycentricCoordinates(pcl::PointXY, pcl::PointXYZ, pcl::PointXYZ, pcl::PointXYZ, pcl::PointXY, pcl::PointXY, pcl::PointXY)’:
/code/modules/odm_georef/src/Georef.cpp:817:11: warning: conversion to ‘float’ from ‘double’ may alter its value [-Wfloat-conversion]
     res.x = (1.0 - l1 - l2)*vert0.x + l1*vert1.x + l2*vert2.x;
           ^
/code/modules/odm_georef/src/Georef.cpp:818:11: warning: conversion to ‘float’ from ‘double’ may alter its value [-Wfloat-conversion]
     res.y = (1.0 - l1 - l2)*vert0.y + l1*vert1.y + l2*vert2.y;
           ^
/code/modules/odm_georef/src/Georef.cpp:819:11: warning: conversion to ‘float’ from ‘double’ may alter its value [-Wfloat-conversion]
     res.z = (1.0 - l1 - l2)*vert0.z + l1*vert1.z + l2*vert2.z;
           ^
/code/modules/odm_georef/src/Georef.cpp: In member function ‘bool Georef::loadObjFile(std::__cxx11::string, pcl::TextureMesh&)’:
/code/modules/odm_georef/src/Georef.cpp:1506:28: warning: conversion to ‘int’ from ‘std::size_t {aka long unsigned int}’ may alter its value [-Wconversion]
             normal_x_field = i;
                            ^
/code/modules/odm_georef/src/Georef.cpp:1635:23: warning: conversion to ‘int’ from ‘std::size_t {aka long unsigned int}’ may alter its value [-Wconversion]
                     v = (v < 0) ? v_idx + v : v - 1;
                       ^
/code/modules/odm_georef/src/Georef.cpp:1640:41: warning: conversion to ‘std::map<int, int>::key_type {aka int}’ from ‘std::size_t {aka long unsigned int}’ may alter its value [-Wconversion]
                     f2vt[3*(f_idx) + i-1] = vt-1;
                                         ^
/code/modules/odm_georef/src/Georef.cpp:1664:80: warning: conversion to ‘std::map<int, int>::key_type {aka int}’ from ‘std::size_t {aka long unsigned int}’ may alter its value [-Wconversion]
                 Eigen::Vector2f vt = mesh.tex_coordinates[0][f2vt[3*faceIndex+i]];
                                                                                ^
/code/modules/odm_georef/src/Georef.cpp: In member function ‘bool Georef::readHeader(const string&, pcl::PCLPointCloud2&, Eigen::Vector4f&, Eigen::Quaternionf&, int&, int&, unsigned int&, int)’:
/code/modules/odm_georef/src/Georef.cpp:1822:22: warning: conversion to ‘uint32_t {aka unsigned int}’ from ‘std::size_t {aka long unsigned int}’ may alter its value [-Wconversion]
     cloud.width      = nr_point;
                      ^

pierotofy · 1 February 2020 22:42

Those are just warnings.

You could create a script assure_cogeo.py in your /webodm folder with:

from app.cogeo import assure_cogeo
from django.db import migrations
import glob
import os
from webodm import settings

def find_and_assure_cogeo(apps, schema_editor):
    print("=====================================================================")
    print("Migrating TIFFs to Cloud Optimized GeoTIFFs, this might take a while!")
    print("=====================================================================")

    for asset_filename in ["odm_orthophoto.tif", "dsm.tif", "dtm.tif"]:
        for asset in glob.glob(os.path.join(settings.MEDIA_ROOT, "project", "**", asset_filename), recursive=True):
            try:
                print("Optimizing %s" % asset)
                assure_cogeo(asset)
            except Exception as e:
                print("WARNING: cannot check/optimize %s (%s), skipping..." % (asset, str(e)))

find_and_assure_cogeo(None, None)

And then run it:

source python3-venv/bin/activate
python assure_cogeo.py

This is the same code that’s run in https://github.com/OpenDroneMap/WebODM/blob/2a3bbd93b35748cdc3cd9c12cf828497d54f87e6/app/migrations/0030_assure_cogeo.py#L10, which for some reason probably failed during migration.