Point cloud generation noisy


#21

…unfortunately the new run with --matcher-neighbours does the same thing as a standard exif-included process.

quick comment -smvs creates around 6mill points for this dataset, photoscan creates 29 mill (in High quality mode - medium quality makes around 9 million). opensfm + smvs also really fails to capture detail around vegetation (I can see much more detail in the photoscan product even at medium quality…)

I might try some other SfM matchers prior to smvs (ie cobble it together the old way) or opensfm by itself with all the options exposed. I’m sure there’s a way to clean it all up!


#22

If you want more points, increase --depthmap-resolution (up to the size of your input images).


#23

I’ve done a little testing with the other SfM approaches and they either don’t converge with a typical drone dataset, don’t scale, or don’t improve the point cloud or all three. A third party shared some comparisons of OpenSfM’s camera positions (I don’t think they want their name tied to this) and found it to be quite good, so my hunch is that better camera positioning and orientation doesn’t lead anywhere – we’ve got a pretty good solution there already.

In addition, OpenSfM does seem the most scalable solution for sure – I don’t know of any other solution that uses the hybrid approach to SfM to such effect: 47k images is my current max size reconstruction with split merge, and I haven’t hit the ceiling yet.

I think your insight with respect to precision is a good one, and rerunning with matcher neighbors confirms this theory. Which leaves the question: how do we improve the precision?


#24

Two more questions: Any luck with running through to ortho so we know the vertical error relative to the horizontal sampling distance? Also, what’s the vertical error of Pix4D for the same dataset?


#25

I’m no reader of c++ these days, but having camera positions as float seems a possible culprit: https://github.com/flanggut/smvs/blob/master/lib/view_selection.cc


#26

…right! that would get messy for reprojecting to, say UTM in the real world and dealing with sub-cm numbers. Would a super quick fix be a project -> translate XY such that numbers are small enough -> smvs -> translate results back to real world?

…or am I brave enough to change that to vec3d and make a PR? no doubt other things will breakl :confused:

embarassingly, I don’t have any means to check accuracy in the real world. gear creep time? (dual frequency GPS and some targets…). ODM’s ortho was 0.05m GSD, photoscan’s was 0.002m…

I’ll try increasing --depthmap-resolution :smiley:


#27

ps - that might be a clue about why removing exif works. operating in local coords, numbers are small enough for floats.


#28

Yup, that’s why I had you try that.

Now we don’t work in UTM but a local translated version already for precisely this reason, but it’s likely not enough for storing camera positions in single precision float.


#29

Change it to vec3d and then let me test the branch. It’d at least get us closer to understanding the origin of the issue.


#30

I just changed it all to long double and recompiled without issue, nothing complicated needed it seems. You might try the same, and directly recompile smvs. I think the following is adequate, but @pierotofy or @dakotabenjamin can correct if I’m off here:

cd ~/ODM/SuperBuild/src/elibs/smvs
sudo make

Of course, long double might be overkill, and double might be enough. I spent long enough on IRIX systems back in the day to think they’re the same… .


#31

Well you’re running docker so you will have to apply the changes to the code and then build the image manually. http://docs.opendronemap.org/building.html#docker-installation-cross-platform should do the trick.


#32

I take it the change is in the master branch? so just git pull and build (or rebuild a docker image…)?


#33

It’s not in master – I just did a (careful) find and replace on float in ./SuperBuild/src/elibs/smvs/lib/view_selection.cc of long double and recompiled. In my initial test looking at DSMs it didn’t make a difference, but you’re looking more closely at the point cloud, so you may observe a change I am not.

Here’s the patch as naively written :smiley: :

/*
/*
 * Copyright (c) 2016, Fabian Langguth
 * TU Darmstadt - Graphics, Capture and Massively Parallel Computing
 * All rights reserved.
 *
 * This software may be modified and distributed under the terms
 * of the BSD 3-Clause license. See the LICENSE.txt file for details.
 */

#include "view_selection.h"

SMVS_NAMESPACE_BEGIN

mve::Scene::ViewList
ViewSelection::get_neighbors_for_view (std::size_t const view) const
{
    if (this->bundle != nullptr)
        return this->bundle_based_selection(view);
    else
        return this->position_based_selection(view);
}

mve::Scene::ViewList
ViewSelection::bundle_based_selection (std::size_t const view) const
{
    mve::Scene::ViewList neighbors;
    mve::View::Ptr main_view = this->views[view];
    if (!main_view->has_image(this->opts.embedding))
        return neighbors;

    math::Matrix4f main_view_wtc;
    math::Matrix3f main_view_iproj;
    main_view->get_camera().fill_world_to_cam(*main_view_wtc);
    main_view->get_camera().fill_inverse_calibration(*main_view_iproj,
        main_view->get_image_proxy(this->opts.embedding)->width,
        main_view->get_image_proxy(this->opts.embedding)->height);

    mve::Bundle::Features const& features = this->bundle->get_features();

    /* Create list of features for main view */
    mve::Bundle::Features main_view_features;
    std::vector<long double> main_view_footprints;
    for (std::size_t f = 0; f < features.size(); ++f)
        if (features[f].contains_view_id(main_view->get_id()))
        {
            main_view_features.push_back(features[f]);
            math::Vec3f pos(features[f].pos);
            main_view_footprints.push_back(main_view_wtc.mult(pos, 1)[2] *
                main_view_iproj[0]);
        }

    /* Collect common features in neighboring views */
    neighbors = this->get_sorted_neighbors(view);
    std::multimap<std::size_t, std::size_t,
        std::greater<std::size_t>> common_features_map;
    for (std::size_t i = 0; i < neighbors.size() && i < 50; ++i)
    {
        mve::View::Ptr v = neighbors[i];
        std::size_t id = v->get_id();
        if (id == view || v == nullptr || v->get_camera().flen == 0.0
            || !v->has_image(this->opts.embedding))
            continue;
        math::Matrix4f neighbor_view_wtc;
        math::Matrix3f neighbor_view_iproj;
        v->get_camera().fill_world_to_cam(*neighbor_view_wtc);
        v->get_camera().fill_inverse_calibration(*neighbor_view_iproj,
            v->get_image_proxy(this->opts.embedding)->width,
            v->get_image_proxy(this->opts.embedding)->height);

        std::size_t num_matches = 0;
        for (std::size_t f = 0; f < main_view_features.size(); ++f)
        {
            if (main_view_features[f].contains_view_id(v->get_id()))
            {
                math::Vec3f pos(main_view_features[f].pos);
                long double neighbor_footprint = neighbor_view_wtc.mult(pos, 1)[2] *
                    neighbor_view_iproj[0];
                if (std::min(neighbor_footprint, main_view_footprints[f]) /
                    std::max(neighbor_footprint, main_view_footprints[f]) > 0.6)
                    num_matches++;
            }
        }
        common_features_map.insert(std::make_pair(num_matches, id));
    }
    neighbors.clear();

    /* Use views with most common features as neighbors */
    for (auto features_for_view : common_features_map)
    {
        if (features_for_view.first > 10)
            neighbors.push_back(this->views[features_for_view.second]);
        if (neighbors.size() >= this->opts.num_neighbors)
            return neighbors;
    }
    return neighbors;
}

mve::Scene::ViewList
ViewSelection::position_based_selection (std::size_t const view) const
{
    mve::Scene::ViewList neighbors;

    mve::View::ConstPtr main_view = this->views[view];
    mve::CameraInfo const& main_cam = main_view->get_camera();
    math::Vec3f main_cam_dir;
    main_cam.fill_viewing_direction(*main_cam_dir);
    math::Vec3f main_cam_up;
    main_cam_up[0] = main_cam.rot[2];
    main_cam_up[1] = main_cam.rot[5];
    main_cam_up[2] = main_cam.rot[8];

    neighbors = this->get_sorted_neighbors(view);
    for (auto n = neighbors.begin(); n != neighbors.end();)
    {
        mve::CameraInfo const& cam = (*n)->get_camera();
        if (cam.flen == 0.0)
            continue;
        math::Vec3f sub_cam_dir;
        cam.fill_viewing_direction(*sub_cam_dir);
        math::Vec3f sub_cam_up;
        sub_cam_up[0] = cam.rot[2];
        sub_cam_up[1] = cam.rot[5];
        sub_cam_up[2] = cam.rot[8];

        if (main_cam_up.dot(sub_cam_up) < 0
            || main_cam_dir.dot(sub_cam_dir) < 0.65)
            n = neighbors.erase(n);
        else
            n++;
    }
    return neighbors;
}

mve::Scene::ViewList
ViewSelection::get_sorted_neighbors(std::size_t const view) const
{
    mve::Scene::ViewList neighbors;
    mve::View::ConstPtr main_view = this->views[view];
    mve::CameraInfo const& main_cam = main_view->get_camera();
    math::Vec3f main_cam_pos;
    main_cam.fill_camera_pos(*main_cam_pos);
    std::map<long double, std::size_t> distances;
    for (std::size_t i = 0; i < views.size(); ++i)
    {
        if (views[i] == nullptr || i == view)
            continue;

        mve::CameraInfo const& cam = views[i]->get_camera();
        if (cam.flen == 0.0)
            continue;
        math::Vec3f pos;
        cam.fill_camera_pos(*pos);
        long double dist = (main_cam_pos - pos).norm();
        distances[dist] = i;
    }
    for (auto d : distances)
        neighbors.push_back(views[d.second]);

    return neighbors;
}

SMVS_NAMESPACE_END

#34

Ugh, so with docker, he’ll need to fork smvs, point ODM to that fork, change that fork with the patch to the floats and then build in docker as opposed to remaking just smvs, correct?


#35

Or just enter the docker environment:

docker run -ti --entrypoint bash -v /path/to/datasets:/datasets opendronemap/opendronemap
cd ./SuperBuild/src/elibs/smvs/lib/
apt update && apt install vim -y
rm view_selection.cc
vi view_selection.cc
=== copy paste patched code, :wq ===
cd /SuperBuild/build
make
cd /code
./run.sh --project-path /datasets datasetname

Might have to apt install cmake and one or two other programs if the shell complains at the make command.


#36

Looks like there are some float matrices in here too… . I’m not sure what to replace Matrix4f with – do I replace it with Matrix4d? Probably more importantly, there are a bunch of Vec3f’s in there. Not sure if they can just be changed to Vec3D or not… .


#37

Ok, updated version dealing with what I suspect is the real meat of the problem:

/*
 * Copyright (c) 2016, Fabian Langguth
 * TU Darmstadt - Graphics, Capture and Massively Parallel Computing
 * All rights reserved.
 *
 * This software may be modified and distributed under the terms
 * of the BSD 3-Clause license. See the LICENSE.txt file for details.
 */

#include "view_selection.h"

SMVS_NAMESPACE_BEGIN

mve::Scene::ViewList
ViewSelection::get_neighbors_for_view (std::size_t const view) const
{
    if (this->bundle != nullptr)
        return this->bundle_based_selection(view);
    else
        return this->position_based_selection(view);
}

mve::Scene::ViewList
ViewSelection::bundle_based_selection (std::size_t const view) const
{
    mve::Scene::ViewList neighbors;
    mve::View::Ptr main_view = this->views[view];
    if (!main_view->has_image(this->opts.embedding))
        return neighbors;

    math::Matrix4f main_view_wtc;
    math::Matrix3f main_view_iproj;
    main_view->get_camera().fill_world_to_cam(*main_view_wtc);
    main_view->get_camera().fill_inverse_calibration(*main_view_iproj,
        main_view->get_image_proxy(this->opts.embedding)->width,
        main_view->get_image_proxy(this->opts.embedding)->height);

    mve::Bundle::Features const& features = this->bundle->get_features();

    /* Create list of features for main view */
    mve::Bundle::Features main_view_features;
    std::vector<long double> main_view_footprints;
    for (std::size_t f = 0; f < features.size(); ++f)
        if (features[f].contains_view_id(main_view->get_id()))
        {
            main_view_features.push_back(features[f]);
            math::Vec3d pos(features[f].pos);
            main_view_footprints.push_back(main_view_wtc.mult(pos, 1)[2] *
                main_view_iproj[0]);
        }

    /* Collect common features in neighboring views */
    neighbors = this->get_sorted_neighbors(view);
    std::multimap<std::size_t, std::size_t,
        std::greater<std::size_t>> common_features_map;
    for (std::size_t i = 0; i < neighbors.size() && i < 50; ++i)
    {
        mve::View::Ptr v = neighbors[i];
        std::size_t id = v->get_id();
        if (id == view || v == nullptr || v->get_camera().flen == 0.0
            || !v->has_image(this->opts.embedding))
            continue;
        math::Matrix4f neighbor_view_wtc;
        math::Matrix3f neighbor_view_iproj;
        v->get_camera().fill_world_to_cam(*neighbor_view_wtc);
        v->get_camera().fill_inverse_calibration(*neighbor_view_iproj,
            v->get_image_proxy(this->opts.embedding)->width,
            v->get_image_proxy(this->opts.embedding)->height);

        std::size_t num_matches = 0;
        for (std::size_t f = 0; f < main_view_features.size(); ++f)
        {
            if (main_view_features[f].contains_view_id(v->get_id()))
            {
                math::Vec3d pos(main_view_features[f].pos);
                long double neighbor_footprint = neighbor_view_wtc.mult(pos, 1)[2] *
                    neighbor_view_iproj[0];
                if (std::min(neighbor_footprint, main_view_footprints[f]) /
                    std::max(neighbor_footprint, main_view_footprints[f]) > 0.6)
                    num_matches++;
            }
        }
        common_features_map.insert(std::make_pair(num_matches, id));
    }
    neighbors.clear();

    /* Use views with most common features as neighbors */
    for (auto features_for_view : common_features_map)
    {
        if (features_for_view.first > 10)
            neighbors.push_back(this->views[features_for_view.second]);
        if (neighbors.size() >= this->opts.num_neighbors)
            return neighbors;
    }
    return neighbors;
}

mve::Scene::ViewList
ViewSelection::position_based_selection (std::size_t const view) const
{
    mve::Scene::ViewList neighbors;

    mve::View::ConstPtr main_view = this->views[view];
    mve::CameraInfo const& main_cam = main_view->get_camera();
    math::Vec3d main_cam_dir;
    main_cam.fill_viewing_direction(*main_cam_dir);
    math::Vec3d main_cam_up;
    main_cam_up[0] = main_cam.rot[2];
    main_cam_up[1] = main_cam.rot[5];
    main_cam_up[2] = main_cam.rot[8];

    neighbors = this->get_sorted_neighbors(view);
    for (auto n = neighbors.begin(); n != neighbors.end();)
    {
        mve::CameraInfo const& cam = (*n)->get_camera();
        if (cam.flen == 0.0)
            continue;
        math::Vec3d sub_cam_dir;
        cam.fill_viewing_direction(*sub_cam_dir);
        math::Vec3d sub_cam_up;
        sub_cam_up[0] = cam.rot[2];
        sub_cam_up[1] = cam.rot[5];
        sub_cam_up[2] = cam.rot[8];

        if (main_cam_up.dot(sub_cam_up) < 0
            || main_cam_dir.dot(sub_cam_dir) < 0.65)
            n = neighbors.erase(n);
        else
            n++;
    }
    return neighbors;
}

mve::Scene::ViewList
ViewSelection::get_sorted_neighbors(std::size_t const view) const
{
    mve::Scene::ViewList neighbors;
    mve::View::ConstPtr main_view = this->views[view];
    mve::CameraInfo const& main_cam = main_view->get_camera();
    math::Vec3d main_cam_pos;
    main_cam.fill_camera_pos(*main_cam_pos);
    std::map<long double, std::size_t> distances;
    for (std::size_t i = 0; i < views.size(); ++i)
    {
        if (views[i] == nullptr || i == view)
            continue;

        mve::CameraInfo const& cam = views[i]->get_camera();
        if (cam.flen == 0.0)
            continue;
        math::Vec3d pos;
        cam.fill_camera_pos(*pos);
        long double dist = (main_cam_pos - pos).norm();
        distances[dist] = i;
    }
    for (auto d : distances)
        neighbors.push_back(views[d.second]);

    return neighbors;
}

SMVS_NAMESPACE_END


#38

Yup, now I broke it since mve does everything in floats:

g++ -Wall -Wextra -Wundef -pedantic -msse2 -msse3 -msse4 -mpopcnt -funsafe-math-optimizations -fno-math-errno -std=c++11 -g -O3 -pthread -I../../mve/libs -I../lib -msse4.2 -pthread  -c -MM correspondence.cc global_lighting.cc sgm_stereo.cc surface.cc sse_vector.cc delaunay_2d.cc gauss_newton_step.cc stereo_view.cc surface_patch.cc view_selection.cc depth_triangulator.cc light_optimizer.cc depth_optimizer.cc bicubic_patch.cc mesh_generator.cc surface_derivative.cc mesh_simplifier.cc >Makefile.dep
g++ -Wall -Wextra -Wundef -pedantic -msse2 -msse3 -msse4 -mpopcnt -funsafe-math-optimizations -fno-math-errno -std=c++11 -g -O3 -pthread -I../../mve/libs -I../lib -msse4.2 -pthread  -c -o view_selection.o view_selection.cc
view_selection.cc: In member function ‘mve::Scene::ViewList smvs::ViewSelection::position_based_selection(std::size_t) const’:
view_selection.cc:106:50: error: no matching function for call to ‘mve::CameraInfo::fill_viewing_direction(double*) const’
     main_cam.fill_viewing_direction(*main_cam_dir);
                                                  ^
In file included from ../../mve/libs/mve/view.h:52:0,
                 from ../../mve/libs/mve/scene.h:18,
                 from view_selection.h:13,
                 from view_selection.cc:10:
../../mve/libs/mve/camera.h:49:10: note: candidate: void mve::CameraInfo::fill_viewing_direction(float*) const
     void fill_viewing_direction (float* viewdir) const;
          ^
../../mve/libs/mve/camera.h:49:10: note:   no known conversion for argument 1 from ‘double*’ to ‘float*’
view_selection.cc:119:48: error: no matching function for call to ‘mve::CameraInfo::fill_viewing_direction(double*) const’
         cam.fill_viewing_direction(*sub_cam_dir);
                                                ^
In file included from ../../mve/libs/mve/view.h:52:0,
                 from ../../mve/libs/mve/scene.h:18,
                 from view_selection.h:13,
                 from view_selection.cc:10:
../../mve/libs/mve/camera.h:49:10: note: candidate: void mve::CameraInfo::fill_viewing_direction(float*) const
     void fill_viewing_direction (float* viewdir) const;
          ^
../../mve/libs/mve/camera.h:49:10: note:   no known conversion for argument 1 from ‘double*’ to ‘float*’
view_selection.cc: In member function ‘mve::Scene::ViewList smvs::ViewSelection::get_sorted_neighbors(std::size_t) const’:
view_selection.cc:141:43: error: no matching function for call to ‘mve::CameraInfo::fill_camera_pos(double*) const’
     main_cam.fill_camera_pos(*main_cam_pos);
                                           ^
In file included from ../../mve/libs/mve/view.h:52:0,
                 from ../../mve/libs/mve/scene.h:18,
                 from view_selection.h:13,
                 from view_selection.cc:10:
../../mve/libs/mve/camera.h:36:10: note: candidate: void mve::CameraInfo::fill_camera_pos(float*) const
     void fill_camera_pos (float* pos) const;
          ^
../../mve/libs/mve/camera.h:36:10: note:   no known conversion for argument 1 from ‘double*’ to ‘float*’
view_selection.cc:152:33: error: no matching function for call to ‘mve::CameraInfo::fill_camera_pos(double*) const’
         cam.fill_camera_pos(*pos);
                                 ^
In file included from ../../mve/libs/mve/view.h:52:0,
                 from ../../mve/libs/mve/scene.h:18,
                 from view_selection.h:13,
                 from view_selection.cc:10:
../../mve/libs/mve/camera.h:36:10: note: candidate: void mve::CameraInfo::fill_camera_pos(float*) const
     void fill_camera_pos (float* pos) const;
          ^
../../mve/libs/mve/camera.h:36:10: note:   no known conversion for argument 1 from ‘double*’ to ‘float*’
../../mve/Makefile.inc:29: recipe for target 'view_selection.o' failed
make[1]: *** [view_selection.o] Error 1
make[1]: Leaving directory '/home/useruser/ODM/SuperBuild/src/elibs/smvs/lib'
Makefile:3: recipe for target 'all' failed
make: *** [all] Error 2

#39

Can’t we just ask the compiler to use double and change no code? cc @pierotofy @adamsteer


#40

A very bad idea could be to add this at the top of the first header:

typedef float double; 

:laughing: I’ve never tried it, but it’s valid C++ code.