Slight global rotation error observed when projecting bboxes in camera coordinate into polygons in world coordinate

Hello everyone,

I am trying to project bboxes defined in camera coordinate onto world coordinate using the parameters stored in /opensfm after orthoimage generation based on ODM with precision gcp information.
An issue is that I observe a slight global rotation of the projection compared with (seemingly) more precise projection based on Metashape (see the following animation).

metashape_opensfm2

I assume the possible reason is that there exists some levels of biases in extracting camera origins and the bearing vectors, but I am not sure.
I’ll highly appreciate if you contribute to finding the reasons and sharing the solutions with me.
The detailed procedure of the projection is as follows,

  1. load camera poses from /Project/opensfm by

data = dataset.DataSet(’/Project/opensfm’)

as per this discussion.

  1. load reconstruction in world coordinate by

rec = data.load_reconstruction()[0]

as per this discussion

  1. load pose and camera parameters, pose and cam, by

pose = rec.shot[filename].pose

and

cam = rec.shots[filename].camera

  1. given a vertex (x,y) of bboxes in the camera (image) coordinate, convert (x,y) into normalized coordinate by

p2 = cam.pixel_to_normalized_coordinates((x,y))

  1. convert the normalized position into bearing vector, bearing, by

bearing = cam.pixel_bearing(p2)

  1. project the bearing vector into a point, p3, in world coordinate by

p3 = pose.inverse().transform(bearing)

  1. extract an origin of the camera coordinate, p1, by

p1 = pose.get_origin()

  1. calculate viewing vector

v = p3-p1

  1. generate a concatenated vector of p1 and v by

concat = np.concatenate([p1, v])

  1. generate viewing rays by

rays = o3d.core.Tensor([concat, concat],dtype=o3d.core.Dtype.Float32)

Note that I need to put two concats because the number of rays should be more than one for the following procedure 12.

  1. load 3D mesh by the following lines

cube = o3d.io.read_triangle_mesh(’/Project/odm_msshing/odm_mesh.ply’)

scene = o3d.t.geometry.RaycastingScene()

t_cube = o3d.t.geometry.TriangleMesh.from_legacy(cube)

cube_id = scene.add_triangles(t_cube)

  1. calculate intersection between the rays and the mesh by

ans = scene.cast_rays(rays)

  1. retreave the distance between the camera origin p1 and the intersection by

dist = ans[‘t_hit’][0].numpy()

  1. calculate the projection of the vertex (x,y) into the world coordinate (X,Y)

(X,Y,Z) = p1 + dist*v

Thank you.

1 Like

I add some more materials for (hopefully) active discussions.

The objective of our project is to specify positions of some rice plants in world coordinate.
The following image shows examples of mapped bboxes on world coordinate (EPSG:32654).

The mapping of each vertex has been done by the following lines.

crs_4326 = CRS.from_epsg(4326)
crs_32654 = CRS.from_epsg(32654)
transformer = Transformer.from_crs(crs_4326,crs_32654)
local_pos = p1 + dist*v
gps_world = rec.get_reference().to_lla(*local_pos)
gps_world2 = tuple(transformer.transform(*gps_world))

As you can see, Metashape’s projections (red polygons) accurately surround rice plants while OpenSfM’s projections (blue polygons) are out of position.

As for the accuracy of the georeferencing, please see the following GCP error details and the projections.


I don’t see any significant error in ODM georeferencing that would cause the global rotation.

Does anyone have any idea?

1 Like

Welcome!

This is some in-depth analysis going on…

Could you help us out a bit with understanding your system and the software packages you’re using to get these results?

Something like:

  1. Operating System and Version
    eg: Windows 11, MacOS 15.1, Ubuntu Linux 20.04LTS, etc…
  2. Hardware Specifications
    eg: 32GB RAM, i7-6700k, NVIDIA GTX 1050TI OC, 1TB SSD, etc…
  3. WebODM Install Method
    eg: Native installer, Docker, Snap, GitHub download, compiled from source
  4. WebODM/ODM Version
    eg: WebODM v1.9.12 Build 55, ODM v2.8.0, etc…
  5. Procesing Node
    eg: Automatic, Lightning, local (node-odm-1), etc…

But also for MetaShape. There can be significant changes in behavior between different versions of the same program.

1 Like

Hi,

At first, I made the ODM project via docker on Windows 11 by using the following command.

docker run -ti --rm -v F:/ODM/datasets:/datasets --gpus all opendronemap/odm:gpu --project-path /datasets Project

Then, projection of bboxes defined in camera (image) coordinate onto world coordinate was done by mapillary/OpenSfM on Ubuntu.

  1. Operating systems
  • ODM: Windows 11 Pro
  • OpenSfM: Ubuntu 20.04.4 LST
  1. Hardware specifications
  • ODM
    Memory: 128 GB RAM,
    CPU: i9-11900K 3.50GHz
    GPU: NVIDIA GeForce RTX 3080 Laptop GPU
    SSD1: 2TB SSD (sytem)
    HDD1: 4TB HDD (ODM data)

  • OpenSfM:
    Memory: 32 GB RAM
    CPU: i7-1065G7 1.30GHz
    GPU: Intel Iris Plus Graphics G7
    SSD2: 2TB SSD (system)
    SSD3: 4TB SSD (ODM data, copied from HDD1)

  1. WebODM
    not installed

  2. ODM version
    2.8.2

  3. OpenSfM environment on Ubuntu 20.04.4
    I cerated by the following command under /ODM folder.

git clone --recursive GitHub - mapillary/OpenSfM: Open source Structure-from-Motion pipeline
cd OpenSfM
docker build -t my_opensfm -f Dockerfile .
docker run -ti -v /ODM:/ODM my_opensfm

1 Like

Hmm… So to be clear, you’ve not tested this behavior under our current v2.8.4 release without any other customizations, correct?

Would you be able to to see if there is any difference?

1 Like

Thanks for your suggestion.
I downloaded the latest soruce codes (v2.8.4) from https://github.com/OpenDroneMap/ODM/archive/refs/tags/v2.8.4.zip, then built a docker image in Windows 11.
The result was more or less identical to the previous … (see the following image. Red polygons: Metashape’s projection, dashed blue polygons: OpenSfM’s projections).

This time, the project was built by the following command with some customizations.

docker run -ti --rm -v F:/ODM/datasets:/datasets --gpus all odm284:gpu --project-path /datasets Project --pc-quality high –-orthophoto-resolution 1.0 --ignore-gsd

I also compared the result with a result without any customization (dashed orange polygons in the image) to see there is no difference occurred by the options.

1 Like

Awesome work, thank you!

And as a final run, would you be able to test it without GPU acceleration as well?

We’ve noticed some differences in reconstruction behavior between the CPU and GPU pipeline in the past and I don’t know if that will influence the rotation you’re finding or not.

1 Like

Comparison between CPU and GPU.


Red polygons: Metashpae’s projection, orange polygons: ODM’s projection (GPU), dashed magenta polygons: ODM’s projection (CPU).

CPU’s customizations is as follows,

docker run -ti --rm -v F:/ODM/datasets:/datasets odm284 --project-path /datasets Project_cpu --pc-quality high –-orthophoto-resolution 1.0 --ignore-gsd

The difference between GPU and CPU seems to be minor.

1 Like

Beautiful stuff! Thank you so much for taking the time to help test out the various combinations and to ensure current builds are tested as well.

At this juncture, I’m not quite sure what could be going on here for this systematic error.

Would you be willing/able to open an issue with what you’ve provided above with OpenSFM’s GitHub, or would you prefer that I do so for you?

1 Like

I’ve just been looking at Google Earth images to see what date images are being used in WebODM, and noticed a slight rotation between GE and what is shown in WebODM. Could that be the issue here?

2 Likes

It looks like the trial plot boundaries which appear to be surveyed are rotated when reconstructed via OpenSFM/OpenDroneMap, irrespective of the background. Or at least I hope the boundaries were surveyed not from Google Earth imagery.

But yes, it’s interesting that the canvas has a slight difference between Google Earth and our basemap…

2 Likes

@kuniakiuto, can you please provide a bit more detail about how the boundaries of the plots were established? RTK survey for the corners of the plots? I’m trying to see how we can approach looking further into this, but in order to do so, we need to be confident that what we’re looking at is pretty unambiguous ground-truth data.

1 Like

Thank you!!
The GCP locations were collected by a RTK W-band bluetooth GNSS receiver (see the following link).

I believe the positional accuracy is less than a few cm.

1 Like