Tracking 2d image pixel coordinates to geographic coordinates from sparse cloud

[Jasper]

Hello all,

Given an input image with a bounding box around an object (essentially a known pixel coordinate), I want to return the real-world geographic coordinates in the sparse cloud for those corresponding pixels.

There are many potentially useful files such as the reconstruction.json,
(example: OpenSfM/data/berlin/reconstruction_example.json at main · mapillary/OpenSfM · GitHub,
file format documentation: Dataset Structure — OpenSfM 0.5.2 documentation),
the cameras.json and a few others that could potentially help.

I’d greatly appreciate if anyone has advice on how to approach this best, and if I need to project the 3d points on each shot, or simply access corresponding 2d pixel locations somewhere hidden in the file structures for the point cloud.

This is similar problem to a post by Tim:
Calculating geocoordinates for image pixels after processing , but instead of using the orthophoto for returning the corresponding pixel/ bounding box location I want to use the sparse cloud as it can happen faster (before ortho creation) and with less computational resources.

Thank you in advance!

[Jasper]

Two simplifications that may make this problem easier are:

1. I am mainly interested in returning the xy (Lat Lon or UTM) of the corresponding pixels in the point cloud. The z is not required.
2. The imagery I am collecting is nadir.