Partial multipart uploads in WebODMAPI

[hcwinsemius]

Hello!

I want to use the WebODM API to stream images from one remote location to the other. It has to work chunk by chunk to prevent lots of memory problems, and to be able to pick up a partial upload if for some reason connections are broken.

I found this post, that also has a similar use case. Apparently there is a “partial”: True option that I can pass along, but I couldn’t get that to work. I now tried the following code piece.

for n, img in enumerate(imgs):
    c = s.get("file://" + img, stream=True).content
    images[n] = ('images', (os.path.split(img)[-1], c, 'image/jpg'))
    if n > 0:
        images[n-1] = ('images', (images[n-1][1], None, 'image/jpg'))
    offset = index + len(c)
    if n < len(imgs)-1:
        length = offset
        partial = True
    else:
        length = offset - 1
        partial = False
    headers["Content-Range"] = f'bytes {index}-{offset-1}/{length}'
    url = f"{odmconfig.host}:{odmconfig.port}/api/projects/{project_id}/tasks/"
    res = requests.post(
        url,
        files=images,
        headers=headers,
        data={
            'options': options,
            'partial': partial,
        }
    )
    index = offset

As you can see, I am testing with a stream of quasi-remote files. This leads to one task being created for each post, with a uuid for each. each task holds zero photos. So my questions are:

1. How do you create only one unique task in this manner?
2. What is the appropriate way to upload one chunk of data to an existing task as partial upload?
3. How do you notify in the post that uploads are done and the task can be executed?
   Thanks!

Hessel

[Saijin_Naib]

Welcome, Hessel!

Sounds like a very interesting development.

Unfortunately, I will not be able to provide any insight. Hopefully someone else comes by to help

[pierotofy]

1. Create the task. (Note the partial: true which tells WebODM not to start the task).

await fetch("http://localhost:8000/api/projects/1/tasks/", {
        "body": "{\"name\":\"Sheffield Court - 4/29/2018\",\"options\":[{\"name\":\"dsm\",\"value\":true}],\"processing_node\":1,\"auto_processing_node\":false,\"partial\":true}",
    "method": "POST"
});

The response will give you a task ID.

id: 8ebf58c8-689e-4ce4-bb60-d9da654ab0f7

2. Upload files (repeat this for every file)

await fetch("http://localhost:8000/api/projects/1/tasks/8ebf58c8-689e-4ce4-bb60-d9da654ab0f7/upload/", {
    "headers": {
        "Content-Type": "multipart/form-data; boundary........"
    },
    "body": "formdata binary"
    "method": "POST",
});

3. Commit

await fetch("http://localhost:8000/api/projects/1/tasks/8ebf58c8-689e-4ce4-bb60-d9da654ab0f7/commit/", {
    "method": "POST",
});

That’s it!

This is not documented in https://docs.webodm.org (we should probably add it at some point).

[hcwinsemius]

Thanks a lot! I have some progress. The partial task creation works as expected. Now I am using requests to get a post with a single image in place. I have done this as follows, the variable ‘c’ contains the binary data of the image.

headers["Content-type"] = 'multipart/form-data; boundary="XXXXX"'
images = [('images', ('SOME_IMG.JPG', c, 'image/jpg'))]
url = f"{odmconfig.host}:{odmconfig.port}/api/projects/{project_id}/tasks/{task_id}/upload/"
res = requests.post(
    url,
    headers=headers,
    data={
        images
     },

I have also tried this with files=images instead of data = {images}. The upload route is definitely reached but the request.FILES part remains empty. I receive a response back saying that there are not files.

It is probably an obvious thing. What am I doing wrong? Thanks for the help.

[pierotofy]

Sorry, perhaps my example threw you off (the content-type was just an example, with xxxxxxxx being a placeholder), but you’re not using requests properly. You probably want to use requests-toolbelt · PyPI to make multipart form uploads.

[hcwinsemius]

Thanks @pierotofy I have a solution. Brief code extract below.

import requests
from requests_toolbelt.multipart.encoder import MultipartEncoder
folder = "/some/folder/containing/your/photos"
url = f"http://localhost:8000/api/projects/{project_id}/tasks/"
token = f"{get this from another request}"
headers = {'Authorization': 'JWT {}'.format(token)}
data = {"partial": True}

# create a new task
r = requests.post(
    url,
    headers=headers,
    data=data,
)
# get the task id to construct partial uploads
task_id = r.json()["id"]
print(f"New task has id: {task_id}")
url_upload = url + task_id + "/upload/"
url_update = url + task_id + "/update/"
url_commit = url + task_id + "/commit/"
for fn in fns:
    with open(fn, "rb") as f:
        c = f.read()
        fields = {"images": (os.path.split(fn)[-1], c, 'image/jpg')}
    m = MultipartEncoder(
        fields=fields
        )
    headers["Content-type"] = m.content_type
    r = requests.post(
        url_upload,
        data=m,
        headers=headers,
    )
    print(r.json())

# now commit the task for processing
r = requests.post(
    url_commit,
    # headers=headers,
    headers={'Authorization': 'JWT {}'.format(token)},
)
print(r.json())