Importing Images for Processing in WebODM API

As per API documentaion to import images for prcoessing

images = [
    ('images', ('image1.jpg', open('image1.jpg', 'rb'), 'image/jpg')), 
    ('images', ('image2.jpg', open('image2.jpg', 'rb'), 'image/jpg')),
    # ...
]

In my case webODM and nodeODM running in an EC2 instance and my 12GB images are stored in different instance. When I call the Create Task API by forming image array as above, the RAM usage goes up to 12GB (as open(‘img’) creates BufferedArray). So to process 30GB of data I need 30GB of RAM to make api call and pass Images Array.

Is there any way of optimizing or other way of doing ?
Else is there any better method to process images from s3 bucket in webODM API?
Thanks

1 Like

You can upload images in chunks by making sure to pass a partial parameter when creating a new task: WebODM/tasks.py at 2643aef14b6fb8b67e495b64c286c1db0e2b429a · OpenDroneMap/WebODM · GitHub

Then just do multiple POST uploads.

There’s no S3 support at the moment. We’d welcome a contribution!

Can you share some details as to what you are building?

4 Likes

That was great feature. Is there is any example or documentation. I couldn’t able to find it here
https://docs.webodm.org/#create-a-task

2 Likes

Hi pierotofy,
Thanks for your response and this great feature. I am experinmenting like processing lot of images

I have tried to implement as per your answer I cannot figure out how to make chunks of multiple images. I tried with with one image chunk like below

     def read_in_chunks(file_object, chunk_size=65536):
           while True:
                 data = file_object.read(chunk_size)
                 if not data:
                      break
                 yield data

           def createImageChunks():
                webodm_url = "http://url:8000"
                resAuth = requests.post(webodm_url+'/api/token-auth/',
                                data={'username': 'usr',
                                'password': 'pwd'}).json()
                token = resAuth['token']
                res = requests.post(webodm_url+'/api/projects/{}/tasks/'.format(int('10')), 
                        headers={'Authorization': 'JWT {}'.format(token)},
                        data = {'partial':True}) .json()
                task_id = res['id']

                """ f = [open('./DJI_0003.JPG', 'rb'), open('./DJI_0004.JPG', 'rb'),  open('./DJI_0005.JPG', 'rb')] """
           
                 f = open('./DJI_0003.JPG', 'rb')
                index = 0

                offset = 0

                headers = {}

                for chunk in read_in_chunks(f):

                    offset = index + len(chunk)

                    index = offset

                    try:

                        res = requests.post(webodm_url+'/api/projects/{}/tasks/{}/upload/'.format(int('10'), task_id), 

                                headers={'Authorization': 'JWT {}'.format(token),

                                'Content-Type' : 'multipart/form-data', 

                                },

                                files = chunk) .json()  

                        print(res)                

                    except Exception as e:

                        print (e)


            createImageChunks()

The res variable prints as No files uploaded.
what are the parameter to be use when calling upload . Is there any sample code or documentation on how to do chunked uploads and how to call upload api

Thanks

2 Likes