Batch uploading of files

I’m confused about this comment https://github.com/GoogleCloudPlatform/google-cloud-python/blob/master/storage/google/cloud/storage/blob.py#L589

What is the correct way of uploading blobs in a batch?

    client = storage.Client.from_service_account_json('client-secret.json', project=PROJECT_NAME)
    bucket = client.get_bucket(BUCKET_NAME)
    with client.batch():
        for i in range(10):
            with open('base.py', 'rb') as my_file:
                blob = storage.Blob('/test/{}'.format(i), bucket)
                blob.upload_from_file(my_file, client=client)

This, of course, won’t work because is using client._base_connection instead of client.current_batch.

[tseaver]

@arvindnrbt with consulting hat on “It depends…”. You could:

• Iterate and upload files one by one.
• Populate a queue.Queue with the filenames and have worker threads upload items from the queue.
• Populate a cross-process queue and have worker processes upload items from the queue.

You can’t:

• Upload a zipfile and have GCS automagically explode its entries into blobs / objects.
• Create a “batched” MIME request and do multiple uploads in a single request.

[xucian]

For anyone that might want to batch upload/download. I wanted to speed up this process and it turned out that spawning a few threads and using a different google storage client instance in each did improve the speed significantly. Tested on a 6-core machine. I just needed to partition the list of files to upload/download myself. Hope this will be helpful for some.