Not able use Pillow inside Dask

• dask-image version: 0.2.0
• Python version: Python 3.6.8
• Operating System: MacOS 10.13.6

Description

I am trying to access an Image by using Pillow library inside Dask task.

What I Did

I am trying to access an Image by using Pillow library inside Dask task.

Code:

from dask.distributed import Client

from PIL import Image

IMAGE_RESULT_FOLDER = '/Users/xyzabc/datasets/sample_images/result/'

def resize_single_image(image_path):
    """[summary]

    Arguments:
        image_path {string} -- Image path to access the image and resize it

    Returns:
        string -- resized image location
    """

    size = 128, 128

    fileparts = image_path.split('/')
    old_filename = fileparts[len(fileparts)-1]
    new_filename = fileparts[len(fileparts)-1].replace('.png', '_new.png')
    new_filepath = IMAGE_RESULT_FOLDER + new_filename

    print('reading : ', image_path)

    try:

        # resize the image here
        current_image = Image.open(image_path)
        current_image.thumbnail(size, Image.ANTIALIAS)

        current_image.save(new_filepath, "PNG")

    except IOError:
        print('Image Error, so skipping')

    return new_filepath

if __name__ == '__main__':

    list_images = [
        '/Users/str-kwml0011/datasets/sample_images/1001.png',
        '/Users/str-kwml0011/datasets/sample_images/1002.png',
    ]

    client = Client('tcp://192.168.2.26:8786')

    A = client.map(resize_single_image, list_images)
    total = client.submit(list, A)

    print(total.result()) # print all processed image list

The worker throws this error:

ImportError: cannot import name 'UnidentifiedImageError'

Error as it is:

distributed.protocol.pickle - INFO - Failed to deserialize b'\x80\x04\x95o\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x0e_fill_function\x94\x93\x94(h\x00\x8c\x0f_make_skel_func\x94\x93\x94h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x07K\x0bKCC\x92d\t}\x01|\x00j\x00d\x02\x83\x01}\x02|\x02t\x01|\x02\x83\x01d\x03\x18\x00\x19\x00}\x03|\x02t\x01|\x02\x83\x01d\x03\x18\x00\x19\x00j\x02d\x04d\x05\x83\x02}\x04t\x03|\x04\x17\x00}\x05t\x04d\x06|\x00\x83\x02\x01\x00y(t\x05j\x06|\x00\x83\x01}\x06|\x06j\x07|\x01t\x05j\x08\x83\x02\x01\x00|\x06j\t|\x05d\x07\x83\x02\x01\x00W\x00n\x1c\x04\x00t\nk\nr\x8c\x01\x00\x01\x00\x01\x00t\x04d\x08\x83\x01\x01\x00Y\x00n\x02X\x00|\x05S\x00\x94(\x8c\xa1[summary]\n\n    Arguments:\n        image_path {string} -- Image path to access the image and resize it\n\n    Returns:\n        string -- resized image location\n    \x94K\x80\x8c\x01/\x94K\x01\x8c\x04.png\x94\x8c\x08_new.png\x94\x8c\nreading : \x94\x8c\x03PNG\x94\x8c\x18Image Error, so skipping\x94K\x80K\x80\x86\x94t\x94(\x8c\x05split\x94\x8c\x03len\x94\x8c\x07replace\x94\x8c\x13IMAGE_RESULT_FOLDER\x94\x8c\x05print\x94\x8c\x05Image\x94\x8c\x04open\x94\x8c\tthumbnail\x94\x8c\tANTIALIAS\x94\x8c\x04save\x94\x8c\x07IOError\x94t\x94(\x8c\nimage_path\x94\x8c\x04size\x94\x8c\tfileparts\x94\x8c\x0cold_filename\x94\x8c\x0cnew_filename\x94\x8c\x0cnew_filepath\x94\x8c\rcurrent_image\x94t\x94\x8c\x16image_resizer_error.py\x94\x8c\x13resize_single_image\x94K\x17C\x1a\x00\n\x04\x02\n\x01\x10\x01\x18\x01\x08\x02\n\x02\x02\x03\n\x01\x0e\x02\x10\x02\x0e\x01\x0e\x02\x94))t\x94R\x94J\xff\xff\xff\xff}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94\x8c\x16image_resizer_error.py\x94u\x87\x94R\x94}\x94(\x8c\x07globals\x94}\x94(h\x19h\x00\x8c\tsubimport\x94\x93\x94\x8c\tPIL.Image\x94\x85\x94R\x94h\x17\x8c2/Users/xyzabc/datasets/sample_images/result/\x94u\x8c\x08defaults\x94N\x8c\x04dict\x94}\x94\x8c\x0eclosure_values\x94N\x8c\x06module\x94h0\x8c\x04name\x94h)\x8c\x03doc\x94h\x0b\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0bannotations\x94}\x94\x8c\x08qualname\x94h)\x8c\nkwdefaults\x94NutR.'
Traceback (most recent call last):
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/worker.py", line 3237, in loads_function
    result = cache_loads[bytes_object]
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/utils.py", line 1497, in __getitem__
    value = super().__getitem__(key)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/collections/__init__.py", line 991, in __getitem__
    raise KeyError(key)
KeyError: b'\x80\x04\x95o\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x0e_fill_function\x94\x93\x94(h\x00\x8c\x0f_make_skel_func\x94\x93\x94h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x07K\x0bKCC\x92d\t}\x01|\x00j\x00d\x02\x83\x01}\x02|\x02t\x01|\x02\x83\x01d\x03\x18\x00\x19\x00}\x03|\x02t\x01|\x02\x83\x01d\x03\x18\x00\x19\x00j\x02d\x04d\x05\x83\x02}\x04t\x03|\x04\x17\x00}\x05t\x04d\x06|\x00\x83\x02\x01\x00y(t\x05j\x06|\x00\x83\x01}\x06|\x06j\x07|\x01t\x05j\x08\x83\x02\x01\x00|\x06j\t|\x05d\x07\x83\x02\x01\x00W\x00n\x1c\x04\x00t\nk\nr\x8c\x01\x00\x01\x00\x01\x00t\x04d\x08\x83\x01\x01\x00Y\x00n\x02X\x00|\x05S\x00\x94(\x8c\xa1[summary]\n\n    Arguments:\n        image_path {string} -- Image path to access the image and resize it\n\n    Returns:\n        string -- resized image location\n    \x94K\x80\x8c\x01/\x94K\x01\x8c\x04.png\x94\x8c\x08_new.png\x94\x8c\nreading : \x94\x8c\x03PNG\x94\x8c\x18Image Error, so skipping\x94K\x80K\x80\x86\x94t\x94(\x8c\x05split\x94\x8c\x03len\x94\x8c\x07replace\x94\x8c\x13IMAGE_RESULT_FOLDER\x94\x8c\x05print\x94\x8c\x05Image\x94\x8c\x04open\x94\x8c\tthumbnail\x94\x8c\tANTIALIAS\x94\x8c\x04save\x94\x8c\x07IOError\x94t\x94(\x8c\nimage_path\x94\x8c\x04size\x94\x8c\tfileparts\x94\x8c\x0cold_filename\x94\x8c\x0cnew_filename\x94\x8c\x0cnew_filepath\x94\x8c\rcurrent_image\x94t\x94\x8c\x16image_resizer_error.py\x94\x8c\x13resize_single_image\x94K\x17C\x1a\x00\n\x04\x02\n\x01\x10\x01\x18\x01\x08\x02\n\x02\x02\x03\n\x01\x0e\x02\x10\x02\x0e\x01\x0e\x02\x94))t\x94R\x94J\xff\xff\xff\xff}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94\x8c\x16image_resizer_error.py\x94u\x87\x94R\x94}\x94(\x8c\x07globals\x94}\x94(h\x19h\x00\x8c\tsubimport\x94\x93\x94\x8c\tPIL.Image\x94\x85\x94R\x94h\x17\x8c2/Users/xyzabc/datasets/sample_images/result/\x94u\x8c\x08defaults\x94N\x8c\x04dict\x94}\x94\x8c\x0eclosure_values\x94N\x8c\x06module\x94h0\x8c\x04name\x94h)\x8c\x03doc\x94h\x0b\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0bannotations\x94}\x94\x8c\x08qualname\x94h)\x8c\nkwdefaults\x94NutR.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 55, in loads
    return pickle.loads(x)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 1096, in subimport
    __import__(name)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/PIL/Image.py", line 44, in <module>
    from . import (
ImportError: cannot import name 'UnidentifiedImageError'
distributed.worker - WARNING - Could not deserialize task
Traceback (most recent call last):
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/worker.py", line 3237, in loads_function
    result = cache_loads[bytes_object]
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/utils.py", line 1497, in __getitem__
    value = super().__getitem__(key)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/collections/__init__.py", line 991, in __getitem__
    raise KeyError(key)
KeyError: b'\x80\x04\x95o\x04\x00\x00\x00\x00\x00\x00\x8c\x17cloudpickle.cloudpickle\x94\x8c\x0e_fill_function\x94\x93\x94(h\x00\x8c\x0f_make_skel_func\x94\x93\x94h\x00\x8c\r_builtin_type\x94\x93\x94\x8c\x08CodeType\x94\x85\x94R\x94(K\x01K\x00K\x07K\x0bKCC\x92d\t}\x01|\x00j\x00d\x02\x83\x01}\x02|\x02t\x01|\x02\x83\x01d\x03\x18\x00\x19\x00}\x03|\x02t\x01|\x02\x83\x01d\x03\x18\x00\x19\x00j\x02d\x04d\x05\x83\x02}\x04t\x03|\x04\x17\x00}\x05t\x04d\x06|\x00\x83\x02\x01\x00y(t\x05j\x06|\x00\x83\x01}\x06|\x06j\x07|\x01t\x05j\x08\x83\x02\x01\x00|\x06j\t|\x05d\x07\x83\x02\x01\x00W\x00n\x1c\x04\x00t\nk\nr\x8c\x01\x00\x01\x00\x01\x00t\x04d\x08\x83\x01\x01\x00Y\x00n\x02X\x00|\x05S\x00\x94(\x8c\xa1[summary]\n\n    Arguments:\n        image_path {string} -- Image path to access the image and resize it\n\n    Returns:\n        string -- resized image location\n    \x94K\x80\x8c\x01/\x94K\x01\x8c\x04.png\x94\x8c\x08_new.png\x94\x8c\nreading : \x94\x8c\x03PNG\x94\x8c\x18Image Error, so skipping\x94K\x80K\x80\x86\x94t\x94(\x8c\x05split\x94\x8c\x03len\x94\x8c\x07replace\x94\x8c\x13IMAGE_RESULT_FOLDER\x94\x8c\x05print\x94\x8c\x05Image\x94\x8c\x04open\x94\x8c\tthumbnail\x94\x8c\tANTIALIAS\x94\x8c\x04save\x94\x8c\x07IOError\x94t\x94(\x8c\nimage_path\x94\x8c\x04size\x94\x8c\tfileparts\x94\x8c\x0cold_filename\x94\x8c\x0cnew_filename\x94\x8c\x0cnew_filepath\x94\x8c\rcurrent_image\x94t\x94\x8c\x16image_resizer_error.py\x94\x8c\x13resize_single_image\x94K\x17C\x1a\x00\n\x04\x02\n\x01\x10\x01\x18\x01\x08\x02\n\x02\x02\x03\n\x01\x0e\x02\x10\x02\x0e\x01\x0e\x02\x94))t\x94R\x94J\xff\xff\xff\xff}\x94(\x8c\x0b__package__\x94N\x8c\x08__name__\x94\x8c\x08__main__\x94\x8c\x08__file__\x94\x8c\x16image_resizer_error.py\x94u\x87\x94R\x94}\x94(\x8c\x07globals\x94}\x94(h\x19h\x00\x8c\tsubimport\x94\x93\x94\x8c\tPIL.Image\x94\x85\x94R\x94h\x17\x8c2/Users/xyzabc/datasets/sample_images/result/\x94u\x8c\x08defaults\x94N\x8c\x04dict\x94}\x94\x8c\x0eclosure_values\x94N\x8c\x06module\x94h0\x8c\x04name\x94h)\x8c\x03doc\x94h\x0b\x8c\x17_cloudpickle_submodules\x94]\x94\x8c\x0bannotations\x94}\x94\x8c\x08qualname\x94h)\x8c\nkwdefaults\x94NutR.'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/worker.py", line 2415, in _maybe_deserialize_task
    function, args, kwargs = _deserialize(*self.tasks[key])
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/worker.py", line 3248, in _deserialize
    function = loads_function(function)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/worker.py", line 3239, in loads_function
    result = pickle.loads(bytes_object)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/distributed/protocol/pickle.py", line 55, in loads
    return pickle.loads(x)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/cloudpickle/cloudpickle.py", line 1096, in subimport
    __import__(name)
  File "/Users/xyzabc/anaconda3/envs/py36/lib/python3.6/site-packages/PIL/Image.py", line 44, in <module>
    from . import (
ImportError: cannot import name 'UnidentifiedImageError'

[GenevieveBuckley]

Ok, so what happens if you do this?

from PIL import Image

img1 = Image.open('/Users/str-kwml0011/datasets/sample_images/1001.png')
img2 = Image.open('/Users/str-kwml0011/datasets/sample_images/1002.png')

The UnidentifiedImageError you got strongly suggests that PIL is trying to open a filename that is not a valid image. Could there be a corrupt file somewhere in your folder?

[GenevieveBuckley]

Could you post a minimal, reproducible example of the code you used that gives you this error?

dask-image doesn’t actually handle opening image files ourselves, we use the pims library, which is then calling pillow in your example. Can you open the image that’s giving you trouble using Pillow directly (no dask)?