[transforms] Troubles with padding for PIL image in mode P

🐛 Describe the bug

Hello there 👋

I was running some training for semantic segmentation using the implementation of RandomCrop provided in the references folder. And apparently there is an issue with the padding function for PIL images. Please note that this was not happening a few months back.

The following snippet

from PIL import Image
from torchvision.transforms.functional import pad

img = Image.open('/path/to/any/img.jpg').convert('P')
padded = pad(img, (0, 0, 5, 10), fill=255)

produces

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-e151dc3a245c> in <module>
----> 1 padded = pad(img, (0, 0, 5, 10), fill=255)

~/miniconda3/lib/python3.8/site-packages/torchvision/transforms/functional.py in pad(img, padding, fill, padding_mode)
    470     """
    471     if not isinstance(img, torch.Tensor):
--> 472         return F_pil.pad(img, padding=padding, fill=fill, padding_mode=padding_mode)
    473 
    474     return F_t.pad(img, padding=padding, fill=fill, padding_mode=padding_mode)

~/miniconda3/lib/python3.8/site-packages/torchvision/transforms/functional_pil.py in pad(img, padding, fill, padding_mode)
    162         if img.mode == "P":
    163             palette = img.getpalette()
--> 164             image = ImageOps.expand(img, border=padding, **opts)
    165             image.putpalette(palette)
    166             return image

~/miniconda3/lib/python3.8/site-packages/PIL/ImageOps.py in expand(image, border, fill)
    401 
    402         draw = ImageDraw.Draw(out)
--> 403         draw.rectangle((0, 0, width - 1, height - 1), outline=color, width=border)
    404     else:
    405         out = Image.new(image.mode, (width, height), color)

~/miniconda3/lib/python3.8/site-packages/PIL/ImageDraw.py in rectangle(self, xy, fill, outline, width)
    257             self.draw.draw_rectangle(xy, fill, 1)
    258         if ink is not None and ink != fill and width != 0:
--> 259             self.draw.draw_rectangle(xy, ink, 0, width)
    260 
    261     def rounded_rectangle(self, xy, radius=0, fill=None, outline=None, width=1):

TypeError: an integer is required (got type tuple)

I investigated a bit on the main branch, and I believe the problem comes from this section: https://github.com/pytorch/vision/blob/main/torchvision/transforms/functional_pil.py#L163

With other padding modes or image modes, extra steps are taken to format the padding but here it’s not working. Regarding the PIL documentation, PIL.ImageOps.expand does not support tuple as a border argument. However there is the PIL.ImageOps.pad which does. I haven’t experimented with that function so I don’t know if that’s an easy fix 🤷‍♂️

Since this was not happening earlier, I checked the recent changes on PIL side, and their 8.3.0 release changed the expand function: https://github.com/python-pillow/Pillow/blob/8.3.0/src/PIL/ImageOps.py#L396-L402 So perhaps the quickfix would be to update the version constraint on PIL (https://github.com/pytorch/vision/blob/main/setup.py#L66) or perform a conditional operation depending on PIL’s version!

Happy to help, with some guidance 😃

Versions

PyTorch version: 1.10.0
Is debug build: False
CUDA used to build PyTorch: 11.3
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.3 LTS (x86_64)
GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.8.10 (default, Jun  4 2021, 15:09:15)  [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-5.11.0-40-generic-x86_64-with-glibc2.17
Is CUDA available: True
CUDA runtime version: 11.4.100
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2070 with Max-Q Design
Nvidia driver version: 470.57.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.2.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.2.2
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.2.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.2.2
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.2.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.2.2
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.2.2
HIP runtime version: N/A
MIOpen runtime version: N/A

Versions of relevant libraries:
[pip3] mypy==0.910
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.19.5
[pip3] torch==1.10.0
[pip3] torchcam==0.3.2.dev0+35e4e67
[pip3] torchvision==0.11.1
[conda] blas                      1.0                         mkl  
[conda] cudatoolkit               11.3.1               ha36c431_9    nvidia
[conda] mkl                       2021.3.0           h06a4308_520  
[conda] mkl-service               2.4.0            py38h7f8727e_0  
[conda] mkl_fft                   1.3.1            py38hd3c417c_0  
[conda] mkl_random                1.2.2            py38h51133e4_0  
[conda] mypy                      0.910                    pypi_0    pypi
[conda] mypy-extensions           0.4.3                    pypi_0    pypi
[conda] numpy                     1.19.5                   pypi_0    pypi
[conda] pytorch                   1.10.0          py3.8_cuda11.3_cudnn8.2.0_0    pytorch
[conda] pytorch-mutex             1.0                        cuda    pytorch
[conda] torchcam                  0.3.2.dev0+35e4e67           dev_0    <develop>
[conda] torchvision               0.10.1                   pypi_0    pypi

[frgfm]

Hi @oke-aditya 👋

Yes I saw that it excluded 8.3.0, but as mentioned in my previous comment, this issue is present for all 8.3.* 😅 Here is the implementation (with the bug) in 8.3.2: https://github.com/python-pillow/Pillow/blob/8.3.2/src/PIL/ImageOps.py#L384-L407

which is in every point identical to the one in 8.3.0: https://github.com/python-pillow/Pillow/blob/8.3.0/src/PIL/ImageOps.py#L383-L406

It has been fixed in >=8.4.0, so I think we should also exclude all the 8.3.* 😃

[oke-aditya]

Hi @frgfm

torchvision has excluded the Pillow versions before because of such issues.

https://github.com/pytorch/vision/blob/main/setup.py#L66

Can you also have a look at #4310 #4146. I guess it is duplicate of same.

https://github.com/pytorch/vision/issues/4146#issuecomment-875287090

We can close this now. Unless someone explicitly specifies Pillow==8.3.0, pip install torch torchvision will now install a compatible Pillow version.