Intermittent 500 errors after moving AWS accounts and Serverless Image Handler version

Describe the issue We have transferred our site to a new AWS account. Which meant moving all the images into a new bucket and setting up Serverless Image Handler in CloudFormation. I think we have probably upgraded from v5.2.0 > v6.0.0. When we now load the website we get 500 errors from an intermittent number of images (within ‘img’ tags). The erroring images seem to lower over time. I.e. if we have 10 images on the page broken today, it could be 5 tomorrow.

If you copy one of these errored images you get the following:

ERRORED IMAGE

curl 'https://img2.picle.io/eyJlZGl0cyI6eyJyb3RhdGUiOm51bGwsInJlc2l6ZSI6eyJ3aWR0aCI6NjAwLCJoZWlnaHQiOjQ1MCwiZml0IjoiY292ZXIifX0sImJ1Y2tldCI6InByb2QuaW1nMi5waWNsZS5pbyIsImtleSI6ImltYWdlc1wvWW5QcDFLMkpcLzI3ZTAxOWVlLTMwY2ItNDU0ZS05ODAxLWZhMzJlOTdkYTE1OSJ9' \
  -H 'authority: img2.picle.io' \
  -H 'accept: image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8' \
  -H 'accept-language: en-US,en;q=0.9,nb;q=0.8,it;q=0.7,la;q=0.6' \
  -H 'cache-control: no-cache' \
  -H 'pragma: no-cache' \
  -H 'referer: https://picle.io/' \
  -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: image' \
  -H 'sec-fetch-mode: no-cors' \
  -H 'sec-fetch-site: same-site' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36' \
  --compressed

This returns:

{"message": "Internal server error"}

But if we load the same image URL directly within a browser we get a correct 200 image:

curl 'https://img2.picle.io/eyJlZGl0cyI6eyJyb3RhdGUiOm51bGwsInJlc2l6ZSI6eyJ3aWR0aCI6NjAwLCJoZWlnaHQiOjQ1MCwiZml0IjoiY292ZXIifX0sImJ1Y2tldCI6InByb2QuaW1nMi5waWNsZS5pbyIsImtleSI6ImltYWdlc1wvWW5QcDFLMkpcLzI3ZTAxOWVlLTMwY2ItNDU0ZS05ODAxLWZhMzJlOTdkYTE1OSJ9' \
  -H 'authority: img2.picle.io' \
  -H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'accept-language: en-US,en;q=0.9,nb;q=0.8,it;q=0.7,la;q=0.6' \
  -H 'cache-control: no-cache' \
  -H 'pragma: no-cache' \
  -H 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="102", "Google Chrome";v="102"' \
  -H 'sec-ch-ua-mobile: ?0' \
  -H 'sec-ch-ua-platform: "macOS"' \
  -H 'sec-fetch-dest: document' \
  -H 'sec-fetch-mode: navigate' \
  -H 'sec-fetch-site: none' \
  -H 'sec-fetch-user: ?1' \
  -H 'upgrade-insecure-requests: 1' \
  -H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36' \
  --compressed

Note, after a while both the above will probably start working. I’m not sure if its an import issue, it could take a while to re-process all the images again.

Version: v6.0.0 Region: us-east-1 Was the solution modified from the version published on this repository? No.

[amcfarlane]

@fisenkodv Thanks again for your help with this. Finally a solution. 🎉

Our old AWS account has the default quota of concurrent Lambda executions of 1000. The new account had 10. 😅 So on pages of a large number of images after the 10 concurrent was reached we got the 500 error. This was then cached for a period of time by CloudFront. So we were basically rendering 10 (probably more) images on each view of the page.

Asking AWS to increase our concurrent Lambda executions to 1000 has fixed the issue. Once all the new images are cached again this won’t be an issue.

It might be helpful to add a note about this to the Serverless Image Handler Implementation Guide. Does anyone know who I can contact about that?

[amcfarlane]

@fisenkodv thanks for your ongoing support with this. Unfortunately we have an hopefully temporary issue with our entire AWS account at the moment which is making debugging impossible. Before the issues we where experiencing the issue on other browsers and devices. But I can debug more once we are back up and running.