Upgrading from 6.4.1 to 6.5.X impacts response time and cpu usage

We have upgraded our docker image from thumbor 6.4.1 to 6.5.1 yesterday. Both CPU and response time are increasing considerably ever since, that’s the only change that happened.

Expected behaviour

Nothing should dramatically change…

Actual behaviour

Both CPU usage and response-time went up considerably.

• Orange - time to fetch
• Blue - p95 of response time
• Yellow dashed - Max response time
• Red - deployment (first ones to upgrade, second one to downgrade)

Once I’ve downgraded back to 6.4.1 the response time went down again…

• CPU Usage per pod after the deployments…

The spike matches the deployment time, the CPU usage went down over night because of the less intensive traffic but the response time was still higher (1.5x) the one before.

Operating system

Ubuntu 16.04 on K8s

Your thumbor.conf

    import re

    ALLOWED_SOURCES=[
      # https://regexper.com/https://regexper.com/#^https:\/\/*(.+)?hey[.]car\/*(.+)
      re.compile(r'^https://*(.+)?hey[.]car/*(.+)'),
    ]

    MAX_SOURCE_SIZE = 500
    SECURITY_KEY = "REDACTED"
    MAX_WIDTH = 2000
    MAX_HEIGHT = 2000

    # Cache
    MAX_AGE = 365 * 24 * 60 * 60
    MAX_AGE_TEMP_IMAGE = 240

    # Defaults
    QUALITY = 100

    # Engines
    HTTP_LOADER_CURL_ASYNC_HTTP_CLIENT = True

    # Metrics
    METRICS = "thumbor.metrics.statsd_metrics"
    STATSD_PORT = 8125
    STATSD_HOST = "datadog-statsd.kube-system"
    STATSD_PREFIX = "image_proxy_service"

    # GC Options
    GC_INTERVAL = 1800

    # AutoWebp
    AUTO_WEBP = True

    # Sentry
    ERROR_HANDLER_MODULE = "thumbor.error_handlers.sentry"
    SENTRY_DSN = "REDACTED"

Any thoughts on what could’ve caused this? Is there any new config that I need to take into account when upgrading? I’ve been reading the changelog and haven’t found anything clearly related to that.

[kkopachev]

I’d say compile on image build time with libjpeg-turbo. Library and headers are probably already preinstalled on debian. We use mozjpeg in our project, though, but that might create different images.

RUN PILLOW_VERSION=$(python -c 'import PIL; print(PIL.__version__)') && \
    pip uninstall -y pillow || true && \
# https://github.com/python-pillow/Pillow/pull/3241
    LIB=/usr/lib/x86_64-linux-gnu/ \
# https://github.com/python-pillow/Pillow/pull/3237 or https://github.com/python-pillow/Pillow/pull/3245
    INCLUDE=/usr/include/x86_64-linux-gnu/ \
    pip install --no-cache-dir -U --force-reinstall --no-binary=:all: "pillow<=$PILLOW_VERSION" \
#    --global-option="build_ext" --global-option="--debug" \
    --global-option="build_ext" --global-option="--enable-imagequant" \
    --global-option="build_ext" --global-option="--enable-lcms" \
    --global-option="build_ext" --global-option="--enable-zlib" \
    --global-option="build_ext" --global-option="--enable-jpeg" \
    --global-option="build_ext" --global-option="--enable-tiff"

Above should work on base python:2 image without installing any additional libraries.

For mozjpeg, install it first, then pass additional PKG_CONFIG_PATH=/opt/mozjpeg/lib64/pkgconfig/ to install line.

[gingerlime]

@kkopachev deserves a medal 🏅

@kkopachev - What’s the best workaround for now for docker-based images in your opinion? pinning Pillow to 5.0.0? Compiling it into the docker image? (if so, would appreciate any tips…)