Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Header transfer-encoding make Splash API return 504 Gateway Timeout

See original GitHub issue

I was developing a crawler using Splash when suddenly i started to receive a lot of gateway timeouts. Trying to troubleshooting the problem, i discover the cause of this is header transfer-encoding: chunked, i made a PoC (the url httpbin.org/headers returns the same headers i sent on request):

import requests
import json

ENDPOINT_SPLASH = 'http://localhost:8050/execute'


def test_with_custom_headers():
    lua_script = """
    function main(splash, args)
     splash:set_custom_headers({
       ["x-custom-header"] = "splash"
     })
     assert(splash:go(args.url))
     assert(splash:wait(0.5))
     return {
       html = splash:html()
     }
    end
    """

    payload = {
        'lua_source': lua_script,
        'url': 'https://httpbin.org/headers',
        'timeout': 15,
    }

    r = requests.post(url=ENDPOINT_SPLASH,
                      json=payload)

    result = json.loads(r.text)

    return result.get('html', result)


def test_with_content_encoding():
    lua_script = """
    function main(splash, args)
     splash:set_custom_headers({
       ["transfer-encoding"] = "chunked"
     })
     assert(splash:go(args.url))
     assert(splash:wait(0.5))
     return {
       html = splash:html()
     }
    end
    """

    payload = {
        'lua_source': lua_script,
        'url': 'https://httpbin.org/headers',
        'timeout': 15,
    }

    r = requests.post(url=ENDPOINT_SPLASH,
                      json=payload)

    result = json.loads(r.text)

    return result.get('html', result)


print("test_with_custom_headers: \n{}\n".format(test_with_custom_headers()))
print("test_with_content_encoding: \n{}".format(test_with_content_encoding()))

Results:

test_with_custom_headers: 
<html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">{
  "headers": {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", 
    "Accept-Encoding": "gzip, deflate", 
    "Accept-Language": "en,*", 
    "Host": "httpbin.org", 
    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/602.1 (KHTML, like Gecko) splash Version/9.0 Safari/602.1", 
    "X-Custom-Header": "splash"
  }
}
</pre></body></html>

test_with_content_encoding: 
{'info': {'timeout': 15.0}, 'type': 'GlobalTimeoutError', 'error': 504, 'description': 'Timeout exceeded rendering page'}

Issue Analytics

State:
Created 4 years ago
Comments:5

Top GitHub Comments

2reactions

Granitosauruscommented, Nov 19, 2019

I’m having the same issue but weirdly enough only when using proxies via splash:on_request. My splash is patched with decompression patch described in this issue: https://github.com/scrapinghub/splash/issues/324 if you aren’t using proxies this might solve the issue for you.

0reactions

bpgallaghercommented, Jan 2, 2021

I’m having the same issue. Is there a solution for this problem yet? Thank you.