scrapy different with requests? requests work, but scrapy not

1. use requests library, it work.

import requests

headers = {}
headers['Host'] = 'www.bloomberg.com'
headers['User-Agent'] = 'Charles/4.2.1'

response = requests.get('https://www.xxx.com/quote/700:HK', headers=headers

1. When use scrapy, it was block because the Website realize scrapy is a robot. what’s the different with requests?

2018-08-15 16:48:15 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (307) to <GET https://www.xxx.com/tosv2.html?vid=&uuid=f3262880-a067-11e8-922c-0f4bf8a1cda2&url=L3F1b3RlLzcwMDpISw==> from <GET https://www.xxx.com/quote/700:HK>

[wRAR]

@EdgarMagalhaes is it for bloomberg or for some other website? The bloomberg case is special as it’s most likely a sophisticated anti-bot measure.

[wRAR]

Hello @zeroleo12345,thank you for forgetting to remove the website name from the code, it allowed me to play with this for some time. It’s definitely an anti-bot protection but I don’t know why it trips on Scrapy requests more often than on similar curl requests. I say “more often” because I got both 200 with Scrapy and 307 with curl. You should have more luck by setting a real user-agent but it’s definitely not enough. My only guess is that Scrapy does something with HTTPS differently because it seems the only thing that can be different when all headers are the same.