bindaddress of meta never work right.

I have multi local IPs with different outgoing port to internet. simply:

        yield scrapy.Request('http://httpbin.org/ip', self.parse3, meta={'bindaddress': ('192.168.0.2', 0)}, dont_filter=True)
        yield scrapy.Request('http://httpbin.org/ip', self.parse3, meta={'bindaddress': ('192.168.0.3', 0)}, dont_filter=True)

responsed same result.

ps: already verified the two bind-address works using curl.

does anyone know how to fix and be willing to fix it?

[tarnenok]

@NewUserHa, as @alwaysused mentioned above, twisted caches connections by the key. That key is composed in line and don’t take into account bind address property. So, for the second link in your example it uses cached connection with this meta={'bindaddress': ('192.168.0.2', 0)}

Your example with empty custom_settings works properly, because twisted caches connection in line after http response is received. So If there is no defined DOWNLOAD_DELAY or it’s smaller than a server time response, than the second request starts earlier then connection from the first request is cached. Consequntly it creates a new connection.

To fix this problem you can patch connection key composed in twisted in line to the following

key = (parsedURI.scheme, parsedURI.host, parsedURI.port, endpoint._bindAddress)

But it access the private member and violates incapsulation properties. If this functionality is relly needed by community it worths to think better and maybe update ScrapyAgent to take into account bindAddress.

[alwaysused]

sorry for replying so late, that was midnight in my country and i got to sleep. About the debug are you using an ide for python? I’m using pycharm and the library can be found in the project sidebar, and if you are not using this you can google or just search it in the ide’s website.The ‘twisted’ network library is a pure python project instead a c extesion so debugging the library is possible.