Coupling random user_agent (scrapy_fake_useragent) extension with cfscrape

I try to make a scrapy script which use cfscrape, privoxy, and scrapy_fake_useragent, but i have some difficulties to understand how i can pass the random user_agent generated by scrapy_fake_useragent extension to your great cfscrape extension 😃

I crosspost this issue/question on scrapy-fake-useragent issue and stackOverflow with more explanation because i think other person can be interested by this coupling :

• http://stackoverflow.com/questions/41589391/share-user-agent-between-scrapy-fake-useragent-and-cfscrape-scrapy-extension

• https://github.com/alecxe/scrapy-fake-useragent/issues/9

[reyman]

It’s not the best solution, but it’s work like that :

from fake_useragent import UserAgent
   
class AirportsSpider(scrapy.Spider):
    name = "airports"
    start_urls = ['https://...']
    allowed_domains = ['...']
    ua = UserAgent()

    ...
    def start_requests(self):
        cf_requests = []
        user_agent = self.ua.random
        self.logger.info("RANDOM user_agent = %s", user_agent)
        for url in self.start_urls:
            token , agent = cfscrape.get_tokens(url,user_agent)
            self.logger.info("token = %s", token)
            self.logger.info("agent = %s", agent)

            cf_requests.append(scrapy.Request(url=url,
                                              cookies= token,
                                              headers={'User-Agent': agent}))
        return cf_requests

[reyman]

Thanks @Xonshiz it’s seems this is a good alternative solution, but i need to rewrite all my scrapy code i suppose 😕 And i lost the capacity offered by scrapy_fake_useragent. If ask the question to dev of the extension to know if it’s possible to catch the random user_agent value generated before using it into cfscrape on project issue page.