[question] How to reuse existing browser context?

Long into my program, Scrapy is starting at a particular point. I’ve already launched a browser, and logged into an application. I would like to reuse the existing browser context and crawl through the application.

I can pass:

• browser instance: <Browser type=<BrowserType name=chromium executable_path=/.../chrome> version=104.0.5112.20>
• it’s active context: <BrowserContext browser=<Browser type=<BrowserType name=chromium executable_path=/.../chrome> version=104.0.5112.20>>
• and the even Page object: <Page url='https://demo.testfire.net/index.jsp'>

to scrapy-playwright configuration.

meta = {
    'dont_merge_cookies': True,
    'handle_httpstatus_list': [404, 302],
    'playwright': system().lower() in {'linux', 'darwin'},
    'playwright_context': # ???
}

But looking at #supported-settings & #browser-contexts sections in the README, I’m not sure how to put it as a dictionary precisely.

Please help. Thanks!

[elacuesta]

Glad to hear, thanks for the update.

[joe733]

Hi, yes. I resolved it yesterday evening. Here’s what I did:

• Cleaned up (persistent) browser cache before and after each program run, which was causing loading issues.
• Got rid of manual page object management flags, and let the scrapy-playwright library handle it.
• Used the “default” browser context and fed storage_state (had a typo earlier) to it.
• Added HTTP 'referer' as the response.url for request continuity.

For anyone who might stumble upon this. Updated code:

self.rq_meta = {
    'dont_merge_cookies': True,
    'handle_httpstatus_list': [302, 403, 404],
    'playwright': system().lower() in {'linux', 'darwin'},
    'playwright_context_kwargs': {
        'storage_state': kw.get('file_store'),
        'ignore_https_errors': True,
    },
}

scrapy_ps.crawl(
    LarvaeSpider,
    allowed_domains=[n_loc, ],
    start_urls=[s_url, ],
    user_agent=header_['User-Agent'],
    url_dump_path=f'dump/{f_pth}.txt',
    file_store=brw_cxt_store,
)

Key takeaway: Diligently read the docs!

Feel free to (re)close this a complete. Thanks for the help!