Expose cookiejars

Scrapy cookiejar API is limited:

• meta key is called cookiejar, but you can’t put CookieJar object there, in fact it means cookiejar_id or session_id, not cookiejar; this is confusing. It should have been called session_id IMHO.
• there is no way to get or set current cookies; it is a popular issue we don’t have a solution for (see http://stackoverflow.com/questions/8708346/access-session-cookie-in-scrapy-spiders and https://github.com/scrapy/scrapy/issues/1448).

I think we should provide a better API for ‘sessions’. It should allow to

1. access current session cookies;
2. ‘fork’ a session - start separate sessions from the current session.

Currently I’m using an ugly hack to access cookies:

class ExposeCookiesMiddleware(CookiesMiddleware):
    """
    This middleware appends CookieJar with current cookies to response flags.

    To use it, disable default CookiesMiddleware and enable
    this middleware instead::

        DOWNLOADER_MIDDLEWARES = {
            'scrapy.downloadermiddlewares.cookies.CookiesMiddleware': None,
            'autologin.middleware.ExposeCookiesMiddleware': 700,
        }

    """
    def process_response(self, request, response, spider):
        response = super(ExposeCookiesMiddleware, self).process_response(
            request, response, spider)
        cookiejarkey = request.meta.get("cookiejar")
        response.flags.append(self.jars[cookiejarkey])
        return response


def get_cookiejar(response):
    for obj in response.flags:
        if isinstance(obj, CookieJar):
            return obj

I don’t have a concrete API proposal, but likely it should use a word ‘session’ 😃

[pawelmhm]

getting and setting cookies in Scrapy is really huge pain so big 👍 from me.

In project I’m working on now we use following solution that sets “jars” from Cookie Middleware on spider, and then allows you to use it.

class CustomCookiesMiddleware(cookies.CookiesMiddleware):
    @classmethod
    def from_crawler(cls, crawler):
        o = super(CustomCookiesMiddleware, cls).from_crawler(crawler)
        crawler.signals.connect(o.spider_opened, signal=signals.spider_opened)
        return o

    def spider_opened(self, spider):
        self.enabled = getattr(spider, 'cookies_enabled', self.enabled)
        spider._cookiejars = self.jars

def BaseSpider(Spider):
    def get_cookie(self, name, cookiejar=None):
           if cookiejar not in self._cookiejars:
                raise KeyError(u'cookiejar {} does not exist'.format(cookiejar))
           _dict = {c.name: c.value for c in self._cookiejars[cookiejar]}
            return _dict.get(name)

but this is just for getting cookies, we dont have anything for setting cookies, we should definitely add something, last time I had to replace cookie value I had to write ugly code like this

locale_cookie = self._cookiejars[None]._cookies[".xbox.com"]["/"].get("defCulture")
locale_cookie.value = self.locale

[kmike]

https://github.com/scrapy/scrapy/pull/3563#issuecomment-505105007 has yet another syntax proposal (haven’t thought about it in depth though).