response.css can't find elements that is there

Hello,

When running scrapy shell https://www.belezanaweb.com.br/homens/

And executing a simple:

In [2]: response.css('div.item').extract()
Out[2]: []

The item is over there:

It isn’t load using JS because it is visible in curl: curl https://www.belezanaweb.com.br/homens/ | less:

Could it be a parser bug? Running on Scrapy1.1.0rc4 Thank you

[eLRuLL]

this is a lxml problem for unclosed tags, but it could be fixed using something like BeautifulSoup:

In [10]: from BeautifulSoup import BeautifulSoup
In [11]: soup = BeautifulSoup(response.body)
In [12]: from scrapy import Selector
In [13]: sel = Selector(text=soup.prettify())
In [14]: sel.css('div.item')
Out[14]: 
[<Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' item ')]" data=u'<div class="item js-item item-variacao" '>,
 <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' item ')]" data=u'<div class="item js-item item-variacao" '>,
 <Selector xpath=u"descendant-or-self::div[@class and contains(concat(' ', normalize-space(@class), ' '), ' item ')]" data=u'<div class="item js-item " data-layer=\'{'>,
...

[rafaelcapucho]

The target HTML have some errors, like some unclosed div tags:

I download the pure html, fixed the tags and now scrapy response.css is processing fine. There are any configuration in the scrapy parser to ignore or auto close those unclosed tags?

Thank you