Add scrapeIt.fromStream function

What do you think of a fromStream function which feeds the HTML parser chunk by chunk instead of allocating one big string in memory?

I have already implemented it and am using it in one of my projects: https://github.com/IonicaBizau/scrape-it/compare/master...ComFreek:master.

I also have a unit test at hand, which requires an upgrade of the package tester, as already mentioned at https://github.com/IonicaBizau/tester/issues/15.

t.it("scrape from stream", cb => {
    return t.expect(http.get(URL, response => {
        scrapeIt.fromStream(response, {
            title: "h1.title"
          , description: ".description"
          , date: {
                selector: ".date"
              , convert: x => new Date(x)
            }
        })
    })).resolves.toEqual({
        title: "Title"
      , description: "Lorem ipsum"
      , date: new Date("1988-01-01")
    })
})

Open points:

• The normal scrapeIt method uses cheerio-req, which actually could benefit from the chunk by chunk loading as well.
• Investigate how htmlparser2 deals with encodings.
• [from a later comment] Wait for Cheerio adopting/discussing the Cheerio.load(dom) public API: https://github.com/cheeriojs/cheerio/issues/1126

[ComFreek]

My code only stores the DOM tree in memory, not the HTML string. I think the DOM is necessary for CSS-like selectors in scrape-it anyway.*

*) Actually, this exercise seems quite interesting: “Given a set of CSS selectors, does a program exist, which extracts the results from sequential HTML parsing?” or “Can we pre-compile a set of selectors, so that SAX-like parsing suffices to extract all the matches?”

[IonicaBizau]

I am going to close this, but contributions are welcome via PRs! 🚀