Duplicates crawled data when limit is set

Subject of the issue

There is a fixed number of pages which I crawl, e.g. 67. Each page contains 25 articles, which I crawl, but the last page (67-th) contains 3 articles. The total number of articles is 1653. I want x-ray to crawl articles from all pages by itself, without setting the limit equal to number of pages by myself, so I set the limit to ‘n’ :

function generateURLs(){
  x(url, '.h3 a', [{
    link: '@href'
  }])
    .paginate('.page a@href')
    .limit('n')
    .write('results.json');
}

And in that case x-ray keeps crawling articles from the pages infinitely, producing duplicate data in results.json. Otherwise, if I set limit to 67:

function generateURLs(){
  x(url, '.h3 a', [{
    link: '@href'
  }])
    .paginate('.page a@href')
    .limit(67)
    .write('results.json');
}

it generates 1675 articles instead of 1653. Is there any way to crawl all pages without setting limit? And if limit is set, is there any way to get crawled data without duplicates?

Your environment

• version of node: v4.4.7
• version of npm: 3.3.9

[hielfx]

I was having the same issue when I was using the following pattern:

xray(...)
(function(err,res){
    if(err){
        //stuff here
    }else{
        //more stuff there
    }
})
.paginate(...)
.limit(...)
.write(...);

But if I don’t use the function it works as expected.

[lathropd]

I believe this is a matter of selector handling…