Workflow will stop if 1 submission page only has gym submissions

How to reproduce:

• Set CF handle to I_love_Hoang_Yen,
• Run harwest codeforces -p 5

What happens: the crawler stop without crawling anything, even though I have 150+ pages of submissions.

I think the reason is because page 5 has only my non-AC or gym submissions. So self.client.get_user_submissions returns an empty array, thus stopping the crawler.

[nileshsah]

Great thinking there @s-i-d-d-i-s! It does seem like a possible idea that we can use. I remember the reason why I first went with the pagination approach of 50 was to keep it in parity with the submissions page on codeforces for easy tracking, though it might not be completely necessary. Let’s take up your approach as a first iteration for dealing with this problem if more people request this feature. Hopefully should not hurt the user experience much.

[s-i-d-d-i-s]

@nileshsah I would suggest increasing the page size to 1000 or some huge number,

I was partly aware of the possibility of this situation arising though thought that cases like these would be fairly uncommon.

i think with such a huge number it would be very unlikely to occur? unless someone did 1000+ gym submission, also that would reduce the number of api calls as well