Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.
And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.
get_posts returns only 20 posts when logged in
See original GitHub issueI am trying to scrap group=120778514747417 which is a public group.
print(len(list(get_posts(group=120778514747417))))
If I don’t run set_cookies first, the scraper works fine but I get banned pretty quickly, probably because I am unlogged.
If I run set_cookies first with my account which is part of this group, scraper only returns 20 posts and stops there.
Logger says the following at the end:
No raw posts (<article> elements) were found in this page.
Page parser did not find next page URL
20
How can I scrap more posts being logged in ? If not possible, what is the recommended throttling to scrap without cookies ? Thanks and great work btw!
Issue Analytics
- State:
- Created 2 years ago
- Comments:14
Top Results From Across the Web
Top Related Medium Post
Top Related StackOverflow Question
Troubleshoot Live Code
Top Related Reddit Thread
Top Related Hackernoon Post
Top Related Tweet
Top Related Dev.to Post
Top Related Hashnode Post
Try load https://m.facebook.com/groups/120778514747417/ in your browser, and scroll down. Posts don’t load there either. This looks like a bug in Facebook
Oh, I’ve just realised - this kind of group pagination URL is already handled by this regex -
href[=:]"(\/groups\/[^"]+bac=[^"]+)". All group pagination should be handled by the GroupPageParser class - given you’ve put it in PageParser, suggests you’re not calling get_posts correctly @chribell. For a group, callget_postslike this:get_posts(group="group_name")