question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Best way to get new (comments and reactors) after caching data

See original GitHub issue

Hi. Not an issue but just like a question. Considering a scenario where i want to get comments and reactors from a post and keep them updated. I did a file cache to save updated data. Which is the best way to get new post comments and reactors while improving speed? does numbering get items from newer to older? Any other trick on this?

Many thanks Example: options={"reactors": 20, "comments": 20, "allow_extra_requests": False}))

Issue Analytics

  • State:open
  • Created 2 years ago
  • Comments:14

github_iconTop GitHub Comments

1reaction
neon-ninjacommented, Oct 7, 2021

https://github.com/kevinzg/facebook-scraper/commit/201a6681189cb707c8cbcfddebbe467d9f7d8c20 refactors reactor fetching a bit so that it becomes possible to get reactors as a generator (it was already possible to get comments as a generator). Then, the last part of the code could be like so, if you prefer:

post = next(get_posts(
    post_urls=[post_id],
    cookies="cookies.json",
    options={
        "allow_extra_requests": False,
        "reactions": True,
        "comments": "generator",
        "reactors": "generator",
        "comment_reactors": False,
    },
))
print(
    f'Live version of post has {post["comments"]} comments and {post["reaction_count"]} reactions'
)
if post["comments"] > prev_post.get("comments", 0) or post["reaction_count"] > prev_post.get("reaction_count", 0):
    print("Post has new data, fetching comments and reactors")

    post["comments_full"] = utils.safe_consume(post["comments_full"])
    for comment in post["comments_full"]:
        comment["replies"] = utils.safe_consume(comment["replies"])
    post["reactors"] = utils.safe_consume(post["reactors"])
    
    with open(post_id + "_new.json", "w") as f:
        json.dump(post, f, default=str, indent=4)
0reactions
netquikcommented, Oct 7, 2021

Yes ok already doing that. Thanks for all!

Read more comments on GitHub >

github_iconTop Results From Across the Web

Spring Reactor how to cache once and distribute response to ...
My application scrapes the currency data from a website for each 10 seconds. I would like to cache the example currency data below:....
Read more >
Clever cache for Reactor's Mono objects - Java Code Geeks
Check our article explaining how to Clever cache for Reactor's Mono objects. ... Data caching is a widespread technique in the programming.
Read more >
Add upper limit time to Mono#cache(Function ... - GitHub
i.e. you are not likely to cache a ton of different sources, but rather cache once and reuse the cached Mono . Since...
Read more >
Caching strategies to speed up your API - LogRocket Blog
First, check whether data exists in the cache. If it does, read from the cache and send the response. If it doesn't, the...
Read more >
Simple caching with local storage - Level Up Coding
Our getFreshData() function now takes a url and a cacheResponse boolean. The url is used to request our data from the API, and...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found