Comment stream dropping comments?

Issue Description

A simple consumer like the below seems to not be processing all comments.

I observed a drop in total comment throughput with the praw stream sometime around december 2018 and it has never really recovered.

I have tested by manually making a number of comments and observing that some of them don’t get captured by the praw stream.

IO on the client side is not a limiting factor.

for comment in reddit.subreddit('all').stream.comments():
   do_something(comment)

System Information

• PRAW Version: 6.0.0
• Python Version: 3.7
• Operating System:

[Pyprohly]

I think removing the before adjusting would be an acceptable fix for now, although know that everyone who’s not streaming from r/all is going to be slightly worse off, which makes me a little bit uncomfortable.

But it’s not like any other reddit api library does any sort of fancy before param adjusting either.

I’ll reintroduce before adjusting with a more optimal algorithm by the time I’m though with #1025. (My new streaming implementation already tries to detect the target listing’s activity and adjusts before accordingly, and in a future edit I’ll get it choose to keep using a None value for active listings like r/all.)

[nmtake]

I tried this script

import logging
import sys
import praw

logging.basicConfig(
        level=logging.DEBUG, stream=sys.stdout,
        format='%(asctime)s %(message)s')

reddit = praw.Reddit(...)
for comment in reddit.subreddit('all').stream.comments():
    print(comment.id)

And here is the result log. As we can see, there are many dropped comments as @bicubic pointed out, and some of them can be retrieved later via api/info:

>>> import praw
>>> reddit = praw.Reddit(...)
>>> dropped = ['t1_eivp01' + c for c in 'pquvxyz']
>>> for comment in reddit.info(dropped):
...     print(comment.id, comment.author)
...
eivp01p InnerRisk
eivp01q Skewered_Planets
eivp01u None
eivp01v Cirkah
eivp01x Chishikii
eivp01y sowaffled
eivp01z None

I too am guessing it’s a cache problem (rather than private subreddit or unapproved comment). As the second GET /comments request in the log suggests (it returns only 9 comments and too many holes), PRAW is requesting too fast?