Scrapping break error

I’m using your script and it seems really promising! But unfortunately I’m having some problems with it, and I hope you can help me.

I’m using Ubuntu, I followed the instructions I found in the README.md file but when I run the script with my internet connection - I’m on optical fiber- or through Amazon Web Services hosting I keep getting the same error:

If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py

Of an Instagram account, the script tells me

Getting first 1116 posts but check 1110 posts only

Then the script goes quite slowly, after an hour or so when it reaches the 42nd post It breaks.

Scrolling profile 42 / 1116
breaking in 2 ...
If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py
Scrolling profile 42 / 1116
breaking in 1 ...
If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py
Scrolling profile 42 / 1116
breaking in 0 ...
If you believe this is only caused by slow internet, increase sleep time 'sleep_time_between_post_scroll' in settings.py
Not getting any more posts, ending scrolling

1 / 42
INFO [2018-11-07 16:49:27] Scrapping Post Link:

In the end, I’ve got only the first 42 posts without comments, but I don’t think my line or AWS, both quite fast, may cause the problem. Could you please tell me what do you think is caused?

During the scrapping I also get this error 10 times then it goes in “exit getting comments” :

click button for loading more
error on clicking - next try

Maybe Instagram changed something?

And sometimes I get this other one:

ERROR something went wrong getting user_commented
ERROR [2018-11-07 16:55:58] ERROR something went wrong getting comment 

Thanks in advance, I really hope you can help me.

[justdvl]

Whoever was changing this program before made some bugs. I have fixed them. It works flawlessly for me now. https://github.com/timgrossmann/instagram-profilecrawl/pull/104

[davidetroise]

Dear David, Thanks to your intervention now the script don’t stop anymore at 42 posts, but I’m still having the other problems:

I don’t get the comments’ text and neither the correct number of comments of a single post:

click button for loading more error on clicking - next try (tried: 0)comments:26) click button for loading more error on clicking - next try (tried: 1)comments:26)

INFO [2018-11-13 11:25:17] found comments2 ERROR [2018-11-13 11:25:27] ERROR something went wrong getting user_commented ERROR [2018-11-13 11:25:27] ERROR something went wrong getting comment ERROR [2018-11-13 11:25:38] ERROR something went wrong getting user_commented ERROR [2018-11-13 11:25:38] ERROR something went wrong getting comment INFO [2018-11-13 11:25:38] 0 comments.

I tried with two the different accounts, a bigger one https://www.instagram.com/scottecss/, and also https://www.instagram.com/fibrax/ since it has very few post and comments.

I’m attaching to you the JSON file, the commenters list, and the terminal log of the scottecss account.

scottecss.zip

Thanks for your help, D.