Expecting value: line 1 column 1 (char 0) issue with save_hashtag_video_urls()

Thanks for all your hard work in this module! I’ve been using it to scrape TikTok videos by hashtag for a research study, but today when I try to run save_hashtag_video_urls() with any hashtag, I keep getting the following output:

Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)
Expecting value: line 1 column 1 (char 0)

and it keeps repeating. I’m still able to use the other functions, and I first noticed this issue around 12 pm CDT today. Would this be an issue with the TikTok API changing?

[dfreelon]

OK, I looked into this over the weekend and have a few observations:

• as @stefanuq noted, TikTok has stepped up their security around comment retrieval. Comments are not stored in the page source, and while the JSON object stored inside each video page contain a field called comments, it’s been blank for every video I’ve checked.
• I’ve figured out two ways to pull comments using Selenium-based browser emulation:

1. Pull the comments directly from the emulated source using something like BeautifulSoup, or
2. Capture the relevant XHR request to /api/comment/list/, resend the request with all necessary URL params, and capture the results.

I hesitate to implement anything using Selenium for several reasons:

• Installation is complex and OS-dependent.
• It’s slow.
• Getting more than 20 comments would probably require multiple Selenium requests, which compounds the slowness problem.
• TikTok is apparently releasing a research API next year, so I kinda want to wait and see what it can do before investing lots of time in rigging together a janky comment pipeline only to have it superseded by something much better.

But if anyone figures out anything faster for comments between now and then, LMK and I’ll consider implementing it.

[dfreelon]

@Jmallone I’m willing to wait and see in the short term (also because I have other commitments…) but I will at least fix the search function, likely over the next few days