question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

Using YouTube json instead of scraping

See original GitHub issue

https://m.youtube.com/results?search_query=[query]&app=m returns search results in json. Maybe using those results would be better than using bs4 (or regex) on the HTML results returned for a normal search (https://www.youtube.com/results?search_query=[query]&app=desktop).

Pagination might be an issue - how to get the next page for m.youtube.com results?

Issue Analytics

  • State:closed
  • Created 4 years ago
  • Comments:8 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
SuperSandro2000commented, Jul 8, 2020

You should really think about using youtube-dl because it removes a lot of maintenance work keeping up with youtube.

1reaction
BrainDamagecommented, Jan 25, 2020

Why not rely on youtube-dl to get search results? 'ytsearch:query" gives a list of results in youtube-dl In principle if you fully exploit youtube-dl, you’d be able to support any side that the script does, without having to code them yourself. A quick search in youtube-dl gives:

youtube-dl --list-extractors | grep search
mailru:music:search
screen.yahoo:search
soundcloud:search
video.google:search
youtube:search
youtube:search:date
youtube:search_url
Read more comments on GitHub >

github_iconTop Results From Across the Web

Scraping data from a website into JSON - YouTube
Download ParseHub for Free: https://bit.ly/3np4b8gToday, we'll show you how you can easily scrape data from any website and store the data ...
Read more >
Hidden APIs with Scrapy - easy JSON data extraction - YouTube
I've shown this web scraping method before but never using Scrapy, and given that the Scrapy framework gives us some reaslly good features...
Read more >
Python Project to Scrape YouTube using YouTube Data API
In this video, we build a Python Project to Scrape YouTube data using YouTube Data API. Using YouTube API, we extract the data...
Read more >
How to extract from JSON Links - YouTube
Octoparse is a SIMPLE but POWERFUL web scraping tool for harvesting structured information and specific data types related to the keywords ...
Read more >
Python Web Scraping: JSON in SCRIPT tags - YouTube
This video covers a simple and easy way to web scrape with python by getting the data out in JSON format from the...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found