FoursquareException: you are using automation tools to browse the website
See original GitHub issueStarting yesterday, I can no longer retrieve Foursquare data with this library.
client.users.checkins()
is raising a FoursquareException
, which appears to contain HTML instead of a JSON response:
Invalid response: <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Access to this page has been denied.</title> <link href="https://fonts.googleapis.com/css?family=Open+Sans:300" rel="stylesheet"> <style> html, body { margin: 0; padding: 0; font-family: 'Open Sans', sans-serif; color: #000; } a { color: #c5c5c5; text-decoration: none; } .container { align-items: center; display: flex; flex: 1; justify-content: space-between; flex-direction: column; height: 100%; } .container > div { width: 100%; display: flex; justify-content: center; } .container > div > div { display: flex; width: 80%; } .customer-logo-wrapper { padding-top: 2rem; flex-grow: 0; background-color: #fff; visibility: hidden; } .customer-logo { border-bottom: 1px solid #000; } .customer-logo > img { padding-bottom: 1rem; max-height: 50px; max-width: 100%; } .page-title-wrapper { flex-grow: 2; } .page-title { flex-direction: column-reverse; } .content-wrapper { flex-grow: 5; } .content { flex-direction: column; } .page-footer-wrapper { align-items: center; flex-grow: 0.2; background-color: #000; color: #c5c5c5; font-size: 70%; } @media (min-width: 768px) { html, body { height: 100%; } } </style> <!-- Custom CSS --> </head> <body> <section class="container"> <div class="customer-logo-wrapper"> <div class="customer-logo"> <img src="" alt="Logo"/> </div> </div> <div class="page-title-wrapper"> <div class="page-title"> <h1>Please verify you are a human</h1> </div> </div> <div class="content-wrapper"> <div class="content"> <div id="px-captcha"> </div> <p> Access to this page has been denied because we believe you are using automation tools to browse the website. </p> <p> This may happen as a result of the following: </p> <ul> <li> Javascript is disabled or blocked by an extension (ad blockers for example) </li> <li> Your browser does not support cookies </li> </ul> <p> Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading. </p> <p> Reference ID: #4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa </p> </div> </div> <div class="page-footer-wrapper"> <div class="page-footer"> <p> Powered by <a href="https://www.perimeterx.com/whywasiblocked">PerimeterX</a> , Inc. </p> </div> </div> </section> <!-- Px --> <script> window._pxAppId = 'PX0UqK4c76'; window._pxJsClientSrc = '/0UqK4c76/init.js'; window._pxFirstPartyEnabled = true; window._pxVid = ''; window._pxUuid = '4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa'; window._pxHostUrl = '/0UqK4c76/xhr'; </script> <script src="/0UqK4c76/captcha/captcha.js?a=c&u=4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa&v=&m=0"></script> <!-- Custom Script --> </body> </html>
with the relevant part of the message being:
Access to this page has been denied because we believe you are using automation tools to browse the website. This may happen as a result of the following:
- Javascript is disabled or blocked by an extension (ad blockers for example)
- Your browser does not support cookies
Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading.
Is there anything this library can do to mitigate this?
Issue Analytics
- State:
- Created 5 years ago
- Comments:9 (4 by maintainers)
Top Results From Across the Web
Zapier | Automation that moves you forward
Automate your to-do listteam updates. Zapier empowers you to automate your work across 5,000+ apps—so you can move forward, faster.
Read more >How to Automate Website Actions with Browser Automation
Learn how to automate login to a website, click a button, download a file and more with web browser automation. No writing scripts....
Read more >Top 10 Free Automation Tools That Will Make Your Life Easier
The list consists of different testing tools that are used in fields like automation/manual testing, unit testing, performance, web, mobile, etc ...
Read more >The Ultimate List of Sales Automation Tools 2022 - Reply.io
Along with access to leads and their contact details, you'll also be able to use the AI assistant to automate the process.
Read more >Which Automation Tool Do I Use? - Salesforce Help
Salesforce provides a robust set of automation features to help you save time and resources. Use Flow Builder to automate most of your...
Read more >Top Related Medium Post
No results found
Top Related StackOverflow Question
No results found
Troubleshoot Live Code
Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start FreeTop Related Reddit Thread
No results found
Top Related Hackernoon Post
No results found
Top Related Tweet
No results found
Top Related Dev.to Post
No results found
Top Related Hashnode Post
No results found
Top GitHub Comments
I am getting this too all of a sudden. What’s strange is the version I have in PHP is working fine, just the Python returns this long error response.
@diegmonti How did you set the user-agent?
Edit: Got it working using
headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}
Does anyone knows how to solve this issue? I’m getting this error even though I’m only using the API. The foursquare dashboard doesn’t show any error. Note that I don’t even scrape data, I just forward user requests injecting our client id/secret. Strangely if I copy paste my server generated URL in my browser or postman it does work.
Using a user agent doesn’t make any difference in my case.