question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

FoursquareException: you are using automation tools to browse the website

See original GitHub issue

Starting yesterday, I can no longer retrieve Foursquare data with this library.

client.users.checkins() is raising a FoursquareException, which appears to contain HTML instead of a JSON response:

Invalid response: <!DOCTYPE html> <html lang="en"> <head> <meta charset="utf-8"> <meta name="viewport" content="width=device-width, initial-scale=1"> <title>Access to this page has been denied.</title> <link href="https://fonts.googleapis.com/css?family=Open+Sans:300" rel="stylesheet"> <style> html, body { margin: 0; padding: 0; font-family: 'Open Sans', sans-serif; color: #000; } a { color: #c5c5c5; text-decoration: none; } .container { align-items: center; display: flex; flex: 1; justify-content: space-between; flex-direction: column; height: 100%; } .container > div { width: 100%; display: flex; justify-content: center; } .container > div > div { display: flex; width: 80%; } .customer-logo-wrapper { padding-top: 2rem; flex-grow: 0; background-color: #fff; visibility: hidden; } .customer-logo { border-bottom: 1px solid #000; } .customer-logo > img { padding-bottom: 1rem; max-height: 50px; max-width: 100%; } .page-title-wrapper { flex-grow: 2; } .page-title { flex-direction: column-reverse; } .content-wrapper { flex-grow: 5; } .content { flex-direction: column; } .page-footer-wrapper { align-items: center; flex-grow: 0.2; background-color: #000; color: #c5c5c5; font-size: 70%; } @media (min-width: 768px) { html, body { height: 100%; } } </style> <!-- Custom CSS -->  </head> <body> <section class="container"> <div class="customer-logo-wrapper"> <div class="customer-logo"> <img src="" alt="Logo"/> </div> </div> <div class="page-title-wrapper"> <div class="page-title"> <h1>Please verify you are a human</h1> </div> </div> <div class="content-wrapper"> <div class="content"> <div id="px-captcha"> </div> <p> Access to this page has been denied because we believe you are using automation tools to browse the website. </p> <p> This may happen as a result of the following: </p> <ul> <li> Javascript is disabled or blocked by an extension (ad blockers for example) </li> <li> Your browser does not support cookies </li> </ul> <p> Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading. </p> <p> Reference ID: #4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa </p> </div> </div> <div class="page-footer-wrapper"> <div class="page-footer"> <p> Powered by <a href="https://www.perimeterx.com/whywasiblocked">PerimeterX</a> , Inc. </p> </div> </div> </section> <!-- Px --> <script> window._pxAppId = 'PX0UqK4c76'; window._pxJsClientSrc = '/0UqK4c76/init.js'; window._pxFirstPartyEnabled = true; window._pxVid = ''; window._pxUuid = '4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa'; window._pxHostUrl = '/0UqK4c76/xhr'; </script> <script src="/0UqK4c76/captcha/captcha.js?a=c&u=4d499fa0-d9ec-11e8-8e00-8d9b5d67cafa&v=&m=0"></script> <!-- Custom Script -->  </body> </html>

with the relevant part of the message being:

Access to this page has been denied because we believe you are using automation tools to browse the website. This may happen as a result of the following:

  • Javascript is disabled or blocked by an extension (ad blockers for example)
  • Your browser does not support cookies

Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading.

Is there anything this library can do to mitigate this?

Issue Analytics

  • State:closed
  • Created 5 years ago
  • Comments:9 (4 by maintainers)

github_iconTop GitHub Comments

1reaction
damageddolphincommented, Oct 30, 2018

I am getting this too all of a sudden. What’s strange is the version I have in PHP is working fine, just the Python returns this long error response.

@diegmonti How did you set the user-agent?

Edit: Got it working using headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'}

0reactions
cglacetcommented, Oct 28, 2021

Does anyone knows how to solve this issue? I’m getting this error even though I’m only using the API. The foursquare dashboard doesn’t show any error. Note that I don’t even scrape data, I just forward user requests injecting our client id/secret. Strangely if I copy paste my server generated URL in my browser or postman it does work.

Using a user agent doesn’t make any difference in my case.

Read more comments on GitHub >

github_iconTop Results From Across the Web

Zapier | Automation that moves you forward
Automate your to-do listteam updates. Zapier empowers you to automate your work across 5,000+ apps—so you can move forward, faster.
Read more >
How to Automate Website Actions with Browser Automation
Learn how to automate login to a website, click a button, download a file and more with web browser automation. No writing scripts....
Read more >
Top 10 Free Automation Tools That Will Make Your Life Easier
The list consists of different testing tools that are used in fields like automation/manual testing, unit testing, performance, web, mobile, etc ...
Read more >
The Ultimate List of Sales Automation Tools 2022 - Reply.io
Along with access to leads and their contact details, you'll also be able to use the AI assistant to automate the process.
Read more >
Which Automation Tool Do I Use? - Salesforce Help
Salesforce provides a robust set of automation features to help you save time and resources. Use Flow Builder to automate most of your...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found