question-mark
Stuck on an issue?

Lightrun Answers was designed to reduce the constant googling that comes with debugging 3rd party libraries. It collects links to all the places you might be looking at while hunting down a tough bug.

And, if you’re still stuck at the end, we’re happy to hop on a call to see how we can help out.

okhttp/4.9.0 user agent is blocked in potential bot crawling lists

See original GitHub issue

Home Assistant Android version: 2021.1.1-full

Android version: 10

Phone model: OP5T

Home Assistant version: 2021.1.4

Last working Home Assistant release (if known):

Description of problem: The user agent used by the android app (okhttp/4.9.0) seems to be pretty generic and it is being blocked in some blacklists (see https://github.com/JayBizzle/Crawler-Detect/issues/437)

I guess it would be better to modify the user-agent with something less generic to avoid being blocked.

Traceback (if applicable, to get the logs you may refer to: https://companion.home-assistant.io/docs/troubleshooting/faqs/#android-crash-logs):

n/a

Screenshot of problem:

Additional information:

bunkerized-nginx uses these lists, and it blocks access to home-assistant when trying to log in as:

2021/02/16 12:55:07 [warn] 24951#24951: *35 [lua] access_by_lua(main-lua.conf:166):80: [BLOCK] User-Agent okhttp/4.9.0 is blacklisted, client: 10.0.2.100, server: www.example.com, request: "POST /auth/token HTTP/2.0", host: "www.example.com"

Issue Analytics

  • State:closed
  • Created 3 years ago
  • Reactions:1
  • Comments:12 (5 by maintainers)

github_iconTop GitHub Comments

2reactions
dshokouhicommented, Mar 10, 2021

The error is safe to ignore as notifications do not work in the debug build. Thank you for confirming the user agent change and that is working for your setup.

2reactions
dshokouhicommented, Mar 10, 2021

So I had tried to test those above changes however I do not have an environment setup to see the headers in the logs nor do they print to logcat. So I could attempt to make changes and submit a PR if you guys are willing to test the debug APK that gets generated. Would you guys be ok with that? Debug APKs are fine to install side by side the production app and can be safely removed once done testing.

Read more comments on GitHub >

github_iconTop Results From Across the Web

List of Blocked User Agents - Service Centre
To prevent spam traffic and bad bots from using the WISEflow platform and API, the following block list and rules are implemented:.
Read more >
Blocking crawler bots by user-agent - CleanTalk
Methods to block crawling bots using Anti-Spam plugin or editing configuration files for Apache and Nginx.
Read more >
List of most active web crawlers User-Agent strings in 2016
We identified all web crawlers and bots that appeared in our User-Agent-based statistics. Majestic-12 bot was the most active, exceeding the ...
Read more >
Control bots, spiders, and crawlers
Bots, spiders, and other crawlers hitting your dynamic pages can cause extensive resource (memory and CPU) usage. This can lead to high load ......
Read more >
How to Debug Blocked Crawls in Lumar
Stealth mode crawls a website slowly using a large pool of IP addresses and user agents. This typically avoids many types of bot...
Read more >

github_iconTop Related Medium Post

No results found

github_iconTop Related StackOverflow Question

No results found

github_iconTroubleshoot Live Code

Lightrun enables developers to add logs, metrics and snapshots to live code - no restarts or redeploys required.
Start Free

github_iconTop Related Reddit Thread

No results found

github_iconTop Related Hackernoon Post

No results found

github_iconTop Related Tweet

No results found

github_iconTop Related Dev.to Post

No results found

github_iconTop Related Hashnode Post

No results found