Trim UA block list for false positives#189

Open

yitzhaq wants to merge 15 commits intomitchellkrogza:masterfrom

yitzhaq:trim_false_positive_uas

Commits on Feb 5, 2023

Remove overly greedy entry "Not", which blocks every phone or tablet in the popular Redmi Note series of phones and tablets, made by Xiaomi: <https://en.wikipedia.org/wiki/Redmi#Redmi_Note_Series>
yitzhaq
committed
Remove "PyCurl" entry, which is just a Python wrapper for cURL, and nothing inherently malicious
yitzhaq
committed
Remove LWP entries - this is just Perl's normal request library
yitzhaq
committed
Remove overly greedy entry "Leap", which blocks every app with "MobileApp" in its UA string
yitzhaq
committed
Remove overly greedy entry "Ninja", which blocks every request from Microsoft Teams. Their agent uses UA strings like this:
yitzhaq
committed
Remove overly greedy entry "oBot", which matches (among any other type of robot) "UptimeRobot", which is supposed to be limited - not blocked
yitzhaq
committed
Remove overly greedy entry WEBDAV, which can (and in many cases will) block legitimate WebDAV clients correctly identifying as such. WebDAV is a very widely used official HTTP standard, and is the …
yitzhaq
committed
Remove "Bullseye" entry. This is the codename for Debian 11 (the current stable release), and the version codename string is used as part of UA for several of its packages, such as Python's pip
yitzhaq
committed
Remove "Siteimprove" - this is a paid service for assessing your site, in terms of features such as accessibility. If you're receiving requests from them, it's likely because you asked (and paid) t…
yitzhaq
committed
Remove "archive.org_bot" - this is Archive.org, a non-profit working to archive digital content and create a library for current and future generations. They are good guys, and if one person is ann…
yitzhaq
committed
Remove "Netcraft" entry. Netcraft is an online intelligence company fighting cybercrime on a plethora of different fronts. When they do mine technical data, it's for a good cause, that benefits all…
yitzhaq
committed
Remove "Shodan" entry. Shodan is an extremely powerful tool for vulnerability scanning. It doesn't do harm, it merely discovers weakness in your setup, which makes it possible for you to fix them. …
yitzhaq
committed
Remove various Chinese entries said to be legit - actual search engine spiders, and embedded views in apps. Closes #133
yitzhaq
committed
Remove "Exabot" entry, this is said to be a good and respectful bot - <https://www.exalead.com/search/webmasterguide#does-the-exalead-robot-respect-the-rules-recorded-in-a-robotstxt-file-and-the-ro…
yitzhaq
committed
Remove "Mojeek" entry, this is said to be a good and respectful bot - <https://www.mojeek.com/bot.html>. Closes #107
yitzhaq
committed