logoalt Hacker News

Bendertoday at 3:21 PM1 replyview on HN

Seems about right. I rotated my logs this morning. Most humans go to access.log and most bots go to botpoop.log. This is the line count:

     2 access.log [1]
    40 botpoop.log [2]
2 is really 1 since a human will grab the CSS file. Most bots do not bother with the style-sheet so that's a 40:1 bots to humans. I could cut that down by blocking data-centers but then I inadvertently block a lot of VPN's which I really don't need to do for a static compressed blog served from ram. The bots just get a TCP Reset but it's still fun to log and study them. The most interesting one I've seen recently is ReadYou which may be a reader but it appears to be much more, possibly acting as a cell phone distributed bot collecting data for a centralized site.

[1] - https://nochan.net/logs/access.log

[2] - https://nochan.net/logs/botpoop.log


Replies

spockztoday at 3:44 PM

Is there some existing mechanism already that counts how often an ip only scrapes the page and not the css and then block those origin IPs if it occurs “too often”?

Unfortunately, the best practice is to make css cacheable so you need to keep long histories.

show 1 reply