alt Hacker NewsGreat feedback, agree I need to filter here. Some website localization is very hard to work around, because they will try to geo-locate the IP address of your bot and redirect it accordingly to a given language...
Great feedback, agree I need to filter here. Some website localization is very hard to work around, because they will try to geo-locate the IP address of your bot and redirect it accordingly to a given language...
The issue I was having was with the query "term+wikipedia" it then shows the wikipedia article in Czech, Hungarian, Russian, some kind of Arab and other before finally showing the English version. Then also a lot of that occur 2,3,4+ times with the same URL, just differing in crawltime by a few minutes.