Serendipity Forums

Hi,

now that my statistics plugin works, I'd like to get rid of all the bots. I put a few of the bots' "hostnames" in the exclusion list, but that doesn't seem to have an effect. I then removed the list again to only contain "msnbot.msn.com" (which was the default after installing the plugin), and since then it doesn't even exclude that one.

What am I doing wrong? Is that a regular expression? If so, should I escape the period? This is a bit hard to test since I don't know when the next bot will come along ...

Thanks,

Robert

Actually, the "hostname" of the bot is not the hostname, but the string of HTTP_USER_AGENT. It is applied as a full string match, not a regular expression.

See line 103 of the serendipity_event_statistics.php plugin, if you care.

Remember that setting this option only affects the tracking of new bots. Old bots that already have been tracked will not be removed from the tracking list.

HTH,
Garvin

Ah, thanks! I guess the default value threw me off. Looks good so far ... ;)

Has anyone made a list of the main bots that I could insert there?

Wikipedia replies: http://en.wikipedia.org/wiki/User_agent#Bots

Has anyone been able to get this feature to work? No matter what string I use, the bots are still being recorded and displayed in the stats page.
I'm using s9y 0.91 with stats plugin 1.23
I'm not php-savvy so haven't accomplished anything by trolling the code.

I've tried the wikipedia strings:

Baiduspider ( http://www.baidu.com/search/spider.htm)|Googlebot/2.1 (+http://www.google.com/bot.html)|Mozilla/5.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html)|Googlebot-Image/1.0| msnbot/1.0 (+http://search.msn.com/msnbot.htm)|Mozilla/5.0 (compatible; Yahoo! Slurp;http://help.yahoo.com/help/us/ysearch/slurp)|Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

as well as the strings the stats plugin spits out:

msnbot/1.0 (+http://search.msn.com/msnbot.htm)|Baiduspider+(+http://www.baidu.com/search/spider.htm)|Mozilla/5.0 (compatible;Googlebot/2.1;+http://www.google.com/bot.html)|Technoratibot/0.7| Googlebot/2.1 (+http://www.google.com/bot.html)|Mozilla/5.0 (compatible; googlebot/2.1; +http://www.google.com/bot.html)|Mozilla/2.0 (compatible; Ask Jeeves/Teoma)

No good either way.

The strings you put in the referrer blocks are interpreted as Regular Expressions. You need to escape all strings that indicate control characters in Regular Expressions in your intput string. Those are, but not limited to:

( ) . +

So your string should look something like:
Read more about regular expressions by searching wikipedia, php.net or google

Regards,
Garvin

Wow, that was fast.

Thanks for the info. That certainly isn't clear through the admin interface.

btw, regex I get (I am a long-time programmer), I just don't do PHP although I'm thinking more and more about changing that.

Uh, hang on. I totally confused this with something different. What I mentioned earlier on is what really applies:

Actually, the "hostname" of the bot is not the hostname, but the string of HTTP_USER_AGENT. It is applied as a full string match, not a regular expression.

See line 103 of the serendipity_event_statistics.php plugin, if you care.

That means, you must specify exactly the string that the Bot submits as its HTTP User agent. You might want to grep your access logfiles to see, if they really submit the HTTP user agents exactly like you entered them?

I'm really sorry for mixing that up.

Regards,
Garvin

If you use FireFox, you can test this by getting the UserAgent extension and setting your user agent to one of the bot agents. Then you can visit your site and see if it's excluing you from the statistics.

maybe a silly idea but would it be possible to use preg_match() to filter out bots ? As i see that bots change hostnames each week nowadays its a endless prayer to just add each new one. Why not filter on search.google.com who cares for the rest. Possibly 'some' normal users wouldnt get through as they have search or bot in theyre referrer, bad luck for them...

option? Im working on the statistics plugin anyway. If possible i could include it right away.

Hi,

OK... I have not much idea about coding or php or anything. I was just wondering if anyone could help me on this issue. I've tried to put the bot's address (which appears on the stats page) on the bit in the plug in i have on the stats but this is not working.

Could someone please explain to me in plain english without too much technical stuff on how I could block the bots from being counted on my stats?

Thanks

Should have been born a blonde,
yati

my blog

Go to the plugin configuration; there you'll find an entry area for referrer blocking.

In that area, enter the exact user agent string of the bot you want to exclude. (Bot user agent strings can be found on Wikipedia.)

Hit save, and you're done.