Page 1 of 1

Host shut us down for resource abuse - help

Posted: Sat Sep 23, 2006 7:19 am
by laze
Hi -- been loving Serendipity over at dailyping.com since we installed it a few weeks ago to replace our outdated, homegrown solution after our domain was disabled for resource abuse, presumably from a DOS attack.

Well, it's happened again, this time with Serendipity.

Here's what we received:
Greetings,

We have suspended web access for the user: **** on our network today for resource abuse issues on our server: ****. At the time of disabling our servers load average was a very unstable: 13.67. After suspending web access for your account the servers load average dropped quickly to its normal, stable range of 1.0-4.0.

Please work with your script providers and/or website developers to make your scripts more shared-resource server friendly(100% updated, secured, and optimized). Until then your site will continue to have its web access disabled.

Processes snippet:

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
ping 26636 21.0 0.3 61052 14332 ? R 21:23 0:00 /usr/bin/php index.php
ping 26639 15.0 0.1 52164 7992 ? R 21:23 0:00 /usr/bin/php index.php
ping 26640 41.0 0.3 60820 13700 ? R 21:23 0:00 /usr/bin/php index.php
ping 26641 40.0 0.3 60676 13940 ? R 21:23 0:00 /usr/bin/php index.php
ping 26642 34.0 0.3 59512 13268 ? R 21:23 0:00 /usr/bin/php index.php
ping 26643 30.0 0.2 58808 12384 ? R 21:23 0:00 /usr/bin/php index.php
ping 26646 33.0 0.3 58720 13072 ? R 21:23 0:00 /usr/bin/php index.php
ping 26644 33.0 0.3 60936 13296 ? R 21:23 0:00 /usr/bin/php index.php
ping 26647 22.0 0.2 54468 10388 ? R 21:23 0:00 /usr/bin/php index.php
ping 26645 11.0 0.1 52064 6816 ? R 21:23 0:00 /usr/bin/php index.php
ping 26648 23.0 0.2 56852 10588 ? R 21:23 0:00 /usr/bin/php index.php
ping 26651 15.0 0.1 51720 7832 ? R 21:23 0:00 /usr/bin/php index.php

Please advise ASAP, thank you!
Any ideas? I'm afraid if this happens again, they'll shut us down for good (as in: "See ya, now get the hell off of our servers.")

Posted: Sat Sep 23, 2006 3:17 pm
by judebert
We've worked hard to make Serendipity small, fast, and secure. Can you get your access and error logs? We'd like to verify that Serendipity is the problem, as opposed to some other PHP script (known or hacked). If it is Serendipity, we need to know whether there were any errors and what the users were trying to do.

We'd also like to know what plugins you have installed. The more plugins you've got, the more CPU Serendipity needs. 10% of server load is a heckuva lot, though! I've never seen Serendipity do anything like that!

Posted: Sat Sep 23, 2006 3:24 pm
by laze
judebert wrote:We've worked hard to make Serendipity small, fast, and secure. Can you get your access and error logs? We'd like to verify that Serendipity is the problem, as opposed to some other PHP script (known or hacked). If it is Serendipity, we need to know whether there were any errors and what the users were trying to do.
Yes, I can... I can zip them up and send them somewhere... let me know where.

At this point, Serendipity is all that's running on that domain (all other scripts have been ditched).
We'd also like to know what plugins you have installed. The more plugins you've got, the more CPU Serendipity needs. 10% of server load is a heckuva lot, though! I've never seen Serendipity do anything like that!
In addition to the standard batch o' plugins, these are ones we've installed (though all may not be active):

serendipity_plugin_popularentries
serendipity_event_livecomment
serendipity_event_spamblock_surbl
serendipity_event_backup
serendipity_plugin_pollbox
serendipity_event_pollbox
serendipity_event_assigncategories
serendipity_event_categorytemplates
serendipity_event_entrypaging
serendipity_event_staticpage
serendipity_event_contactform
serendipity_plugin_comments

Thanks for your help in tracking down the issue... I'm getting a little annoyed with my host for disabling me like this, but at the same time am wondering what could be causing a CPU spike like that.

Posted: Sat Sep 23, 2006 3:53 pm
by pmcaleer
(I'm Paul, and I work on the Daily Ping with laze as well.)

Out of the plugins listed, here are the ones we're definitely using:

serendipity_event_spamblock_surbl
serendipity_event_assigncategories
serendipity_event_categorytemplates
serendipity_event_entrypaging
serendipity_event_staticpage
serendipity_event_contactform
serendipity_plugin_comments

That's it.

Thanks for your help. (We really do like s9y!)

Posted: Mon Sep 25, 2006 11:16 pm
by judebert
None of them looks particularly determinental.

Sorry I've been gone; weekends are primarily dominated with home and family things. If you email me the logs at s9y.10.judebert@spamgourmet.com, I'll take a look at them.

Posted: Mon Sep 25, 2006 11:19 pm
by laze
judebert wrote:None of them looks particularly determinental.

Sorry I've been gone; weekends are primarily dominated with home and family things. If you email me the logs at s9y.10.judebert@spamgourmet.com, I'll take a look at them.
No problem -- our host re-enabled us because they couldn't really provide us with more information to work with. I'll try to get some logs your way if there looks like there's anything helpful there.

Posted: Thu Sep 28, 2006 4:05 pm
by laze
Damn. We've been disabled *again* for resource abuse. This time, I sent them e-mail immediately requesting access to full error logs.

Posted: Fri Sep 29, 2006 4:11 am
by judebert
Let me know when you get the logs. I'm almost as anxious as you to get this resolved.

Re: Host shut us down for resource abuse - help

Posted: Sun Oct 01, 2006 10:54 pm
by garvinhicking
Hi!

First off, I'm awfully sorry for your trouble. Like Judebert mentioned, we really give our best to optimize Serendipity were possible.

That your web presence is able to take up so much resources IMHO more leads to a general misconfiguration of your webserver. The sysadmins might want to consider if their FastCGI implementation is properly setup, and if Apache is configured with enough RAM. And, most importantly, maybe install a PHP opcode cache which can speed up things.

Usually a shared host should be configured with resource limits that should make it impossible for a single VHost to take up 90% of the CPU time.

Anyways, this does not really lead you to a good conclusion, I guess.

The list of plugins you mentioned is rather "mostly harmless". What could be more of a problem is the amount of your entries. MySQL might choke hard on the list of entries to join DB tables. You might want to check the MySQL setup for enough RAM, up-to-date MySQL 4.1 version with enabled query cache and most importantly that all indexes are set.

One thing that weights REALLY heave on your ressources is that you put the number of entries in your entry listing for each category in your HTML sidebar. That means that all entry related tables need to be counted for every page request. This equals to "hurt me plenty". ;-)

You might want to consider turning this option off. If you rely on it, you might rather want to generally think of some kind of caching mechanism. The "serendipity_event_cachesimple" plugin might be able to help you in that effort. Another idea would be to add a caching layer to the categories plugin with a "timeout" setting - It would take some time, but if you rely on it and don'T want to generally deploy full-page caching (which takes up HDD space!), I would add such a feature.

Now for the more generalistic approach: You must check your page hit statistics. Check which URLs are called how often. If you find out that you get 10 requests/second, it simply means that your page is so popular that your current server might not be able to take that pain.

Another thing might be that you are deeply under spam attacks, this can take up considerable CPU time. Especially if you have the SURBL filter active, which will create one network connection for every incoming comment.

That's it for now, I hope something useful lies in my words. :)

Best regards,
Garvin