Performance! How to make it better?

[FishNiX]

Warning: long winded post below

Running a 2GB host @rackspace cloud --
Apache 2.2.x
Mysql 5.1.x ( ~82% hit ratio on query cache)
PHP 5.1.x
APC opcode caching (~100% hit ratio)
Memcache for php sessions
S9y v1.6.0
Extended entry properties plugin with caching

mysql> select count(*) from serendipity_entries;
+----------+
| count(*) |
+----------+
| 4310 |
+----------+
1 row in set (0.00 sec)

Monitoring shows an average of ~40 queries/second on the database (~32 of which are from query cache). Apache accesses are in the range of 12 - 20 req/second. I start 30 apache processes with min spare of 5 and max spare of 25. Keepalives are on, but timeout is very low. Avg busy processes is just a few more than req/sec.


Currently all of these things run on the same machine with negligible I/O wait (<0.8%). My load is high-ish >0.85 and mysql consumes a lot of CPU. When tracking this down (via newrelic), the largest consumer by far is SELECT actions on serendipity_entries from index.php. (I can't get more granular than that without a premium account). I assume this is all a result of cached entries being pulled for each page view. Latency has been creeping up and I'm currently operating under the thought that it's CPU contention. Our ad serving plays a part as well, but it's easier to yell at vendors when everything else is fast.

I'm thinking about two potential solutions... using memcache to cache entries from serendipity_entries and (maybe simplier) a varnish cache running in front of serendipity. The one blocker I see to that is the pragma: nocache delivered by index.php. It's a possibility to split mysql off to it's own server (I've run this way in the past actually), but I'm trying to avoid the cost of a second machine.

Thoughts? Other ideas?

As an aside:
A while back I started writing a plugin for AWS (s3 in particular), but that won't fix this particular issue. I was doing that with the idea that separating media from the install tree might allow me to run many load balanced instances of s9y without shared storage. It would also let me spin up and spin down instances at will.

https://github.com/fishnix/serendipity_event_aws

[garvinhicking]

Hi!

Okay, that definitely looks like a higher traffic blog. Basically, yes, what's killing you is that on each pagerequest s9y compiles the list of entries from the SQL. With the serendipity_event_simplecache we tried to rectify some of that, but the plugin itself is low-maintained and could certainyl be much improved.

I once tried a "Serendipity-Powercache" variant (on github), however that never got around to any feedback.

If you have mostly only reading access with little user and editor participation, you might want to drive a few tests with SQLite, which performs better with READs; however, depending on which plugins you use, joining many tables could neglect that performance advantage.

But now regarding your general suggestion: Varnish or some other kind of forward-proxying or pseudo-loadbalancing is definitely a good way to go. I myself do not have expert knowledge with varnish, but from what I heard both that or squid should provide good results, especially if you have mostly read-access to the blog.

The "nocache" pragma could IMHO be removed in such a case; it's only there for global installations to ensure that people always get latest, dynamical output. If you can live with a non-individual cache then you don't need this. You will definitely have a problem though, if you have blog entries only visisble to certain author groups, or provide any other kind of user-specific content depending on the loginstate.

Best regards,
Garvin

[abdussamad]

Why is load of 0.85 considered high? Even if you only have one CPU core a load of 0.85 isn't particularly high for a Linux web server. You can easily handle a load of 2x CPU core count. What's the CPU idle percentage?

Are you using apache prefork or worker? Most likely it's the former but you can confirm by doing an "apache2ctl -V" or "httpd -V" (CentOS). I'd like to see your apache configuration for MaxClients, MaxKeepAliveRequests, KeepAliveTimeout and MaxRequestsPerChild? Also how much memory usage are you seeing at peak times. Let's see the output of "free -m"?

Do mention the linux disto you are using as well?

[jakky]

Just my two cents. I'd really take varnish cache, as it would not only improve performance but also free resources. Depending on mashine resources you'd get really serious enhancement on the page delivery. There are already some tries to put s9 behind varnish, just like this guy:
http://belsky.info/archives/32-Serendip ... rnish.html
Some serendipity additions might be needed, if you need get cache be automatically updated.