When posting, server load goes thru the clouds

[mdnava]

Everytime I post in Serendipity the server load goes to the clouds, from an average of 1.5 and 4 in peak hours it goes to over 15 and as far as 50 on which the server stops responding.

I own a dedicated FreeBSD server with dual Opterons, 4Gb of ram and IDE hard drives... I'm running the latest software of s9y, cache simple and freetag plugin, latest apache 1, latest php 4 and mysql 4.0

I believe this has something to do with the Freetag plugin which I use to show related entries, when I don't add tags the load doesn't go that high.

Any ideas? anyone is having the same issue?

[garvinhicking]

Hi!

The cache simple plugin could be responsible, because it erases the cache of all files when you post a new article.

Can you see how it behaves without that caching plugin?

Freetag should be unrelated. Other plugins that can likely cause this are XMLRPC / Pinging, but since you didn't list those, I assume you don't have them?

Enable your Apache server-status URL and check that page when you post an article to see with which URLs the children are busy.

HTH,
Garvin

[mdnava]

Hello Garvin,

The cache simple plugin could be responsible, because it erases the cache of all files when you post a new article.

Woah!!... I have over 2200 articles, cache is set to expire after 6 hours, but if that's true it could explain it... I'm now worried about what will happen once we reach twice many articles.

Can you see how it behaves without that caching plugin?

I'd have to try on rush hours since that's when posting becomes an issue, but once I remove that plugin at that time the server goes down in seconds flooded by mysql... At this time for example, I can post without problems, the load doesn't go that high...

Freetag should be unrelated. Other plugins that can likely cause this are XMLRPC / Pinging, but since you didn't list those, I assume you don't have them?

The freetag comment was based only on perception, I do have the ping-services plugin installed but even if I don't ping any service the load goes to the clouds, and I have tried removing this and many other plugins in the past without success...

********
The XMLRPC plugin was installed only a few days ago to make use of the Live Writer software, but it also doesn't work as expected (using MT API), it returns some errors (timeout, error 104) and "sometimes" it doesn't publish entries, but if Tags are set on the post the DO NOT POST rule is guaranteed, in this case Apache returns a "wlwmanifest.xml not found".

[garvinhicking]

Hi!

Yeah, sadly the plugin is really only a "simple" cache -- not a "powerful" cache.

In fact, a better caching plugin for s9y definitely needs to be coded. Sadly my experience with caching possibilities is limited and I'd need help and input from other developers.

When you post a problematic entry, does it maybe have many outgoing links? It could be that the trackback checking is behaving irradically on your blog.

But my bet is the caching plugin, I'm quite sure it's the cause for your publishing problem.

Regards,
Garvin

[abdussamad]

You can use squid to reverse proxy apache and reduce the load on your mysql servers. Squid can be configured to cache your s9y blog posts and other static content. I have written a basic guide on how to do this with squid 2.5:

http://abdussamad.com/archives/121-Squi ... s-4.5.html

You can use the refresh_pattern option to enable caching of s9y blog posts while preventing caching of dynamic pages like search results:

refresh_pattern archivos 60 25% 120 override-expire

More info about refresh_pattern:

http://www.squid-cache.org/Versions/v2/ ... ttern.html
http://www.linuxdevcenter.com/pub/a/lin ... tml?page=2

I think this is a better solution than php based caching because you are using a program that is designed for caching.

[mdnava]

Garvin,

Wouldn't it be more efficient to include caching on the core of s9y than loading a plugin first, doesn't it have some overhead?

Also, I've been reading about "memcache" and looks promising (http://www.danga.com/memcached/) but it depends on having a third party app installed on the server, which I would do, but it may not be suitable for most Blogs.

I'm thinking about giving Squid a try, as recommended by abdussamad... My server load keeps putting pressure.

Have a nice day!

[garvinhicking]

Hi!

Wouldn't it be more efficient to include caching on the core of s9y than loading a plugin first, doesn't it have some overhead?

In fact, caching requires the plugin API, so including it in the core would make things both ugly and complicated and redundant.

The current plugin is not a final, good solution. The whole system needs to be approached from a different angle, I believe.

Best regards,
Garvin