Site went from <1sec loadtime to >4 minutes! help!

[boycat]

The other day, my s9y installation started loading ABHORRENTLY slow.

At first I thought it was an attack (someone might be pissed off at me) but then on closer look, other flat sites seem to work fine, even under the same domain name.

So i've checked the server -- requests are seen in tcpdump quickly. apache isn't logging the requests until (4 minutes later when) they reach my browser, just as it should. So i focused on postgresql.

looks fine, other databases seem to work well, restarted it just to be sure.

so... now i have no idea what to look at! help!

I pasted this into #s9y a bit earlier:
18:39:48| < boycat> help!
18:39:49| < boycat>
18:40:27| < boycat> 2 days ago, my s9y installation started responding.... reallly.... slow
18:40:35| < boycat> like. boycat.org takes forever to load
18:40:46| < boycat> but boycat.org/other/test.html loads right away
18:41:25| < boycat> apache logs show that the request is serviced after a VERY LONG delay
(sometimes 4 minutes or more) (since apache logs are written when the request
is completed)
18:41:43| < boycat> tcpdump on the server shows that the request comes in quickly
18:41:57| < boycat> and then... not sure if it's postgresql or what... it just... sits.
18:42:05| < boycat> anyone have any ideas what i might look at?

ANY help would be appreciated, as I don't know what to do!

Thanks!
-jre

[boycat]

Nothing on our server changed... but trying to get things back to normal, I changed

"Use persistent connections
Enable the usage of persistent database connections, read more here. This is normally not recommended"

from No to Yes.

and.... it works a lot better.

what the heck does that mean?

I read more "Here" (http://php.net/manual/features.persiste ... ctions.php) but still have no idea why this would have all-of-a-sudden cropped up!

Help!

-jre

[boycat]

SOLUTION FOUND!

Here was the deal;

s9y config was configured to use the database server "localhost"

localhost on the box resolved to 127.0.0.1 -- and ALSO ipv6 ::1.

Recently, there was an ipv6 vulnerability on OpenBSD, and since we weren't using ipv6 on the host, the admin just block-dropped all ipv6 traffic.

So, ever since the time that change was applied, on mar 19th, s9y was trying to talk to localhost, and since ipv6 was configured, it would try that first.

it would have to wait and timeout and try ipv4 for each call before a reply was issued.

This probably has other ramifications on the system that use "localhost" and default to ipv6 first if it is configured. ... but the resolution was to change the s9y config from using "localhost" to using "127.0.0.1" --- just like magic, boycat.org loads quick as hell once again.

dang!

thanks all,

-jre

[garvinhicking]

Hi!

Wow, that was a nifty problem. I guess I wouldn't have solved that at all, so I'm very happy you posted your solution here.

Best regards,
Garvin

[d_cee]

Hi

could somebody explain what this means as it sounds very similar to the problem I had - see http://board.s9y.org/viewtopic.php?t=9298

thanks

Dave

[garvinhicking]

Hi David!

Just check what you configured as your database host in the s9y configuration? Make sure it doesn't say 'localhost' but instead some IP address or domainname.

Regards,
Garvin

[judebert]

In fact, you might want to stick with an IPv4 address, as amy domain name could be translated to IPv6.

I think. Boy, I'm rusty.