![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
ramtopswhat a bloody horrible day ...
our main webserver, offler, has been running out of processes from time to time, due to what we think are random syn bombs (Denial of Service attacks). So yesterday,![[livejournal.com profile]](https://www.dreamwidth.org/img/external/lj-userinfo.gif)
perlmonger compiled and installed the latest version of Apache, which is supposed to fix this. But it didn't.
so he installed one copy of Apache on each IP address, on the basis that it would help. Then, all of a sudden, the server kept running out of file handles, and grinding to a halt. We still don't know why this was happening ...
we decided to switch the web sites to fhez, our backup server. This involves running a couple of scripts, bring up ethernet interface with IP addresses, etc., and is generally pretty painless. Unless - as happened today - you lose *all* your bloody internet connectivity in the middle of running the scripts.
yes - we lost both Blueyonder cable _and_ ADSL. We still don't know what happened there, either. Whether the dual WAN router has thrown a wobbly or what, we are currently running on ADSL only, as we simply don't have the energy or brains to investigate further tonight. It tok us hours to get even that running, because all the config details were - yes, you guessed - in both IMAP e-mail, and the Wiki, on offler and fhez. And, would you believe it, our ISP was suffering a power cut, so we couldn't get through to them either. Eventually, with a flash of belated genius, I packed Pete off, with the Powerbook, to a friend up road who is DSL-enabled, and he SSH'd in from there to get first fhez, then offler, back up. But we had over three hours downtime for the public facing servers, which is not good at all :(
I've had enough for one day.
anyway, new GIP - Hommage, by Leopoldo Maler, from the Hess Collection in Napa Valley, California. I can see a lot of use for this one, can't you?
our main webserver, offler, has been running out of processes from time to time, due to what we think are random syn bombs (Denial of Service attacks). So yesterday,
perlmonger compiled and installed the latest version of Apache, which is supposed to fix this. But it didn't.so he installed one copy of Apache on each IP address, on the basis that it would help. Then, all of a sudden, the server kept running out of file handles, and grinding to a halt. We still don't know why this was happening ...
we decided to switch the web sites to fhez, our backup server. This involves running a couple of scripts, bring up ethernet interface with IP addresses, etc., and is generally pretty painless. Unless - as happened today - you lose *all* your bloody internet connectivity in the middle of running the scripts.
yes - we lost both Blueyonder cable _and_ ADSL. We still don't know what happened there, either. Whether the dual WAN router has thrown a wobbly or what, we are currently running on ADSL only, as we simply don't have the energy or brains to investigate further tonight. It tok us hours to get even that running, because all the config details were - yes, you guessed - in both IMAP e-mail, and the Wiki, on offler and fhez. And, would you believe it, our ISP was suffering a power cut, so we couldn't get through to them either. Eventually, with a flash of belated genius, I packed Pete off, with the Powerbook, to a friend up road who is DSL-enabled, and he SSH'd in from there to get first fhez, then offler, back up. But we had over three hours downtime for the public facing servers, which is not good at all :(
I've had enough for one day.
anyway, new GIP - Hommage, by Leopoldo Maler, from the Hess Collection in Napa Valley, California. I can see a lot of use for this one, can't you?
![[identity profile]](https://www.dreamwidth.org/img/silk/identity/openid.png){kind=small}
(no subject)
Date: 2004-11-04 03:15 pm (UTC)