I Am Not A Systems Guy, I Am A Network Guy... so i'll try and not dig myself in too deep. we're also in education, so the network/computing infrastructure is a money sink instead of a money source so double cost for complete redundancy is rarely approved. not that we wouldn't like to, but nobody will shell out the $$ to prevent the loss of services during a downtime (planned or unplanned). we still do increadably well with what we have (>1000 node beowolf cluster, several 15k's, 1G to I2... soon to be 10G to I2 {woot!}, a tape robot bigger than my living room =P), but i'm not too sure how they manage it.

you should get a console concentrator, i can get to the hard serial console of any system from anywhere. =P

but i too tend to rely on filesystem security as about the best that can be done.

Actually, I do lots of work at customers that do shell out the money for redundancy. You often see IT departments selling availability. A nurse in the hospital doesn't really care how it's implemented, but she must have access to a patients medical records at all times. Loss of a connection, and hence requiring logging in again is acceptable. The Oracle service with the records being down for 4 hours because a piece of hardware failed and needed replacement isn't acceptable. That costs money, and perhaps lives. And that's not just hospitals. Banks, government, telcos, energy providers, airlines and others all demand that services are always available.

you should get a console concentrator, i can get to the hard serial console of any system from anywhere. =P

Goodie. You read on your console log you have faulty memory on your mail server, and it refuses to boot. Senior management is pissed. Now what?

Abigail

oh we have spare FRU's for just about if not everything, but the 2x$ for not having to swap something usually isn't there. we tend to be more like an ISP/Colo/Support rather than actual provider (there's nothing stoping anyone from spending the $ if the do need the reliablilty). they get what they're willing to pay for.

mail, dns, and the like are trival to failover somewhere. CPU board won't boot, yank it and let the others take the slack. CRITICAL wierdness, the top level Vendor support will be on the phone in minutes and on site in person with replacement in hours if needed. i've seen it happen more than once.

much worse is when the machine running jobs that take 6 months to over a year to complete has issues. or problems with backup generator (3 days)/power or airconditioning which could comprimise everything. or heaven forbid the metro gigapop burns to the ground ;-). much more troublesome than a 30 minute disruption in something like mail/web/oracle type of failure.

i think i'm trying to say that we spend most of our money on the facility/research/infrastructure side and the application side of things is limited by what the customer (other deparments) is willing to pay for.

or maybe that your dual-everything transparent failover everything will get you naught if somewhere down the line both of your redundant feeds cross through the same manhole over which a gasoline tanker truck has just exploded and your fibers are being fused into a multicolor blob. (much more likely to be a backhoe digging where it shouldn't be digging or rats which have a sweet tooth for fiber cladding). but i ramble...