while not a complete and total solution, keyagent can help out alot. (Warning: IBM Developerworks link might require registration, don't remember but it's worth it if it does.)

keyagent can replace the 'driving to work every night at 2 am to type in the passwords for the nightly scripts that are run.' with 'driving to work to type in the passwords after a nasty reboot.' (but you were on your way in already yes?).

ssh includes built-in support that allows it to communicate with ssh-agent, allowing ssh to acquire your decrypted private keys without prompting you for a password for every single new connection. With ssh-agent you simply use ssh-add to add your private keys to ssh-agent's cache. It's a one-time process; after using ssh-add, ssh will grab your private key from ssh-agent, rather than bugging you by prompting for a passphrase.

the other SSH related articles linked from the keyagent article are well worth a read.

True, it can relief some of the pain. Of course, there's still pain after a reboot - and sometimes, you need passwords at boot time (if one or more boot scripts need them). Also, I presume keyagent isn't storing all its state on disk, which makes it not a solution for cluster environments. Which of course you do you, to minimize downtime in case of failures, don' you? ;-)

It's really a hard issue, and probably not really solvable. Having someone available all the time at the console usually isn't possible. Doing away with passwords isn't the solution either. So, passwords will be have to stored in files sometimes. OTOH, we shouldn't abandon all hope. If you can't rely on file system security, then nothing will be save anyway. /dev and /proc will be great sources of information, and most likely, your write permission won't hold either.

Abigail

I Am Not A Systems Guy, I Am A Network Guy... so i'll try and not dig myself in too deep. we're also in education, so the network/computing infrastructure is a money sink instead of a money source so double cost for complete redundancy is rarely approved. not that we wouldn't like to, but nobody will shell out the $$ to prevent the loss of services during a downtime (planned or unplanned). we still do increadably well with what we have (>1000 node beowolf cluster, several 15k's, 1G to I2... soon to be 10G to I2 {woot!}, a tape robot bigger than my living room =P), but i'm not too sure how they manage it.

you should get a console concentrator, i can get to the hard serial console of any system from anywhere. =P

but i too tend to rely on filesystem security as about the best that can be done.

I left out part of the explanation, sorry. SSH allows you to setup keys which are only able to run a single command which is specified along with the key in the AUTHORIZED_KEYS file. I wrote about it before with some code samples here. You are setting up levels of trust between machines any-way, this way the access is limited and controlled and login access is not allowed. I think a key without a passphrase that can only run one command is somewhat less nasty than having a passphrase/password in plain text which has login access to the remote machine.

No disagreement from me on your other points :)

--
Life is a tale told by an idiot -- full of sound and fury, signifying nothing. William Shakespeare, Macbeth