My first suggestion is to use somethign like log4perl.
Look at your code and look at the log levels.
After knowing what to log, it's a matter of preference from tere on out. Anything that isn't debugging information can usually be logged somewhere permanently, as they shouldn't happen too often. But then again, it depends on hwo you use it. Log4perl provides nice facilities to shoot log messages anywhere. And if you have a real time conifguration option, you can then tweak your app as its running, in case something goes awry.