Well, I've found that no matter how much planning you do, there's always something left to go wrong. I also don't know what you're using for your monitoring, but some of them have a way to keep them from sending out alerts, so that when you know something's down, it won't keep alerting. (A couple of the folks I used to work with were contemplating on making something that would only allow you to set a timer to supress the warnings, and would then escalate if the problem wasn't resolved in time.... after we found out someone had disabled the warnings on their systems, because they didn't like getting paged 'all the time'.)

Depending on how the paging goes out, if it's through a mail gateway, you might also be able to shut down the local mailqueues, if it doesn't send straight to another SMTP server. (fortunately, I have the luxury of being able to sit and think about these things, as opposed to when you've been up all night, and you're hitting the end of your planned outage window, and all you want to do is get out of there, and crash for the night)

Anyway, I find you learn best from doing things. (and I learn the most from making mistakes -- as I don't want to repeat them). No matter how much you read, take classes, or plan for things, it's nothing like the real thing, when you're pulling 16hr days for 2 weeks straight, trying to recover a 30k user mailstore, watching three fibrechannel controllers that you connect to mysteriously fail one after another. Or when you power down your entire 100+ server data center (so they could install a UPS bypass switch...yet we had to shut down the next year for the batteries to be serviced), only to have your terminal server not come back up, so you're rolling two WSYE terms about on carts as you bring the machines up two at a time.

In reply to Re^5: Reinventing the wheel by jhourcle
in thread Reinventing the wheel by bageler

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.