in reply to I cancelled my Christmas vacation

Sorry you are missing your vacation. We have had a couple of horrors like that. One was detailed at The 10 stages of Bug hunting and the sweet smell of success.

If that doesn't cheers you up I'll tell you how I spent the first few days of my vacation last week. So I leave the UK with everything humming. Off the plane in Australia at 0130, switch on the phone: 

SMS devel3.xxxxxxx.org has a problem. Please check your email.

[download]

Well a problem was not quite an accurate description. Chernobyl is more like it. Before I left we noted that our new development server only had 1/2 the expected HDD space. It is a RAID V SCSI setup and should have had 143GB available but was only showing 70 odd. This mattered to us as we use this box to stream all our production backups to. So we ask the datacenter to INVESTIGATE why. They decide they have found the problem and decide the redo the containers (of course the take no backups first, nor have we authorised them to change anything). So the short story is that the change to the raid config led to the system rebuilding the containers which apparently crashed at 94%, and the machine refused to boot. In fact it was completely wiped as you expect when you bugger up the raid.

No great problem I think I'll just restore it from image. But you know Murphy. As luck would have it the machine had been killed while it was writing the daily image to remote backup. We only keep a single full image of this machine due to size constraints and the fact that the data is so volatile (yes that policy is about to change!). And that was now >corrupt<

Net result - a hand rebuild from source followed by restoration of the devel environment. Although we didnt have a full image we did have everything needed in the sequenced backups but it all had to be done by hand. Just what you feel like with a serious case of Jetlag on your first days off in a year.

cheers

tachyon

Replies are listed 'Best First'.
Re: Re: I cancelled my Christmas vacation
by pg (Canon) on Dec 22, 2003 at 05:24 UTC

    Roger, revdiablo and tachyon, Thank you so much for trying to cheer me up. As tachyon's reply is the last one at this time, I thought I just attach my reply here.

    Not be able to have my vacation is not something 100% negative:

    • Now I can continue hanging with you guys, if I go on vacation, I will 100% stop using computer, and just pretending that I never heard the concept of computer
    • I can have a long vacation next year, with the add on of what I carried over
    • Luckily I got it resolved, so the bug actually created joy for me, Thanks.

    It is actually an interesting experience, as the place found the bug was not modified, so one can easily miss it during debug. In this case, it is not someone added a piece of buggy code, but missed a place that required modification. Bascially speaking, an old existing piece of perfectly right code, was turned into a bug. The challenge was to find the bug at a place that unlikely, plus the code was (is) very messy.

    I talked with others that have looked at the code. The fact the code was messy largely reduced their ability to debug. The moment they opened up the code, they started to hate it, and I believe a kind of fear started to grow. Code that is less maintainable is not just technically less maintainable, but in fact is psychologically less maintainable.

    Well indent your code! Code that is not nicely indented can scare people away! This is definitely not a small thing.

    Thanks again, guys!

[reply]
[reply]
      Is missing documentation perhaps part of the problem? It looks to me that somewhere someone should have pointed out that parts of the program relied on some fields being unique?

      CountZero

      "If you have four groups working on a compiler, you'll get a 4-pass compiler." - Conway's Law

[reply]
Re: Re: I cancelled my Christmas vacation
by thraxil (Prior) on Dec 22, 2003 at 05:21 UTC

    i love how that stuff always happens when you're on vacation. last christmas, i was off on vacation and the backup hard drive (just an extra big drive that we used for doing additional quick and dirty backups to) in the webserver died and brought the machine down. fixing it was a simple matter of removing the drive and booting in single user mode to manually change some of the services that were hanging because they couldn't find that drive. not a big deal, but a little more difficult when you have to walk a unix-illiterate coworker through the process on the phone.

    the worst part was that right before i left for vacation, we'd gotten a new server. i'd just gotten it configured but didn't dare deploy it right before leaving for vacation.

    then the last time i went on vacation, the entire northeast US had a blackout. we only ever have downtime or hardware problems when i leave. i think the servers miss me when i'm gone and are just acting out in anger.

    i'm really scared because i'm planning on going to japan for christmas and new year's this year.

    anders pearson

[reply]

In Section Meditations