A few years back, I had the wonderful opportunity to be employed by Bob Bruce of Walnut Creek as a paid FreeBSD advocate. I lived in New Mexico at the time, but I did have a chance to visit their operation and meet many of the California-based FreeBSD core team members. I later moved on to doing custom chip design as a consultant, but I kept tabs on a lot of the mailing list discussions for quite a while.

At that time, WC-CDROM provided a comfortable home for many of the core team members, including the charismatic team leader, Jordan Hubbard. There were a great many people sharing load to get the releases out the door, but my impression was that there was also an unspoken structure of 'would Jordan approve?' in the way things were done. A while after I left, he became part of the Apple team to help them carry Darwin into Tiger and beyond. On top of that, the role of Walnut Creek changed and many of the other people found that the future wasn't safe or as comfortably predictable.

The core team screamed a collective 'oh sh!t!' that lasted about a week, but quickly rallied and got together to get professional in their methodologies for the release process. The whole FreeBSD engineering process was transformed over the course of a year. Not only do they now have processes in operation that have been hammered very deliberately into place, but they also have methodologies for handling changes in the workers and their level of capabilities. Those new structures have since survived many upheavals including an attempt by a payware company to buy and bury, changes in the core personnel and their roles and time commitments, and increasing pressure to 'keep up' with the perceived progress of Linux.

My point in this meditation -- as I'm interviewing new people for Sandia and making decisions about my outside business and its team -- is to consider our organization and its people, and the outside organizations I depend upon for my software, in the light of their survival characteristics. Sandia has a requirement that we make sure that our projects can be recreated thirty years from now and their data be recoverable. The nature of much Net-based open source development and payware business works against that goal. For an example, could you recover data from a tape made on a Colorado Backup portable drive that had only Windows 95 drivers ever written? Likewise, do your systems people document the changes they make to the configuration files and scripts of your systems as they make them? Somewhere other than on the system in question? ;-]

Most importantly, how much knowledge is embedded in the memory and experience of the one guy whose pager is always going off and who's getting tired of working 80 hour weeks. How many fixes were done by the one guy who's on vacation and unreachable until October? Where is your backup backed up? Are your CVS deltas documented? How many fixes have been done by a binary patch and forgotten because some manager never approved the time to go back to fix the software that produced the corrupt and unusable data files?

This meditation isn't specifically about Perl, but it's as important to a Perl team as use strict; and use warnings;.

Replies are listed 'Best First'.
Re: Teams, Personalities, and Getting the Job Done
by Tanktalus (Canon) on Sep 15, 2005 at 15:04 UTC

    30 year recreation timeframe? Sorry, but open-source gives you a way better chance at that than closed-source. There is no way your company could afford to pay Microsoft to dredge up the Win95 code in 2025. Or IBM for OS/2 or AIX or OS/400 or... Or Sun for Solaris/SunOS. Or HP for HP-UX. Or ... Open source? You absolutely must take the snapshot of the code you're using, and if it doesn't work with your RedHat Linux version 34, you at least have a reasonably-priced chance at getting it to work - hire a contractor at $100-200K/year (in today's dollars), and s/he should have it working in 1-6 months.

    Oh, but the data format has changed 15 times since you took that backup? With open-source, you have the original code, and the current code, and that contractor has a reasonable chance of being able to map constructs from one to the other, and building a migration tool.

    That said, 30 years seems like an awfully long time. I'm not sure whether your management has decided on this time frame because it sounds good, or because there's some actual practical or legal reason to do so. Even most financial audits don't go back more than 7 years... and 30 years from now, no one will even know what to look for that we may be doing today. All the documentation will be long lost, buried somewhere in backups that no one knows how to find anyway. Heck, I have a hard time figuring out what happened 10 years ago in my project (I joined this company/project 8 years ago), and it's still all online.

[reply]
      Absolutely right, Tanktalus. And that's not even considering that nobody'd have a Centronics parallel port on their P999. We have one whole guy who spends his time managing all the old CAD system setups so that they either have their original OS or that they work on a newer variant.

      In our case, these are mandates from On Very High, i.e., DoD/Congress etc. I'm not saying that these are a good thing, or that they should be applied everywhere; it's just what I have to operate with.

      This survival-think is pushing us to open source in many ways, although there's a counterforce from Congress that is pressing us to put government purchase dollars into the private sector. More and more of our project and line managers are willing to look at the long term costs of closed source seats and hardware versus commodities and programmer time. I am a perfect example of the result of that new assessment; my whole tasking is using open source to solve problems without $$$.
[reply]
        This survival-think is pushing us to open source in many ways, although there's a counterforce from Congress that is pressing us to put government purchase dollars into the private sector.

        Please note that "open source" and "the private sector" are not mutually exclusive. In the context of the greater thread, you might be using these terms for convenience, to represent ideas you have already been talking about. But I think we should be careful to point out that the ideas you are talking about do not necessarily generalize; open source licensing and private sector dollars are perfectly willing to commingle, and are doing so in fact more and more.

[reply]
Re: Teams, Personalities, and Getting the Job Done
by Jenda (Abbot) on Sep 15, 2005 at 22:55 UTC
    how much knowledge is embedded in the memory and experience of the one guy whose pager is always going off and who's getting tired of working 80 hour weeks. How many fixes were done by the one guy who's on vacation and unreachable until October?

    Lots. Though I don't have a pager, but a cell phone and it's not going off. And I don't have any vacation scheduled until November.

    The question is where and how to keep that information. And how to keep it up-to-date. And not spend 90% of time doing that. So, yes, if I were ran over by a bus (or did decide to leave) my one and only coleague in our team would have a tough time. I'm sure though that even in the first case he'd make it.

    Jenda
    XML sucks. Badly. SOAP on the other hand is the most powerfull vacuum pump ever invented.

[reply]
Re: Teams, Personalities, and Getting the Job Done
by zentara (Cardinal) on Sep 16, 2005 at 13:40 UTC
    could you recover data from a tape made on a Colorado Backup portable drive that had only Windows 95 drivers ever written

    That brings up an interesting point about storage.... do tapes even reliably store data that long? Even cdroms and DVDs, are being critisized for failling after only a few short years ( even though they are touted to last 100's of years). It would seem to me, that with that sort of "30 year requirement", that there should be a organization-wide "data storage standard format" (with simple error recovery methods), and a tested medium to put it on. Something like everything must be dumped to XML before written to some high quality medium, which will be stored in a temp-controlled, lead-lined vault.

    That is a good "breaucratic solution" because it involves alot of new staff, paperwork, and funds. :-)

    I'm not really a human, but I play one on earth. flash japh
[reply]
      If it had been an SCSI-Streamer and all the files had been tared on it, I would say that there would be a bigger chance to get those data restored after some years.

      On the other hand, 30 Years are lot of time in my opinion - 20 years ago we switched from 8 to 16 Bit architecture, approx. 15 years ago we switched to 32 Bit machines and nowadays 64 Bit are affordable to normal people.

      I didn't want to try to restore some files from 20 year old C64-disks now - and 20 years in the future I probably will find those old DVDs to crappy to try to restore some of the data stored onto them.

[reply]

Back to Meditations