Beefy Boxes and Bandwidth Generously Provided by pair Networks
No such thing as a small change
 
PerlMonks  

de-inventing the wheel (discussion)

by deprecated (Priest)
on Jun 21, 2001 at 18:43 UTC ( [id://90377]=perlmeditation: print w/replies, xml ) Need Help??

I've been gone for a while because I have been so busy with work. I have finally gotten into the 'groove' here at this new client of mine, where one of my responsibilities is a "code audit". That is, I am going over an existing site, finding problems, and coming up with methodologies for diagnosing and fixing these problems. In addition to that, I am writing new applications which will hopefully last this organization longer than their current set did, about 2 years. I find that this organization has been de-inventing the wheel for that whole period. Intentionally not using upgraded software, and not using available modules.

We have many machines, all running SunOS 2.6-5.8 (except my lonely little OpenBSD machine), and all running perl 5.003-5.0053. I see continual snippets that look like this:

$mailprog = '/usr/lib/sendmail'; sub send_mail { open (MAIL, "|$mailprog -t -oi"); print MAIL "To: $toemail\n"; print MAIL "From: $fromemail\n"; print MAIL "Subject: $thistitle\n"; # ... close MAIL; }
This really is a rather tame example. There are things more like this deployed:
open FILE, "$string_from_insecure_webpage";
But I digress. What irks me is that there is a definite opposition to "upgrading" and installing modules. I think it was Nat Torkington who said that CPAN was perl's "killer app." He (or whomever) was right. I have a case here where a lot of this client's code could be simplified, modularized, and securified. However, nobody wants to install modules, and nobody wants to upgrade to a new version of, say, CGI.pm, for fear that some of the older scripts will be broken.

tilly continually gets into it with me over my use of our versus use vars. We've discussed other things like this. I have heard from many people that my wanting to upgrade to perl 5.6 from 5.003 is unreasonable.

I'm tired of hearing that!!! Code should be flexible. If you write code that isnt written exactly as the API suggests you write, youre being a bad programmer. If you want to say:

use CGI :standard; # instead of... use CGI qw{ :standard };
I hope your script DOES break, and I hope you pay me to fix it. If I cant write fluid, reasonable code using tools available to me because your code is hanging together by deprecated pieces of syntactic bubblegum and duct tape, you are hurting both of us. I cant write the responsible code I want to write, and your script breaks when you look at it cross-eyed.

The reason this is on my mind is I have written a program that uses CGI.pm's cool sub Vars, which must be imported via :cgi-lib. The older copy of CGI doesnt have it. Looking at the readme from the newer CGI, I see this warning:

Versions 2.44-2.46 introduce two API changes that will affect users of previous versions: 1) The accept() function has been renamed Accept() to avoid conflictin +g with Perl's built-in function of the same name. 2) The sub() function has been renamed Sub() for similar reasons. My apologies for these changes, but they were necessary in order for CGI to pass the perl5.005 regression tests!
Because the site is composed of roughly 200 individual scripts, all using #!/usr/local/bin/perl rather than #!/usr/local/bin/perlX.XXX, I am positive I am going to break at least one or two of them upgrading. Having to install a whole new copy of perl for one piddly module (albeit CGI) is a real waste of resources and time.

I'd really like it if some of the "dont upgrade" people could explain their reasoning, and tell me why I am wrong to want them to write reasonable, fluid, and upgradeable code.

What are you people going to do when perl 6 comes out? Huh?

what is the sound of one dep fuming?

--
Laziness, Impatience, Hubris, and Generosity.

Replies are listed 'Best First'.
Re: de-inventing the wheel (discussion)
by Masem (Monsignor) on Jun 21, 2001 at 19:03 UTC
    I know you've discussed woes at your place of work before, Dep, and it sounds like they have poor IT support; unforunately, there's not much you can do to change their minds about how they feel about updating and the like. The best that you can do is that when you think an update is needed is to compile a list of advantages and deterents to the upgrade, and make your case as clear as possible at the same time as respecting their concerns. Given them cost/time/diskspace/memory estimates, make it as easy as possible for them to yea or nay it without too much additional material.

    In this specific case, if you have a site, I'm surprised that you don't also have a testbed server that isn't publically available. What I would do is to copy the current batch of scripts to a testbed system, install the perl tools that you need, and see which scripts fail to work. Optionally, in the case that you're talking about, if you think that access() and sub() is used throughout the CGI scripts given (and given some of the code that you have shown, I doubt that your coworkers have used these), then simply do a grep for "use CGI" and "access" or "sub", and check to see how many instances that you hit; if it's more than a dozen or so out of 200 files, then you might have some concern, but anything on the low end is easy to fix. (I'd want to go one step further, saying using a sed-like tool to simply convert all 'access' to 'Access' and 'sub' to 'Sub', but that's not the best answer if you don't know what context they are used in, eg is it sub the function or sub the keyword?).

    But if you say that all the scripts simply have #!/usr/local/bin/perl instead of a specific version, and you want to have two different versions of perl installed, it's rather easy to use sed to fix this problem in all 200 scripts automagically. Simply force existing scripts to 5.005, and then have 5.6 around as well; make sure that 5.6's @INCs are separated from 5.005's @INCs as well to avoid module contaimination. Nothing existing breaks with this, and you can get 5.6 stuff out the door as well.


    Dr. Michael K. Neylon - mneylon-pm@masemware.com || "You've left the lens cap of your mind on again, Pinky" - The Brain
      These things I understand, Masem. I was more using my work as an example, and posing the question to people here.

      I hear from a lot of the people here, even, a very ingrained "upgrading is bad" vibe. Like if there isnt a very serious reason to upgrade, dont do it. Don't make use of new features in code (even if the API includes it), dont use modules, and so on.

      I feel it is unprofessional, unwise, and just plain bad coding to do so, and I was hoping to get some response from the people here that code this way.

      Perhaps I should post another piece of code with our all over it to draw the "dont use that"'ers out of the woodwork. :p

      brother dep.

      --
      Laziness, Impatience, Hubris, and Generosity.

Re: de-inventing the wheel
by footpad (Abbot) on Jun 21, 2001 at 20:30 UTC

    I realize you asked for reasons why people drag their feet regarding upgrades and learning new ideas, but it may be more helpful (and constructive) to focus on solving the problem.

    I believe Masem's got the right idea in that you need to set up a separate installation of Perl 5.6, along with the modules you desire and then use that to develop and deploy new work. It may seem a waste of resource at first glance, but consider the other changes, bug fixes, features, and modules you can tap into. It may be wiser to consider it an investment in the long run.

    I'm not certain how your site is implemented, but I believe it's fair to say that it either uses text files or databases at some point--two things that Perl excels at searching, editing, and updating.

    With this in mind, it should be possible to:

    • Locate all potentially troublesome CGI scripts on the server(s) in question.
    • Copy them to a testing area and modify the shebang line to use the 5.6 installation
    • See if any (or all) have compilation problems (e.g. -c) and record those problems in a separate file.
    • Automate the fixes that can be done easily.
    • Document the failures and develop in-house packages that provide convenient, more idiomatic solutions that fix the problem with as few changes as possible.
    • Invoke the modified scripts via LWP::Simple (or whatever) using test parameter values.
    • And so on

    The idea being to borrow the most successful hacking tactic and socially-engineer your managers, sponsors, and co-workers to understand the benefit of your ideas. People usually don't change their minds until they see real advantages to doing so. You can demonstrate these benefits (and also help determine the scope of the problem) pretty easily by using Perl's skills as a glue language.

    (Done right, these utilities may eventually lend themselves to a complete deployment harness, like the one tilly alludes to from time to time.)

    For example, you mentioned that you repeatedly see similar calls to sendmail. Since it's used so repeatedly, you should have an in-house wrapper that provides one or more interfaces into a more appropriate approach. Similarly, you should have a shared openFile() subroutine, one perhaps that tests the filename for taintedness and fails cleanly.

    You may not be able to automatically find every problem, but if you can find most of them, it should be reasonably easy to get people to buy into fixing them--especially if you provide convenient and simple solutions to those problems. Once the majority of problems are found and solved, it shouldn't take long to fix any remaining issues.

    There are several ways to tackle the technical details and I'm sure you can think of several more than I've mentioned. However, you need to show your people why they should upgrade their thinking and I believe the best way is to given them evidence that doing so will provide better results, be far less difficult (and expensive) than they fear, and provide clear and present benefits to the function, stability, and security of your site. It'll take time, patience, diplomacy, and some careful task management, but I think it'll help the organization in the long run.

    --f

Re: de-inventing the wheel (discussion)
by stephen (Priest) on Jun 21, 2001 at 20:49 UTC
    I agree with dep, and will go a step further. I believe that doing constant upgrades is valuable because it forces the re-evaluation and refactoring of old code. (I'm going out on a limb here, but I think that this raises some interesting points, so I'll just keep sawing at the branch...)

    Good, alterable code is living code. Any code that a company is afraid to change is dead code. If programmers refuse to upgrade because they fear breaking code, then think of how hard it will be to change that code to add a feature when a customer needs it. On the other hand, if the code is strong enough to withstand a few upgrades, it's a good sign that it's pretty robust code.

    If code has a good set of regression tests behind it, then upgrading is easy-- just do the upgrade and run the tests, and you can have confidence that you haven't broken anything major. Most companies don't encourage writing automated regression tests, just as most companies don't encourage writing living, flexible code.

    On the other hand, companies will argue that they're in the business of making money, that they don't pay programmers to introduce bugs into already-running code but to add new features instead. There is some justice to this position, and one must make reasoned choices as to how much time and effort one is willing to expend to keep code high-quality. However, I feel that such reasoning is overapplied. In the long term, good, robust, upgradeable code saves a company far more time and money than any number of short-term cost savings.

    The sad truth is, though, that many companies right now are going for short-term gains because they don't expect to be around long enough to HAVE a long term. So I have some sympathy for the devil in this case, although I'm not quite his/her advocate.

    stephen

      There's more to it than that. There is a very significant portion of managers, who oversee either programmers or managers who do so, to whom testing is a "waste of time and money" because "it doesn't add value" to the code. Their concept is that there is a right way to write code and, since they're paying a large sum of money for the programmers, they should get it right the first time. And, in a weird way, that position is a fair one.

      However, that position is only defensible if the programmers have the following characteristics:

      1. Intimately understand every business requirement, including how they might change
      2. Are involved in the project from the initial design phase through retirement, especially in timelines
      3. No new programmers join the project
      4. No one else is involved in any way with the project
      None of those four requirements are ever real-world. Ever. Someone else always writes the requirements (which are impossible to fully comprehend). Someone else always writes the timelines, and then pulls them in. No programmer is ever involved with a project from pre-design to retirement, and staff is always rotating. And, you're lucky if as many as half the people on a project are the programmers.

      So, testing, especially regression suites, become extremely important. However, to write a solid regression suite is a project in and of itself. By its very nature, you cannot have the programmer(s) write it. They don't know enough about the requirements and they know too much about the implementation.

      You should never have the programmer(s) sign off on the testing, anyways. They know too much about the code. Testing, imho, should be (primarily) black-box testing. Programmers, by their nature, do white-box testing, which is good, but isn't stringent enough for release sign-off.

      That said, upgrading should be easy if the programmers were able to write the application in a modular enough fashion. Upgrades very rarely affect more than two or three logical sections of an application. The problem with upgrades is that those logical sections have historically never been separated out. Not even with OO code. (I worked on a project which had objects, but encapsulation was violated left and right.)

      There needs to be better design and requirements. If that is done, better coding will necessarily follow. Bad requirements => Bad design => Bad code => No upgrading.

Re: de-inventing the wheel (discussion)
by chipmunk (Parson) on Jun 21, 2001 at 21:47 UTC
    Why is that we're the bad guys for being satisfied with the current version of Perl, while you're writing code that can't be run under older, but still widely used, versions of Perl? (And I bet perl5.005 is still more widely used than perl5.6, btw.)

    Then, you suggest that the only reason perl5.005 code would break under perl5.6 is that a programmer didn't follow the API. I recommend you peruse the perldelta documentation that comes with 5.6.1; it lists a score of bugs that were fixed from 5.6.0. Looking at that list of bugs, I'm glad we didn't upgrade to 5.6.0 at my company.

    FYI, I can write perfectly good code and yet not have a desire to upgrade to the latest version of Perl. And someone could easily upgrade to perl5.6.1 while continuing to write awful code. You seem to be confusing these two issues.

Re: de-inventing the wheel (discussion)
by jepri (Parson) on Jun 21, 2001 at 22:18 UTC
    I appreciate your point. But you are twisting peoples heads one way, and I'm afraid I want to twist them back the other.

    Your attitude is wonderful, and we need people like you to drag us forward. Without it, we would all still be programming Perl 2. Maybe.

    In my case, I'm one of those sad people who works on the principle that if it isn't broken, or completely shoddy, then it can be left for a bit later. I would like to spend time working on the newest and coolest, but I'll settle for "it works and I can go home". I apologise in advance for the despicable attitude.

    OTOH it annoys me when I am asked to find libraries to support Perl 4 code because some people are stuck in 1993 and just can't seem to make it into the 21st century. Hypocrisy never felt so good...

    And on a slightly different level, code is not flexible. Code is the physical implementation of those crazy ideas that we dream up. The ideas are flexible, but once we implement them in language we have fixed them, and they can't flex anymore. They can be rewritten in a new implementation, but I don't argue that the implementation will look anything like the previous one.

    ____________________
    Jeremy
    I didn't believe in evil until I dated it.

Re: de-inventing the wheel (discussion)
by toma (Vicar) on Jun 22, 2001 at 01:10 UTC
    In order to convince someone to upgrade, you need to convince them that there is a compelling reason to do so. Creating this argument is not a small task, and you may not be in a position in the company where it would be acceptable for you to even try to win this argument. That is, managers may feel that they should do the managing, and you should do the coding. If this is the case, you have several ways to be successful anyway:
    1. Convince one manager to advocate your approach, and have that manager do the business case and work on the office politics. This is the easy way.
    2. Become a manager. Ouch!
    3. Get a temporary assignment to look into this issue, create a project description, schedule, and work breakdown structure. Be sure that you use the same project management tools that the managers use. It will probably involve a suite of programs that all work together.

    One way to get managers to change the way they look at things is to come up with a theme. For example, to ship the existing site, managers probably had a theme like "Get to market". You need to show that you understand that this was the idea, and how it should be changed to something like "Secure the site". Maybe your next theme would be "Improve customization" or "Reduce cost of new features". Only propose one theme, though, and keep the theme short and simple, such as one or two words.

    These themes give the abilility to convince your audience of something that is not an implementation detail. At the level of the company above your department, the manager probably shouldn't care what version of perl you are using. This manager should care about functionality, usability, reliability, performace, and supportability. (If you memorize the previous sentence it will come in handy.) You will win when you convince a decision-maker that you can influence one or more of these important attributes. The decision-maker will understand the plan in terms of a simple theme, and will be convinced that you understand the implementation details needed to carry out the plan. Someone will have the specific assignment to make the plan happen, and will be held accountable if it fails.

    This probably sounds like a lot of work, and it may be that so far no one has had the drive to carry it out.

    It should work perfectly the first time! - toma

Re: de-inventing the wheel (discussion)
by jeroenes (Priest) on Jun 21, 2001 at 19:31 UTC
    find -name '*.pl' . exec perl -pi.boo -e 's*(^#!/[\w/]+/perl)$*$15.005 +*' {};
    'nough said ;^}
Re: de-inventing the wheel (discussion)
by perigeeV (Hermit) on Jun 22, 2001 at 02:55 UTC
    And so, as the great guru of your IS organization , here's the mantra you should repeat to yourself, and that you should encourage everyone in your cult...uh, department...repeat to themselves as a calming transition to an alpha state:

      "I'm paid to have headaches. I'm paid to have headaches. I'm paid to..."

          - Bob Lewis

    I think you're fighting the battle wrongly. Great inertia accumulates to things that work. Arguments for change may have sound technical merit, but I think many will agree that technical correctness rarely carries much weight in an organic(human) environment. Your only true weapon for change in a business setting is the immutable cost/benefit ratio.

    I know time-tracking is a nuissance to most, but how about keeping a log of your efforts. Note the actions you take that are a direct result of the offending sections that are not "reasonable, fluid, and upgradeable code." Compare that with the effort needed to keep the upgrades flowing. I think your results will vary from project to project.

    For what its worth, I agree with you. Yet somehow I am still maintaining some dusty SCO OpenServer boxen:-)

Re: de-inventing the wheel (discussion)
by sierrathedog04 (Hermit) on Jun 22, 2001 at 04:25 UTC
    I agree with you, but my brother, an incredibly successful developer who several years ago made $350 an hour or something like it, also de-invents the wheel. He says that for his website he will never upgrade from Perl 5.00x unless he has some reason to. Guess he figures upgrading can break things.

    That is not my philosophy, but it does make a certain amount of sense. Upgrading does sometimes break things. I once installed a newer version of the CPAN htgroup module on an old Perl 5.00x set up. Broke it. The developer had not tested his latest module on very old Perl interpreters.

    Fortunately the developer fixed it immediately, but still. Maybe de-inventing the wheel can be good in some instances.

Re: de-inventing the wheel (discussion)
by John M. Dlugosz (Monsignor) on Jun 23, 2001 at 01:22 UTC
    I don't understand your thesis. At first you were clearly saying that you want to upgrade to keep current, and are against the "don't upgrade" folks. But you have a problem upgrading cgi because you'll have to update old scripts--updates are always risky. So what are you getting at, here?

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlmeditation [id://90377]
Approved by root
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others having a coffee break in the Monastery: (6)
As of 2024-03-28 08:55 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found