BrowserUk has asked for the wisdom of the Perl Monks concerning the following question:

How do other people work on concurrent versions of the same module?

Update: I'm trying to compare the results of running the old version(0.71) and current version (0.72) of Devel::Size side by side to determine where the differences arise.

Perl won't let you have both installed concurrently, nor easily back the later out in favour of the earlier.

My workaround this impasse is (as most things seem to be with Perl development), clumsy and laborious. I wondered how other people have tackled the problem which must have come up many times in the past.

  • Comment on Module development: concurrent versions (Updated)

Replies are listed 'Best First'.
Re: Module development: concurrent versions
by syphilis (Archbishop) on Dec 23, 2010 at 09:33 UTC
    Most modules that are being developed by multiple developers utilise git for the development process. Less commonly, svn is used - cvs seems to be a thing of the past.

    Is that what you're asking about ?

    Not sure what the options are for obtaining git and svn clients on MS Windows. I have Cygwin installed, and its git, svn and cvs clients all work fine for me, so I haven't really looked much further.

    I know you have (or had) msys installed - and I've seen good reports about msys' git ... but no first hand experience.

    Cheers,
    Rob

      Please see my update to the OP......

Re: Module development: concurrent versions (Updated)
by Anonyrnous Monk (Hermit) on Dec 23, 2010 at 09:58 UTC
    Perl won't let you have both installed concurrently,...

    In such cases, I usually don't install them, but leave them in their blib/ directories, and then do something like

    $ perl -Iblib/lib -Iblib/arch -MDevel::Size testscript.pl

    In other words, I have two or more build directories with different versions (opened in different terminals), and simply switch between them.  Not sure if that's what you mean, but that's my 2 cents, anyway.

      Yeah. I've done that before, but it means that you cannot use the two versions in the same program.

      The problem I'm trying to track down only manifests itself when the size of the arrays being sized gets above about 500,000 elements which make comparing trace output by eye impossible. So then you want to automate the comparison--diff or a custom script--but doing this with trace lines that (have to) contain addresses, means that all lines are different. (Ie. as different modules are loaded, the memory allocator has been exercised in different ways, so the addresses of test arrays and it elements are all different.)

      I needed to be able to load both modules concurrently so I can run them on the same array.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
        I needed to be able to load both modules concurrently so I can run them on the same array

        I tried the following experiment with Math-GMPz (a perl extension):
        1) Place version 0.31 of GMPz.pm in ./temp31/Math;
        2) Place its GMPz.dll in ./temp31/auto/Math/GMPz;
        3) Place version 0.32 of GMPz.pm in ./temp32/Math;
        4) Place its GMPz.dll in ./temp32/auto/Math/GMPz;

        Then run the following script:
        use warnings; use strict; unshift @INC, "temp31"; require "temp31/Math/GMPz.pm"; print $Math::GMPz::VERSION, "\n"; unshift @INC, "temp32"; require "temp32/Math/GMPz.pm"; print $Math::GMPz::VERSION, "\n";
        which outputs (as desired):
        0.31 0.32
        and just goes to prove that one can easily (albeit a bit kludgily) load 2 different versions of the same perl extension into the same script.

        You should be able to apply the same technique with Devel::Size.

        Cheers,
        Rob

        UPDATE: However, I note it's not quite so simple if I try to load them in the reverse order (ie first load 0.32, then 0.31):
        0.32 Math::GMPz object version 0.32 does not match bootstrap parameter 0.31 + at C:/_32/ap1007/lib/DynaLoader.pm line 224. Compilation failed in require at try.pl line 10.
        Not sure why that happens - perhaps some %INC manipulation would counter it.
        And, I guess, if you have to keep switching between the 2, then things get even messier.
        ...you cannot use the two versions in the same program.

        In that case, the only option I see is to give them different names/namespaces (temporarily), because you generally can't load two versions of a shared lib (with same name and exported symbols) twice, or load two Perl packages into the same namespace...

      Yeah. I've done that before, but it means that you cannot use the two versions in the same program.

      The problem I'm trying to track down only manifests itself when the size of the arrays being sized gets above about 500,000 elements which make comparing trace output by eye impossible. So then you want to automate the comparison--diff or a custom script--but doing this with trace lines that (have to) contain addresses, means that all lines are different. (Ie. as different modules are loaded, the memory allocator has been exercised in different ways, so the addresses of test arrays and its elements are all different.)

      I needed to be able to load both modules concurrently so I can run them on the same array. I now can, but getting there was messy.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Module development: concurrent versions
by Corion (Patriarch) on Dec 23, 2010 at 09:35 UTC

    Personally, I try to work on "feature branches", where I develop each new feature in a separate branch and merge the branch(es) into the main/release branch when they are ready for release.

    But most of the time, I simply develop on the main branch. This is a problem if I'm in a larger overhaul with failing tests and discover a bug(+fix) for the currently released version. So I should do development in branches more.

      Please see my update to the OP....

Re: Module development: concurrent versions (Updated)
by bingos (Vicar) on Dec 23, 2010 at 10:02 UTC

    Probably by using PERL5LIB environment variable to first point to the old version of the module, then the new one

    Or -I switch to achieve the same as the above

    Or the version control idea could work too, use git ( or some such ) to place the perl installation under version control, then you could have the old version on one branch and the new on another branch and just checkout the different branches as you test.

Re: Module development: concurrent versions (Updated)
by ELISHEVA (Prior) on Dec 23, 2010 at 23:54 UTC

    If you were in Java I would say, load each class in a seperate sibling class loader and put any classes shared in common into a parent class loader. Java identifies classes using class loader + name, so even if two classes have the same name if you've loaded them into sibling class loaders Java won't confuse their inheritance chains and inter-class relationships.

    Although Java is not Perl, and Perl is not Java, it appears you can use Perl threads in much the same way as the Java class loaders. Each thread gets its own copy of @INC and %INC. Furthermore, each thread inherits the contents of @INC and %INC from its parent thread, as illustrated in this little demo:

    Given that, why not just use two sibling threads in your program and load one version in thread 1 and the other in thread 2?

    Update: changed choice of classes to use in demo so that there is at least one package each thread loads that the other does not. (Turns out Scalar::Util pulls in List::Util, the two modules used in the original version - who knew!).

      why not just use two sibling threads in your program and load one version in thread 1 and the other in thread 2?

      Cool thought++ But ... :)

      I need to instrument (with trace) the same (address for address identical) array using both versions of total_size().

      This is impossible if the two routines are loaded into different threads.

      It's still good to see someone else around here considering threads for a possible solution, rather than a problem :)


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.

        Sadly it appears that there is no way to trick threads::shared into allowing a shared array reference. Perhaps an internals person might have an idea of how to get around that without causing ithreads to die, refuse to work with your array, or silently convert the array reference into a different address per thread.

        Clearly threads::shared has a way of knowing that array ref X on thread 1 is equivalent to array ref X on thread 2 because it has to know both of them in order to keep the two arrays in sync. Also the main reason for ithreads to refuse to share array references is concern that data not meant to be shared will be inadvertantly shared and lead to nasty race conditions and total nonsense as two threads try to insert elements into the same data structure using non-atomic operations. However, if an array and all of its elements are marked readonly in the C guts, then that should not be an issue. The question is there a way to get ithreads to agree it isn't an issue without rewriting part of ithreads? And further, would your tests be reliable if they ran on such a read-only marked array?

        But before wading deep into ithread internals, perhaps it is worth asking: what is the reason behind needing to instrument the exact same address? Is this because you don't trust the results unless they are physically working off the same spot in memory? Or is it because you can't properly scan the instrumentation output if addresses are different.

        If it is for ease of scanning (either by human or computer), I wonder if you could just come up with a way of transforming the trace output so that each stream replaces its own address with a like placeholder?

Re: Module development: concurrent versions
by Anonymous Monk on Dec 23, 2010 at 09:23 UTC
    I think I would assign each version to one person, or dedicate a day (or more) to each version.

    I'm not sure i understand this scenario, when/why would one person work on multiple versions of one module?

    I think I would refuse this job :)

      Please see my update to the OP.

Re: Module development: concurrent versions (Updated)
by andal (Hermit) on Dec 24, 2010 at 09:18 UTC

    This is the same problem as package name clashes. Ideally, instead of pulling in a package and making its name globally available to the whole program, the loading mechanism should allow the puller to associate with the loaded file a name that is local to its namespace. In this case it should be possible to "alias" the same module to different namespaces within the calling module. Internally perl could maintain the list of loaded files and avoid loading the same file multiple times. This approach would remove the need for unique package names.

    But there's no sense to talk about "ideal ways" here. In the current setup, one has to play with package namespaces. Something like

    BEGIN{ # load Pack1 defining "package Abc;" require Pack1; # copy the Abc namespace to Cde namespace my $ref = $main::{'Abc::'}; foreach my $n (keys %$ref) { *{"Cde::$n"} = $ref->{$n}; } # remove Abc namespace delete $main::{'Abc::'}; } BEGIN { # load Pack2 also defining "package Abc;" require Pack2; } # now the first version is in Cde package, # the second version is in Abc package Cde::test(); Abc::test();

    Actually, there might be some catches with the shared libraries. I don't know exactly how they are loaded. On Linux, one can load multiple shared libraries defining the same function names and then obtain pointers to functions in desired library. But I don't know if it is done this way everywhere.

      Actually, there might be some catches with the shared libraries. I don't know exactly how they are loaded. On Linux, one can load multiple shared libraries defining the same function names and then obtain pointers to functions in desired library. But I don't know if it is done this way everywhere.

      On Windows it is possible to dynamically load two dlls with the same name and exporting the same entrypoints (from different directories at least) at the C level.

      The problems that arise appear to come from the Perl internal mappings?

      How perl's dynaloader works, is one of those areas of dark, cross-platform voodoo that I find totally impenetrable.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
Re: Module development: concurrent versions (Updated)
by pajout (Curate) on Dec 27, 2010 at 10:07 UTC
    I suffered for it, but after some unsatisfying experiments I accepted following philosophy, assuming both versions of the module exists in some concurrent version system:

    If it is not absolutely necessary to use both versions in the same process, just put proper versions to special paths, dedicated for this purpose. Then you can have two or more paths for two or more versions and you can run one process for every version and compare their behavior, having remaining parts of the program identical.

    If it is absolutely necessary, rename versions (MyModule0_71.pm) for testing purposes. Though it is ugly work, it consumes less time than fiddling Perl guts, and it does not make chance that fiddled solution will work differently from renamed solution.

      If it is absolutely necessary, rename versions (MyModule0_71.pm) for testing purposes. Though it is ugly work,

      Yeah. That's what I did. But as you say, it is ugly.

      Simple things like the bootstrap entry point having a specific rather than generic name is what causes the problems. And it is unnecessary as there would be no conflict in calling a XS_bootstrap entrypoint in many .dll/.so's.


      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.