Hi,

I've got a problem I'm rather stumped by...

We have an architecture that sits inside a mod_perl handler, it has a debugging mechanism that debugs key points in the code (message + caller stack) to a list in a namespace that is _not_ cleared down for each request. It uses class based exceptions and a custom SIG DIE handler that can either log the error, environment and debug history and generate a friendly-ish message or can allow execution to continue if appropriate. The architecture itself is fairly mature (been using it for 5 years in production environments).

We have one use case where Apache has to stay loaded at all costs (because its holding a key that requires two members of staff to decode each time it is restarted). Therefore the inner handler uses Module::Reload->check to attempt to get any code changes since the last request hit the server.

The whole block of code in question is:

        Module::Reload->check;
        clear_module_exists();
        my $mh = Zymonic::Decryptor::Message->new(
            xmldef           => { class => $in->{messagetype} },
            zname            => $in->{messagetype},
            decryptor_server => $Zymonic::ZDECRYPTOR{$Zymonic::system}
+,
        );
        my $response = $mh->respond($in);
[download]

(clear_module_exists() - causes our internal list of loaded classes to be cleared).

Zymonic::Decryptor::Message is the base of a set of polymorphic classes which use zname/xmldef attributes to determine which sub-class to request from the 'module_selector' during new; that bit of code is:

    eval {
        eval "require $module";
        die $@ if $@;
        $MODULE_EXISTS->{$module} = 'true';
    } or do
    {
        my $err = $@;
        if ( $err =~ /Can't locate/ )
        {
            $MODULE_EXISTS->{$module} = '';
        }
        else
        {
            rethrow_exception($err);
        }
    };
[download]

i.e. when new() is called it passes a list of possible fulfilling modules Zymonic::Decryptor::Message::symmetrex_contact then Zymonic::Decryptor::Message in this case to a wrapping method that then does the above check to require each of the module, ignore ones that don't exist, throw errors for any that won't compile / require and return the first one that succesfully loads.

Roughly speaking across our 3 development environments, 1 QA environment and 1 UAT environment the architecture has served over 500,000 requests with approx. 30 Zymonic::Decryptor::Message sub-classes - the code has been 'stable' for over a year now.

That's all background; the issue is that we've had two requests that have logged the following error:

GENERIC Perl Died: Attempt to reload Zymonic/Decryptor/Message/symmetr
+ex_contact.pm aborted.
Compilation failed in require at (eval 1073) line 2, <GEN20> lin
+e 132.
[download]

The caller stack shows it was picked up and rethrown by die $@ if $@; in the above snippet. In both cases the error was thrown by the same symmetrex_contact module albeit with different eval line numbers.

perldiag tells me that the error occurs if the module failed to load previously and is thus still in %ENV but not being reloaded due to the previous error.

The logs show that in the two error cases symmetrex_contact was the first message class attempted to be loaded by the apache child in question (symmetrex_contact appears to work fine when other classes are loaded first, but also sometimes works fine when loaded first).

The symmetrex_contact module uses a number of other modules some of which are very old (in production for 20 years!), my guess is that some of these may have circular dependencies and only load properly if used in the correct order.

I've attempted to load and syntax check the module from the command line:

[Tue Feb 14 15:44:05] alex@zq2 ~ $ perl -c /usr/local/lib64/perl5/5.22
+.3/Zymonic/Decryptor/Message/symmetrex_contact.pm
/usr/local/lib64/perl5/5.22.3/Zymonic/Decryptor/Message/symmetrex_cont
+act.pm syntax OK

[Tue Feb 14 15:46:38] alex@zq2 ~ $ perl -e 'use Zymonic::Decryptor::Me
+ssage; my $tmp = Zymonic::Decryptor::Message->new( xmldef => { class 
+=> 'symmetrex_contact' }, zname => 'symmetrex_contact' ); print ref($
+tmp);'
Zymonic::Decryptor::Message::symmetrex_contact
[download]

Both of which work fine.

I have an idea for a hack ( thanks to this node : Node 1029344 ) where I check for the attempt to reload error, clear the module from $ENV and try again, but a) its a bit hacky and b) it may get me no closer to the solution if the reload fails and still prints; Compilation failed in require...

So my key question is now is there any way that I can get more information out of the error:

GENERIC Perl Died: Attempt to reload Zymonic/Decryptor/Message/symmetr
+ex_contact.pm aborted.
Compilation failed in require at (eval 1073) line 2, <GEN20> lin
+e 132.
[download]

is there some extended var that Perl might set to tell me which module failed to compile?

Additionally I'm confused as to why its a 'reload' error; nothing in the debug log of the apache child that generated the error suggested it had tried to load anything other than the basic framework before this error occurred.

Thanks in advance,

Alex