My guess would be that the DBD::mysql shared object file (XS part of the module) is linked against the MySql client libs (i.e. another shared lib), which might not be found in the Apache environment. Try setting an appropriate LD_LIBRARY_PATH...

P.S. you're most likely not interested in the .bs file (see Re: Empty perl *.bs files), but rather the .so file, which should reside in the same directory — do an ldd on that .so file in order to see which other libs it depends on...

It would help if you post the actually reported error instead of just "the following comment just above the line that fails"

I am in a CGI environment here – not mod_perl.

I strongly suspect that something like LD_LIBRARY_PATH will turn out to be the crux of this issue. I would be grateful if you could kindly submit (references to...) more, and more-specific, information about exactly what I ought to do. (I theorized that use lib might be enough...)

Oh Monks, this is arkane magick beyond my ken...

A cursory Super Search on LD_LIBRARY_PATH reveals many interesting-looking posts ... but are they merely rabbits, inviting me to useless chase? If not, what exactly would the proper value be in my case? (And why, prithee, are the proper .so files “automagically found, as I would expect,” when the test-suites are run from a shell-prompt?)

“I know not!” the confused Monk exclaimed in frustrated anguish. “I know not!”

if you could kindly submit ... more-specific, information

Locate your mysql.so file (presumably ./lib/perl/5.8.8/auto/DBD/mysql/mysql.so) and run (from outside of Apache, where things (hopefully) work — but on the same machine):

$ ldd /path/to/mysql.so
[download]

This should list its dependencies, among which there is probably something like libmysqlclient.so.15 (if everything is OK, there should be no "... => not found" entries). Note the path where this lib is found, and then arrange for the LD_LIBRARY_PATH environment variable to contain that directory in the environment of the CGI program. See "man ld.so" for what this is about. One way to set it could be in a BEGIN block in your CGI script, e.g. (I'm appending here, just in case it's non-empty)

BEGIN {
    $ENV{LD_LIBRARY_PATH} .= ":/path/to/lib";
}
[download]

Other ways would be to use a shell wrapper, or via the SetEnv Apache config directive.

BTW, which platform are you on? I'm asking because (a) the respective system tools (like ldd) might be different, (b) there is a special issue with LD_LIBRARY_PATH on Solaris, which might apply here... sorry, overlooked the "linux host" in your OP

As I read other postings with LD_LIBRARY_PATH in their titles, I see people talking about the need to “re-exec” things in order to make it actually work properly. Something about DynaLoader cacheing things...

Also, this seems vaguely like a Band-Aid® ... because I obviously do not yet understand the underlying mechanism (presumably DynaLoader) that Perl is using here. (I do understand the Linux dynamic-loader.) It seems vaguely odd to me that use base does not take care of “this, too.” Having to track-down a bunch of library-path locations does not feel right... as in “it might well work, but TMTOWTDI, and therefore is this really TBWTDI?”

Nevertheless... where's the right perldoc?

Hmmm... perldoc DynaLoader ... I'm looking hard at the discussion of the @dl_library_path variable.

The business of referring to the @INC variable looks very appealing, but... would it actually be reliable in the CGI-environment? I mean, “sure, it works in the command-line, but then again, so does my program.” As you can see, I am already specifying (I think...) the @INC entries by means of the use lib statement in the prologue of my code. (MENTAL FLASH!) Should I have put that statement in a BEGIN{} block?)

Well, apparently not ... my “mental flash” apparently did not help.

• I do know, from actual debug-logs in my code, that the @INC array does contain a path to a directory which contains auto/DBD/mysql/mysql.so, in both the command-line and the CGI environments.
• Running the CGI program, on the target server, from the command-line, works correctly.
• So, there is something wrong or not-specified in the CGI environment. That is the cause, and apparently the extent, of the problem.
• Looking at the ldd output, as suggested, I see that libmysqlclient.so.15 is located in the trusted-library, /usr/lib. So there should be no problems there...

Looking at perldoc DynaLoader, I observe (in the discussion of the bootstrap subroutine, which is described as “the normal entry point for automatic dynamic loading in Perl”), that it “locates an auto/$module directory by searching @INC” ... So I know that I am missing something that will soon cause me to slap my forehead vigorously.

But right now, I have “hit a mine” and I am dead-in-the-water with the clock ticking fast.

Maybe it's easier to debug this by just taking a look at what is actually going on under the hood — e.g. using strace.  Just create a wrapper something like this

#!/bin/sh
strace -o /tmp/strace.out -efile /path/to/your-cgi-script.pl
[download]

(The -efile is optional, and filters the output to only log file-related system calls, such as open, stat, access,...  In case your script involves subprocesses, you might want to use option -f, in order to trace forked processes, too.)

Then compare non-working vs. working, i.e. the output you get when running it via the webserver against the output you get when you call it on the command line.

This should give you a rather precise idea of which paths are being searched, what files are being opened, or aren't found, etc.  Maybe it's worth mentioning upfront that it's perfectly normal to get lots of "ENOENT (No such file or directory)" — this just shows what's being tried, not necessarily that there's a problem. It only indicates a problem if some file which needs to be loaded, is producing nothing but ENOENTs, or some other error. You get the idea...  In case you're having difficulties interpreting the details, just post the output here.

use DBD::mysql;
print "INC=$_\n" for @INC;
print "DBD::mysql $INC{'DBD/mysql.pm'}\n";
print "$_ = $INC{$_}\n" for keys %INC;
[download]

Also check

print "$_\n" for grep /_</, %::;
[download]

See perldebguts, perl5db.pl for explanation. or

perl "$_\n" for @DynaLoader::dl_shared_objects;
[download]

See DynaLoader

What flavor of linux are you using?

This might be a permission issue (chmod). Or if you're running SELinux it could be a security context issue (chcon). I think Fedora comes with this feature on by default, so it's a common "gottcha".

A good sanity check is to try running the cgi script from the command line. If that works, you know that the script runs as *you*, so the problem has to do with running the script as the apache user ('nobody' or 'www-data').

If your distribution provides a package manager for installs, did you use it? Ubuntu (and other Debian variants) have apt-get. RedHat (and variants) have yum. If you use those systems to install things like apache, mysql, etc. then everything should "just work". The only time I've ever had to muck around with LD_LIBRARY_PATH, etc. is when I downloaded tar balls and tried to compile/install "custom" versions of stuff myself. In fact, I even prefer apt-get for installing perl modules over CPAN, especially if the module has a binary component (i.e, things like DBI).

Package-based installation is not available here, and not relevant here. A regular cpan installation procedure has been used.

As noted later, the same process when run on my Apache development box ... worked.

In my ongoing bulldog-struggle with this thing, I modified CPAN's DBD::mysql.pm to wrap the bootstrap DBD::mysql $VERSION; statement in an eval{} block ... then test for errors. And, voilá! I found one!

Can't load '[...]/local-perl/lib/perl5/i486-linux-gnu-thread-multi/
+auto/DBD/mysql/mysql.so' 
for module DBD::mysql: libz.so.1:
failed to map segment from shared object: 
Cannot allocate memory 
at /usr/lib/perl/5.8/DynaLoader.pm line 225.
[download]

(Pathname "..." shortened for brevity.)

This is the failure which ultimately gives rise to the “Had to create DBD::mysql::dr::imp_data_size unexpectedly at ...” message, as well as all the others. Because the driver-module in fact isn't being bootstrapped at all, “nothing works.”

As previously stated, this message occurs only in Apache CGI mode.

I admit that I am rather mystified that DynaLoader::bootstrap doesn't throw any kind of error ... nor does mysql.pm ... if this sort of thing occurs. This apparently has been a well-known issue, perhaps with various drivers, for a number of years.

And yet, having written the above paragraph, I observe that when I replaced the print STDERR statement with either croak or die, the exception is “silently eaten.” by someone-out-there! I apparently can't surface the error-message from here via either of these two methods. Presumably the same thing is happening elsewhere, thus concealing the underlying error-condition from view.

Anyhow, this particular point in /usr/lib/perl/5.8/DynaLoader.pm is prefixed by an expected comment:

 
# Many dynamic extension loading problems will appear to come from
# this section of code: XYZ failed at line 123 of DynaLoader.pm.
# Often these errors are actually occurring in the initialisation
# C code of the extension XS file. Perl reports the error as being
# in this perl code simply because this was the last perl code
# it executed. 
  my $libref = dl_load_file($file, $module->dl_load_flags) or
   croak("Can't load '$file' for module $module: ".dl_error());
[download]

I see from the error-message that the correct version of mysql.so is being located and loaded, but that it fails during the load attempt. The root cause of the problem, as output by dl_error(), ultimately is:
libz.so.1: failed to map segment from shared object: Cannot allocate memory.

Update:   With a statically-linked driver with -lz included in the --libs="" parameter, the same message occurs, but the library-name is now libnsl.so. Plainly, I think, the root cause of this condition is not actually associated with any particular library:   I deduce that it is a loader issue.

Everything that I read about this sort of thing says, e.g. “This error is not related to shared libraries. You need to set maximum process size in megabytes.” But what I simply don't understand is how the resource-constraints of any reasonable hosting-service (this is 1&1 Internet... one of the big boys) could possibly be “too small.” How could 10 megabytes (or is it 20?) possibly be “too small,” and if so, why does it work fine... in my ulimit test, and in 1&1's own “500 Server Error” test-rig? My intuition tells me that there is something else wrong ... something that could “drop” DBD::mysql but not cause the application itself to completely die. If it truly were “out of memory,” I would expect to get a “500 Server Error” indeed... and nothing less.

The hosting-service's documentation says that shared host CGI programs are allowed a “10 megabyte” memory-limit, which seems more-than-generous to me. But just to be sure, I experimented with running the CGI program in a bash sub-shell with ulimit -Sm 10240 and it completed (in the shell...) without a quibble. (We're not talking about “large tables” or anything remotely-unusual here, anyway...)

That this error is silently eaten sounds to me as if something, somewhere up the call chain, sets up a $SIG{__DIE__} handler or replaces die with its own implementation. If you're interested in pursuing this further, consider at looking at the values of %SIG during the execution of your program.

I have no choice but to “pursue it” until it is solved. The lawyers will be circling, soon.

To make things even stranger yet, the host provides a “test jig” where you can apparently run the program in a mode to flush out “500 Internal Server Error” messages ... and it works there, too!

Really, the only environment where it doesn't work is when I go to it through the web-URL.

P.S.:   When I dumped out environment-variables, after putting a SetEnv directive in .htaccess, I was mildly surprised to see that the LD_LIBRARY_PATH variable that I was trying to set, had become REDIRECT_LD_LIBRARY_PATH. Why is that?

This is a shared-hosting environment that does not permit access to the error-logs.

As I said, the program runs absolutely cleanly in the command-line environment of the shared host. I can invoke the program and a few milliseconds later I am rewarded with the correct HTML output.

As I also said, I have verified that the @INC array does include an entry from where the auto/DBD/mysql/mysql.so file can be reached. (I verified this by adding a debug-output to the web-site code so that the output is known to come from the server's environment.)

My reading of man DynaLoader seems to say that the @INC array contents are consulted to find the auto... entry; that fiddling of DL_LIBRARY_PATH ought not to be required...

My next attempt will be to see if I can replicate the problem from my own Apache server.

My next attempt will be to see if I can replicate the problem from my own Apache server.

Whimper! It works there, too!

I can't see the problem, and I can't replicate it. All that this does seem to suggest is that the problem is not related to environment-variables... that is to say, that the use lib stuff does the right thing in both cases. (I do not install the Perl packages that are used for web-development “globally” on my system. I also rely upon .htaccess files just as shared-hosts do. The setup is designed to replicate shared-hosting, and I fully believe that it does so.)

This basically means that, on my box, the mysql.so module is being located and run correctly from the Apache environment.

<panic> OMG, I am so.... scroo'd ... </panic>

So you've installed CPAN modules with binary parts locally? And it runs from the command line but not in the browser? Did you try something like this:

chmod -R 755 ~/lib
[download]

(where I'm assuming all your local modules are under ~/lib)

Have you tried calling the host's customer support? Are you the first customer they've ever had to try using mysql?

If so - maybe their installation is broke, and if you report it they will fix it. Else, if other customers have done it, maybe they can tell you what they did different (assuming they've done something non-standard they requires extra steps)

Final option - switch hosts!

Nope! :-O

It did not change the situation at all!

Re-he-he-healy?? Something you thought was something you now think is something, but its really something? Big shocker.

The confusion continues...

Within the error-output (that I am now able to see), I have forced it to generate the output of a ldd command for mysql.so and separately also for libmysqlclient.so.15, after confirming that there are no conflicting libraries in sight.

With or without an LD_LIBRARY_PATH in place, the output of both ldd commands contains no “not found” messages. All of the dependencies point to corresponding libraries. This suggests that I am not, in fact, looking at a search-path issue.

I have even gone so far as to use dl_load_file() (from DynaLoader) in order to force the libraries to be loaded early. (Note that I did not (yet) do this in a .bs file...) When I did that, (a) all of them succeeded, but (b) I got an out of memory message(!) when the library tried to connect.

So... I am by now really “I'm confused as hell and I'm not gonna take it any more” here. It works just fine interactively; it fails with only these cryptic messages in Apache. Nothing I have tried has succeeded as “a workaround, however crude.”

Oh... now this thing is getting weirder and weirder...

I inserted some prologue-code that would issue a DBI->connect() call, before doing anything else, and...

• The connect() succeeds...
• The earlier message does not appear.
• The DBI->installed_drivers list (originally empty...) expands to include both mysql and DBI::dr=HASH(0x83a81a8) (?!)
• The out of memory message appears, just as it did earlier.
• And yet, everything works just fine in command-line!

Calm down. Stop spinning, and ignoring advice you've been given. If it works from command line and not cgi, stop guessing, and figure out what the differences are, its that simple. Write a program to load DBI/mysql, connect, disconnect, then dump @INC, %INC, %ENV, %::... and compare the output from commandline and from cgi. If you can't figure it out, post the code and output here.

My explorations, alas, are taking me deep into DBIguts ...

I have now established that the software is falling-down because it cannot find the _login subroutine (in the XS)... only in the Apache-CGI environment. And, I have established that in that environment it has successfully DynaLoaded mysql.so as well as a great many other things. I know that DBI->connect() calls, again in that environment, can be issued and do succeed. But the _login subroutine still is not there.

Looking further (in this extemporaneous train-of-thought), I can see that no .so module actually contains that routine, on their system or on mine. Determined viz:

grep -rilw _login . | grep \.so | xargs nm | grep _login

A side-by-side diff of two logs... one from CGI and one from the command-line, shows no difference between them except that one fails and one does not. In particular, DynaLoader's recitation of its search-path (which it gives at startup), is identical in both cases. The LD_LIBRARY_PATH environment-variable in both cases is undef. I frankly no longer believe that the root cause of the failure has to do with dynamic-loading, at least not in any obvious way. But that pesky out of memory message (not seen here in either log, but induced by various ways I've previously related...)... what did it mean?

And so my quest through mounds of dirty DBI “guts” continues. News at 11... (But no video. I promise.)

BULLETIN:   “This case might just have been busted wide open” by the discoveries spelled-out in the sister-thread:   CPANPLUS - what if "later" never comes?.

“Stay tuned for NEWS!”
:-/
... (sniff!) ... no, nevermore ...
R.I.P., Paul Harvey.

After what was (to me...) an extraordinary, week-long fight, I now conclude that this is, in fact, a memory-size problem. But finding the problem was a rather gruesome experience, as I would now like to briefly relate for the edification of anyone who may follow. I shall try to relate both epiphanies and mistakes ... and not to bore you.

Stave One:   the initial symptom

So I went right in to my local CPAN-module repository for this web-site, found the module, and wrapped the bootstrap call to become:

eval {
 ... bootstrap ...
};
print STDERR "bootstrap result: $@\n";
[download]

A serious problem in a shared-hosting environment is that you don't get to see the Apache error logs. (And, as I discovered, any tools that they might provide to help diagnose server-errors might not actually provide the same environment.) I was able to force the STDERR output to go to a place I could actually use by employing use CGI::Carp qw(carpout). With this routine, in a BEGIN{} block, I could actually see the messages.

In order to actually discover what DynaLoader was doing ... I did this:

 $ENV{PERL_DL_DEBUG} = 1;

The program itself threw a herring at me because it didn't “flop over and die.” Instead, it caught the error and kept going. And yet, it might be said that it didn't catch the right error, because the bootstrap failure did not immediately reveal itself... only the consequences thereof.

Stave three:   zeroing-in on memory

print STDERR "Ulimit is:\n\t" . `bash -c "ulimit -a"` . "\nnow you know...\n";

The output (also listed earlier...) shows that the basic limit is 32meg, which I erroneously supposed must be “plenty.” Although I stumbled-upon the ulimit command (described in man bash ...) fairly early-on, I wasn't using it correctly.

There are two resource-limits that you can impose... “soft” and “hard.” But only the “soft” limit matters. So I wound up doing this:

bash               # ... to give myself a nested shell
ulimit -Sl32       # ... for example
ulimit -a          # ... to see what else I needed to change
strace -ologfile perl spe-cgi.pl    # to run the program with tracing
# now review the "logfile" file that has magically appeared
[download]

When the program was sufficiently constrained, it finally began to fail, although it still does not do so in exactly the same way. (Right now it falls-dead.)

I'll bet if you examined %::, it would have shown mysql.so was not being loaded.

Obviously I don't know about this variable or its uses. In a separate thread, perhaps, could you describe it?

By means of CGI::Carp I have been able to force the error to reveal itself to me at last.

The error is a familiar one, and expected:

Had to create DBD::mysql::dr::imp_data_size unexpectedly at /usr/lib/perl5/DBI.pm line 1190.

Now, it might be worthwhile or interesting for you to know, at least for the record, how I managed to get this far. Here's the magick:

BEGIN {
  $ENV{'LD_LIBRARY_PATH'} = '/lib/tls';

  use FindBin qw($Bin);
  use lib ("$Bin/../../lib",
           "$Bin/../../../local-perl/lib/perl/5.8.8",
           "$Bin/../../../local-perl/share/perl/5.8.8",
           '/var/www/perl5/lib/perl5/site_perl/5.8.8',
           '/var/www/perl5/lib/perl5/5.8.8/i686-linux',
           '/var/www/perl5/lib/perl5'
          );

# RELEVANT STUFF STARTS HERE:
  my $path="$Bin/../../log/carpout.log";
no strict 'refs';
no strict 'subs';
  open(LOG, ">>$path") or
    die("Unable to open mycgi-log: $!\n");
  carpout(LOG);
}
[download]

To review... The first part of the block is simply my program's way of establishing connection to the local-copy CPAN library modules that it uses. FindBin is the ever-helpful standard module that will tell you where the program that you are running is actually located. The use lib directives then place relative pathnames based on this at the front of the @INC list.

The magick starts after the comment:   # RELEVANT STUFF STARTS HERE.

In my ongoing pursuit of this problem, I have also found the CPAN module on “troubleshooting mod_perl problems,” (simply titled troubleshooting ...) to be particularly informative. I have not yet completely solved my problem, but I'm well on my way.

LOL