davebaker has asked for the wisdom of the Perl Monks concerning the following question:

Monks,

I am migrating from CentOS 6 to CentOS 8 and find that a critical piece of software won't run on the CentOS 8 server.

I've been programming Perl for about 25 years, but I don't know much about what goes on under the hood, especially when it comes to operating systems and compilation.

The application (which was a commercial product that is no longer available), basically runs on perl 5.8.6, although it includes quite a few executables. It's a search engine (the Northern Light Enterprise Search Engine, which was a big player in the early web search engine space, pre-Google).

When the application installs itself, it copies onto the target server a precompiled version of perl 5.8.6 and many Perl modules, such as XML::Simple. It doesn't actually recompile that version of perl. Amazingly the whole thing worked after it was installed onto a CentOS 6 box about five years ago, after having worked for many years (2004-ish) on an earlier server.

On the CentOS 8 server, When I run (from a bash shell) a particular Perl script that ordinarily sets up various users and passwords, the result is

   -bash: /usr/local/ese/bin/ese_configurator.pl: /usr/local/ese/bin/perl: bad interpreter: No such file or directory

I've confirmed that the problem is not line endings (i.e., there is no ^M at the end of the #!/usr/local/ese/bin/perl shebang line). I've also confirmed that /usr/local/ese/bin/perl exists and has executable permissions; it's the perl 5.8.6 executable.

I ran the "file" command on the 5.8.6 on the CentOS 8 server:

   $> file /usr/local/ese/bin/perl

   $> /usr/local/ese/bin/perl: ELF 32-bit LSB executable, Intel 80386,
+ version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2
+, for GNU/Linux 2.2.5, stripped

[download]

I thought (naively) that I might be able to substitute a Perlbrew-installed perl 5.8.9 and modify @INC by adding a 'use lib' statement to the aforesaid configurator script so as to pick up the local directories that were installed by the application, -- but no joy:

   $> /usr/local/ese/bin/ese_configurator.pl

   $> Can't load '/usr/local/ese/perl/lib/5.8.6/i686-linux-thread-mult
+i/auto/Data/Dumper/Dumper.so' for module Data::Dumper: /usr/local/ese
+/perl/lib/5.8.6/i686-linux-thread-multi/auto/Data/Dumper/Dumper.so: w
+rong ELF class: ELFCLASS32 at /usr/local/ese/perl/lib/5.8.6/i686-linu
+x-thread-multi/XSLoader.pm line 68.
    at /usr/local/ese/perl/lib/5.8.6/i686-linux-thread-multi/Data/Dump
+er.pm line 27
   Compilation failed in require at /usr/local/ese/bin/ese_configurato
+r.pl line 6.
   BEGIN failed--compilation aborted at /usr/local/ese/bin/ese_configu
+rator.pl line 6.

[download]

I've also tried moving (copying) modules from the non-working 5.8.6 installation into the site_perl directories of the working perlbrew perl 5.8.9, but I gave up before finishing because I figured that it would be silly to expect that the non-pure-Perl modules from CentOS 8 would run if they were merely copied into the 5.8.9 directories.

FWIW, here are the results of running -V on the 5.8.6 that is running fine on the CentOS 6 server:

Summary of my perl5 (revision 5 version 8 subversion 6) configuration:
  Platform:
    osname=linux, osvers=2.4.20-31.9smp, archname=i686-linux-thread-mu
+lti
    uname='linux research4 2.4.20-31.9smp #1 smp tue apr 13 17:40:10 e
+dt 2004 i686 i686 i386 gnulinux '
    config_args='-d -e'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=define use5005threads=undef useithreads=define usemulti
+plicity=define
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS
+ -DPERL_USE_SAFE_PUTENV -fno-strict-aliasing -pipe -I/usr/local/inclu
+de -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2',
    cppflags='-D_REENTRANT -D_GNU_SOURCE -DTHREADS_HAVE_PIDS -DPERL_US
+E_SAFE_PUTENV -fno-strict-aliasing -pipe -I/usr/local/include -I/usr/
+include/gdbm'
    ccversion='', gccversion='3.4.4', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='gcc', ldflags =' -L/devel/sandbox/_last_build_sandbox/ese/ese/
+db-4.3.27.NC/build_unix -L/devel/sandbox/_last_build_sandbox/ese/ese/
+expat-1.95.7/.libs -L/devel/sandbox/_last_build_sandbox/ese/ese/opens
+sl-0.9.7d/blib/lib -L/devel/sandbox/_last_build_sandbox/ese/ese/gdbm-
+1.8.3/.libs -L/usr/local/lib'
    libpth=/devel/sandbox/_last_build_sandbox/ese/ese/db-4.3.27.NC/bui
+ld_unix /devel/sandbox/_last_build_sandbox/ese/ese/expat-1.95.7/.libs
+ /devel/sandbox/_last_build_sandbox/ese/ese/openssl-0.9.7d/blib/lib /
+devel/sandbox/_last_build_sandbox/ese/ese/gdbm-1.8.3/.libs /usr/local
+/lib /lib /usr/lib
    libs=-lnsl -lgdbm -ldl -lm -lcrypt -lutil -lpthread -lc
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lpthread -lc
    libc=/lib/libc-2.3.2.so, so=so, useshrplib=false, libperl=libperl.
+a
    gnulibc_version='2.3.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fpic', lddlflags='-shared -L/devel/sandbox/_last_buil
+d_sandbox/ese/ese/db-4.3.27.NC/build_unix -L/devel/sandbox/_last_buil
+d_sandbox/ese/ese/expat-1.95.7/.libs -L/devel/sandbox/_last_build_san
+dbox/ese/ese/openssl-0.9.7d/blib/lib -L/devel/sandbox/_last_build_san
+dbox/ese/ese/gdbm-1.8.3/.libs -L/usr/local/lib'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY USE_ITHREADS USE_LARGE_FILES PERL
+_IMPLICIT_CONTEXT
  Built under linux
  Compiled at May 18 2008 21:42:19
  @INC:
    /usr/local/ese/perl/lib/5.8.6/i686-linux-thread-multi
    /usr/local/ese/perl/lib/5.8.6
    /usr/local/ese/perl/lib/site_perl/5.8.6/i686-linux-thread-multi
    /usr/local/ese/perl/lib/site_perl/5.8.6
    /usr/local/ese/perl/lib/site_perl
    .

[download]

Here are the results of running -V on the perl version 5.8.9 that I had installed on the development server (CentOS 8), in hopes of being able to use it in lieu of the 5.8.6 -- perhaps it has details about the CentOS 8 system that are useful to know:

Summary of my perl5 (revision 5 version 8 subversion 9) configuration:
  Platform:
    osname=linux, osvers=4.18.0-193.28.1.el8_2.x86_64, archname=x86_64
+-linux
    uname='linux rs3.benefitslink.com 4.18.0-193.28.1.el8_2.x86_64 #1 
+smp thu oct 22 00:20:22 utc 2020 x86_64 x86_64 x86                   
+                                                                     
+   _64 gnulinux '
    config_args='-de -Dprefix=/home/www/perl5/perlbrew/perls/perl-5.8.
+9 -Aeval:scriptdir=/home/www/perl5/perlbrew/perls/                   
+                                                                     
+   perl-5.8.9/bin'
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultipl
+icity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=define uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -I/usr/local
+/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=6                   
+                                                                     
+   4',
    optimize='-O2',
    cppflags='-fwrapv -fno-strict-aliasing -pipe -I/usr/local/include'
    ccversion='', gccversion='8.3.1 20191121 (Red Hat 8.3.1-5)', gccos
+andvers=''
    intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+6
    ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib/../lib64 /usr/lib/../lib64 /lib /usr/li
+b /lib64 /usr/lib64 /usr/local/lib64
    libs=-ldl -lm -lcrypt -lutil -lc
    perllibs=-ldl -lm -lcrypt -lutil -lc
    libc=libc-2.28.so, so=so, useshrplib=false, libperl=libperl.a
    gnulibc_version='2.28'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
    cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib'


Characteristics of this binary (from libperl):
  Compile-time options: PERL_MALLOC_WRAP USE_64_BIT_ALL USE_64_BIT_INT
                        USE_FAST_STDIO USE_LARGE_FILES USE_PERLIO
  Locally applied patches:
        Devel::PatchPerl 2.02
  Built under linux
  Compiled at Nov 16 2020 19:57:40
  %ENV:
    PERLBREW_HOME="/home/www/.perlbrew"
    PERLBREW_ROOT="/home/www/perl5/perlbrew"
    PERLBREW_SHELLRC_VERSION="0.89"
  @INC:
    /home/www/perl5/perlbrew/perls/perl-5.8.9/lib/5.8.9/x86_64-linux
    /home/www/perl5/perlbrew/perls/perl-5.8.9/lib/5.8.9
    /home/www/perl5/perlbrew/perls/perl-5.8.9/lib/site_perl/5.8.9/x86_
+64-linux
    /home/www/perl5/perlbrew/perls/perl-5.8.9/lib/site_perl/5.8.9
    .

[download]

The CentOS 8 server is running a 5.26 as the system Perl. FWIW, here is the result of running "ldd" on it:

$> ldd /usr/bin/perl
        linux-vdso.so.1 (0x00007fffb67cd000)
        libperl.so.5.26 => /lib64/libperl.so.5.26 (0x00007f151ee10000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f151ebf0000)
        libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f151e9d9000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00007f151e7d5000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f151e453000)
        libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f151e22a000)
        libutil.so.1 => /lib64/libutil.so.1 (0x00007f151e026000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f151dc64000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f151f422000)

[download]

I ran "ldd" on the 5.8.6 that was copied onto the CentOS 8 server:

$> ldd /usr/local/ese/perl/bin/perl5.8.6
        not a dynamic executable

[download]

So, I wonder if you could suggest anything I could do in order to get the search engine running on CentOS 8.

If you have any other ideas for solving my problem, I'd be very much interesting to hear that, too.

THANKS.

Replies are listed 'Best First'.
Re: Running a perl 5.8.6 CentOS 6-compiled app on a CentOS 8 platform
by salva (Canon) on Nov 17, 2020 at 21:50 UTC
    What happens when you run that perl interpreter at /usr/local/ese/bin/perl? For instance: 
    $ /usr/local/ese/bin/perl -V

    [download]

    BTW, the error wrong ELF class: ELFCLASS32 probably means you are trying to dynamically link a 32 bit library into a 64 bit process.

    In any case, getting something so old to work in a new OS is going to be... interesting! In your place, I would probably take the virtualization route, trying to move the old OS into an LXC container, properly secured and running in the new machine. Though, that's not going to be easier either, even if the kernel has a really quite stable API/ABI, after so long, some incompatibilities are probably going to pop up. In the worst case, you can go for full virtualization.

[reply]
[d/l]
[select]
      $> /usr/local/ese/bin/perl -V
$> /usr/local/ese/bin/perl: error while loading shared libraries: libn
+sl.so.1: cannot open shared object file: No such file or directory

      [download]
[reply]
[d/l]
        yum provides '*/libnsl.so.1'

        This is more of a sysadmin problem than a perl problem.

[reply]
[d/l]
Re: Running a perl 5.8.6 CentOS 6-compiled app on a CentOS 8 platform
by bliako (Abbot) on Nov 17, 2020 at 21:37 UTC 
       -bash: /usr/local/ese/bin/ese_configurator.pl: /usr/local/ese/bin/p
+erl: bad interpreter: No such file or directory

    [download]

    Could you make sure that the interpreter does exist and works by running: 

    /usr/local/ese/bin/perl

    [download]
    . Is this a link and what's its endpoint, does it run at all?

    and also

    ldd /usr/local/ese/perl/bin/perl5.8.6
        not a dynamic executable>

    [download]

    what's the relation of the above to the aforementioned interpreter?

    Using Perlbrew to install an older Perl version, while your system uses its own, is the way to go. But you need to be able to use it without interference from system-installed Perl.

    Perl XS modules need recompilation when system libraries that they depend on change. I guess even legacy Perls need their XS modules recompiled when OS changes libraries.

    How about creating a new user. In this user's bash script (login script etc.) you specify to use Perlbrew perl5.8.6 then as that user you should be able to run $ perl -v and get 5.8.6. then install all modules that NorthernLight requires manually (you need to list them somehow, but also running the executables and when they fail on ... XXX not found you install it and move on until it stops complaining, it can work) as this user you just created.

    Edit: I mean, re-install perl5.8.6 with perlbrew manually. Do not depend on the NortherLight's Perl installation at all. Unless they made proprietary changes to the code, compiled it and now are providing binaries to you. Which obviously has a sell-by date ...

[reply]
[d/l]
[select]
       
$> /usr/local/ese/bin/perl
$> /usr/local/ese/bin/perl: error while loading shared libraries: libn
+sl.so.1: cannot open shared object file: No such file or directory

      [download]

      /usr/local/ese/bin/perl seems to be a symbolic link (from "ls -al:")

      lrwxrwxrwx.  1 root root     28 May 18  2008 perl -> /usr/local/ese/pe
+rl/bin/perl

      [download]

      Executing the target file from the shell seems to produce the same result as executing the link:

      $> /usr/local/ese/perl/bin/perl
$> /usr/local/ese/perl/bin/perl: error while loading shared libraries:
+ libnsl.so.1: cannot open shared object file: No such file or directo
+ry

      [download]

      (Edited 11/19/2020 to correct an insignificant typo)

[reply]
[d/l]
[select]

        Generally, this error means that that library was not found. The executable you tried to run depends on it. It could be somewhere but not at a location the system looks for libraries. Could it be that the system was upgraded and that library was upgraded too, to a newer version which is no longer ...so.1 ? Nobody stops you from installing older libraries in your system, check with your package manager how to do that (and once you are there check if libnsl is already installed and what version it is). However, that's a symptom which most likely is going to be followed by many more of the same nature.

        Can you recompile that special perl you have and all the required modules in the new OS? If yes, then it's worth doing this with perlbrew and then use cpan or cpanm to install all required modules at required versions.

        If you can't recompile but you only have binaries, then salva's suggestion building a container with Centos6 + installing those perl binaries is definetely a good option.

        If you feel lucky then just keep adding those missing libraries with your package manager trying not mess up your current Centos. But as salva already said, those error messages indicate a 32-bit/64-bit incompatibility, which is much harder to resolve...

[reply]

        Noting bliako's reply above, this suggests that there may be a lib directory that matches that perl somewhere. Try "find /use/local/ese -type d -name lib" to see if there are additional libaries. If so, try running the ESE's perl with those directories in the LD_LIBRARY_PATH environment variable; something like "LD_LIBRARY_PATH=/usr/local/ese/lib:/usr/local/ese/perl/lib /usr/local/ese/bin/perl" should do if find locates those directories. If that works, then you only need to find a way to pass that environment variable into the ESE CGI scripts and all should work.

[reply]
Re: Running a perl 5.8.6 CentOS 6-compiled app on a CentOS 8 platform
by jszinger (Scribe) on Nov 19, 2020 at 22:57 UTC

    Does it work on CentOS 7? That would get you a few more years of OS support.

    Is it a 32- vs. 64-bit issue?

    If all else fails, I suggest setting up a CentOS 6 container or VM for the application, and using a strong firewall. My feeling is that the bugs and security issues in the EOL application are a bigger problem thean the EOL OS.

[reply]

      Damn, that is very possibly a magnificent, practical solution! CentOS 7 wasn't nearly the jump from 6 (I understand) as is the jump from 7 to 8, and Rackspace will support it for several more years, as you point out. I will keep it in mind! I seem to be having some newfound success (described elsewhere) on 8.

[reply]
Re: Running a perl 5.8.6 CentOS 6-compiled app on a CentOS 8 platform
by perlfan (Parson) on Nov 18, 2020 at 19:48 UTC
    If perl is the only actual binary, then using perlbrew to install that version should get you headed towards a solution. Clearly the perl binary is the issue. Just replace it. If the shebang lines are pointing to a specific location, a softlink to the 64 bit executable should work fine. You may want to set PERL5LIB to the modules installed by the application rather than install the latest version of them. I also agree that creating a Docker container with the expected environment is a nice solution, but I've personally never run a 32bit OS container on a 64bit host so YMMV.
[reply]
[d/l]
[select]
A reply falls below the community's threshold of quality. You may see it by logging in.
A reply falls below the community's threshold of quality. You may see it by logging in.

Back to Seekers of Perl Wisdom