Compiling and deploying Perl XS modules on a QNAP embedded system

virtualdj has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have a problem with Perl deployment, sorry for the long post but I would like to explain it clearly.
Basically I'm trying to compile XMLTV (which is written in Perl) to run it on my QNAP TS-239 embedded system, which is an x86 architecture.

As I want to distribute XMLTV to others, I want also to be able to compile it for several other QNAP NAS models, which run for example x86_64 or armel but that's a further question.

Let's start with the system information: I'm using the QNAP and a Debian Lenny Virtual Machine.

QNAP

GLIBC version = 2.6.1
uname = Linux NAS 2.6.33.2 i686 unknown

Debian VM

GLIBC version = 2.7
uname = Linux debian 2.6.26-2-686 i686 GNU/Linux

Using that VM I was able to compile any C code and then - moving those files on the QNAP - they run successfully.

I first tried building the XMLTV on the Debian VM. As I want to deploy the XMLTV application to others, I've built it together with the required modules in a custom folder: /root/out.

I first installed CPAN, changed its configuration data with:

o conf makepl_arg INSTALL_BASE=/root/out
o conf commit
[download]

then I've compiled all the dependecies like LWP and Date::Manip. Sometimes CPAN fails to build a module automatically, like with XML::Parser, so I've used:

look XML::Parser
perl Makefile.PL INSTALL_BASE=/root/out
make && make install
exit
[download]

... obviously by resolving the required dependencies first (which are printed on the console when running perl Makefile.PL).

Finally I've compiled XMLTV too in the folder /root/out/xmltv using the same commands than before; I've exported the PERL5LIB:

export PERL5LIB=/root/out/lib/perl5:/root/out/xmltv/lib/perl5

Then if I run /root/out/xmltv/bin/tv_grab_it on the Debian VM it works OK.

Moving the whole "out" folder inside the QNAP /share/FTP directory, exporting the PERL5LIB (with the corresponding dirs) and running tv_grab_it however gives this error:

# /share/FTP/out/xmltv/bin/tv_grab_it
Can't locate XML/Parser.pm in @INC (@INC contains: /share/FTP/out/lib/
+perl5 /share/FTP/out/xmltv/lib/perl5 /opt/lib/perl5/5.10.0/i686-linux
+ /opt/lib/perl5/5.10.0 /opt/lib/perl5/site_perl/5.10.0/i686-linux /op
+t/lib/perl5/site_perl/5.10.0 /opt/lib/perl5/site_perl .) at /share/FT
+P/out/lib/perl5/XML/Twig.pm line 100.
BEGIN failed--compilation aborted at /share/FTP/out/lib/perl5/XML/Twig
+.pm line 100.
Compilation failed in require at /share/FTP/out/xmltv/lib/perl5/XMLTV.
+pm line 106.
BEGIN failed--compilation aborted at /share/FTP/out/xmltv/lib/perl5/XM
+LTV.pm line 106.
Compilation failed in require at /share/FTP/out/xmltv/lib/perl5/XMLTV/
+Supplement.pm line 21.
BEGIN failed--compilation aborted at /share/FTP/out/xmltv/lib/perl5/XM
+LTV/Supplement.pm line 21.
Compilation failed in require at /share/FTP/out/xmltv/bin/tv_grab_it l
+ine 233.
BEGIN failed--compilation aborted at /share/FTP/out/xmltv/bin/tv_grab_
+it line 233.
[download]

Actually, the file XML/Parser.pm exists in the folder /share/FTP/out/lib/perl5/i486-linux-gnu-thread-multi; if I try to export that dir also like this:

# export PERL5LIB=/share/FTP/out/lib/perl5:/share/FTP/out/xmltv/lib/pe
+rl5:/share/FTP/out/lib/perl5/i486-linux-gnu-thread-multi
# /share/FTP/out/xmltv/bin/tv_grab_it
/usr/bin/perl: symbol lookup error: /share/FTP/out/lib/perl5/i486-linu
+x-gnu-thread-multi/auto/Storable/Storable.so: undefined symbol: Perl_
+Istack_sp_ptr
[download]

It seems that the XS modules cannot be deployed correctly.
I also tried to compile (with the same procedure) XMLTV on the QNAP using CPAN; the funny thing is that then it runs on the QNAP but it fails (with a similar error) on the Debian VM.

So my questions are: is this the correct procedure? How can I compile and deploy XML::Parser and the other Perl modules that require XS, like Storable, on another system?

I want to distribute that libraries on a custom folder and then make Perl load them; I need to build them in a virtual machine because of the other architectures (for example for armel I'm running QEMU with Debian versatile).

Please help me, I'm new to Perl and that has already been difficult for me to reach this point... Thank you, bye.

Comment on Compiling and deploying Perl XS modules on a QNAP embedded system Select or Download Code

Replies are listed 'Best First'.
Re: Compiling and deploying Perl XS modules on a QNAP embedded system by Eliya (Vicar) on Dec 08, 2011 at 14:30 UTC
`/usr/bin/perl: symbol lookup error: /share/FTP/out/lib/perl5/i486-linu +x-gnu-thread-multi/auto/Storable/Storable.so: undefined symbol: Perl_ +Istack_sp_ptr` [download] Apparently, the two perls aren't binary compatible, i.e. they cannot load each others XS modules. You haven't said anything about the perl binaries/distributions themselves. Where do they come from? Which versions are they, which build/compile options (`perl -V)` ? Did you build them yourself, etc...? Can you build a perl on the Debian VM that will run on the QNAP? If so, that would probably be the way to go to get binary compatible perls on both systems.	[reply] [d/l] [select]
Re^2: Compiling and deploying Perl XS modules on a QNAP embedded system by virtualdj (Initiate) on Dec 09, 2011 at 13:14 UTC
Thank you very much for your answer! Yes, sorry, I forgot to mention the Perl versions because they were the same (or at least I tought that the version number was sufficient to declare the two perl identical...). On QNAP (installed from IPKG which is Optware's application manager for embedded systems): [~] # perl -V Summary of my perl5 (revision 5 version 10 subversion 0) configuration +: Platform: osname=linux, osvers=2.6.24, archname=i686-unknown-linux-gnu uname='linux ts509 2.6.24 #1 smp fri jul 18 01:47:30 cst 2008 i686 + gnulinux ' config_args='-Dcc=gcc -Dprefix=/opt -Duseshrplib -de' hint=recommended, useposix=true, d_sigaction=define useithreads=undef, usemultiplicity=undef useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=und +ef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='i686-unknown-linux-gnu-gcc', ccflags ='-fno-strict-aliasing -p +ipe -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2', cppflags='-fno-strict-aliasing' ccversion='', gccversion='4.2.1', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='i686-unknown-linux-gnu-ld', ldflags =' -L/home/slug/optware/ts +509/staging/opt/lib -Wl,-rpath,/opt/lib -Wl,-rpath-link,/home/slug/op +tware/ts509/staging/opt/lib -Wl,-rpath,/opt/lib/perl5/5.10.0/i686-lin +ux/CORE' libpth=/opt/lib /opt/local/lib /lib /usr/lib libs=-lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil perllibs=-lnsl -ldl -lm -lcrypt -lutil libc=/lib/libc-2.6.1.so, so=so, useshrplib=true, libperl=libperl.s +o gnulibc_version='2.6.1' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E - +Wl,-rpath,/opt/lib/perl5/5.10.0/i686-linux/CORE' cccdlflags='-fPIC', lddlflags='-shared -O2 -L/opt/lib -L/opt/local +/lib' Characteristics of this binary (from libperl): Compile-time options: PERL_DONT_CREATE_GVSV PERL_MALLOC_WRAP USE_LARGE_FILES USE_PERLIO Built under linux Compiled at May 16 2010 15:39:00 @INC: /opt/lib/perl5/5.10.0/i686-linux /opt/lib/perl5/5.10.0 /opt/lib/perl5/site_perl/5.10.0/i686-linux /opt/lib/perl5/site_perl/5.10.0 /opt/lib/perl5/site_perl . [download] On Debian VM (installed from apt-get): debian:~# perl -V Summary of my perl5 (revision 5 version 10 subversion 0) configuration +: Platform: osname=linux, osvers=2.6.26-2-amd64, archname=i486-linux-gnu-threa +d-multi uname='linux puccini 2.6.26-2-amd64 #1 smp fri aug 14 07:12:04 utc + 2009 i686 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dccflags=-DDEBIAN -Dccc +dlflags=-fPIC -Darchname=i486-linux-gnu -Dprefix=/usr -Dprivlib=/usr/ +share/perl/5.10 -Darchlib=/usr/lib/perl/5.10 -Dvendorprefix=/usr -Dve +ndorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/perl5 -Dsiteprefix=/us +r/local -Dsitelib=/usr/local/share/perl/5.10.0 -Dsitearch=/usr/local/ +lib/perl/5.10.0 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/ma +n/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man +/man3 -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Ua +fs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -DDEBUGGING=-g -Doptimize=-O2 + -Duseshrplib -Dlibperl=libperl.so.5.10.0 -Dd_dosuid -des' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=und +ef use64bitint=undef, use64bitall=undef, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='cc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict +-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFS +ET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fno-strict-aliasing + -pipe -I/usr/local/include' ccversion='', gccversion='4.3.2', gccosandvers='' intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1 +2 ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t', + lseeksize=8 alignbytes=4, prototype=define Linker and Libraries: ld='cc', ldflags =' -L/usr/local/lib' libpth=/usr/local/lib /lib /usr/lib /usr/lib64 libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=/lib/libc-2.7.so, so=so, useshrplib=true, libperl=libperl.so. +5.10.0 gnulibc_version='2.7' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -O2 -g -L/usr/local/lib' Characteristics of this binary (from libperl): Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP USE_ITH +READS USE_LARGE_FILES USE_PERLIO USE_REENTRANT_API Built under linux Compiled at Aug 28 2009 22:15:29 @INC: /etc/perl /usr/local/lib/perl/5.10.0 /usr/local/share/perl/5.10.0 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl . [download] The QNAP and the Debian are different systems on how they look for shared libraries; for instance QNAP installs their lib on /opt/lib (because it's stored on the HDD, while /usr/lib is on a RAM disk), but normally with C apps it was a merely matter of putting the right path on LD_LIBRARY_PATH. I may write a stupid thing, but when I was building XML::Parser it asked me for libexpat.so file and header; on the QNAP I can install it with Optware (ipkg install expat), on Debian with APT (apt-get install libexpath-dev) and in both systems I can always build the library from source. But where should I place the library if the paths are different in Debian and QNAP? May the XML::Parser module store the hardcoded path which then fails to find the library on the other system? (LD_LIBRARY_PATH doesn't help, I've alredy tried!). However your last advice to try compiling and running the whole Perl seems to work... I've just to make some additional tests to see if I can replicate the procedure. Is this the only method to make the things working in your opinion?	[reply] [d/l] [select]