Re: Is require still required?

At $WORK, we still use Sybase ASE, a proprietary database system which has ... interesting ... requirements (heh!) on how to connect to it. It comes with its own SDK and libraries, which can't be linked statically. On MySQL/MariaDB/PostgreSQL, it's just a yum/apt/$distro_tool install away to install client libraries and you're off to the races. Even Microsoft SQL Server has a simple RPM file you can install to get at an ODBC driver that doesn't need anything more to connect to their database server. Not so with Sybase ASE, as it needs a bunch of environment variables set up before their libraries will actually work. Of course, for daily use, these are set in the shell's ~/.profile, but your ~/.profile, ~/.bash_profile, ~/.bashrc, ~/.zshrc (insert your favorite shell initialization dotfile here) is not sourced when scripts get run from cron, systemd timers or some other job scheduling framework that just fork out to a binary directly.

So, most of our Perl scripts that touch Sybase ASE start with something like:

# We need the $SYBASE and $LD_LIBRARY_PATH environment variables to be
+ able
# to use the Sybase::Simple library (because it links to libsybct.so i
+n
# $SYBASE/$SYBASE_OCS/lib), but if we are invoked through cron, then
# ~/.profile doesn't get sourced and we are stuck with a nearly empty 
+%ENV
# hash.  So if we detect $SYBASE is not in %ENV, reexec through a shel
+l that
# first sources in the necessary environment variables.

unless (exists $ENV{SYBASE}) {
    # $^X is the path to the current perl interpreter
    # $0 is the script itself
    exec '/bin/sh', '-c', ". /db/sybsdk/SYBASE.sh ; exec $^X $0 @ARGV"
+;
}

# even with the above, we need to wrap the import of the Sybase Perl
# module in a runtime block as to avoid initialization by the Perl com
+piler
# prior to runtime (which checks for the necessary $SYBASE env variabl
+es).
# See 'perldoc perlfaq8' as to the difference between 'use' and 'requi
+re'.
# Basically: 'use' is ran at compile time, 'require' at runtime.
require Sybase::Simple;
[download]

So, to this day, there most definitely is a use (heh!) for require over use.

Comment on Re: Is require still required? Download Code

Replies are listed 'Best First'.
Re^2: Is require still required? by LanX (Saint) on Feb 06, 2024 at 06:46 UTC
Are you interested in other solutions? either: avoid require by putting the `exec` part in a `BEGIN` block to execute it at compile time prior to `use` or: avoid exec-restart-voodoo by importing the necessary ENV directly from a shell file. (see last update in Re^3: How to "source" a shell file in Perl? (Trojan Dump)) combine both Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery}	[reply]
Re^3: Is require still required? by uzb-dev (Acolyte) on Feb 06, 2024 at 20:01 UTC
Thanks for the input and the clever alternative of forking a subshell, dotsourcing any shellscripts in said subshell that modify the environment, and then serializing over the `%ENV` hash and eval'ing it in the parent process. This works for most normal environment variables, but not the ones that these Sybase libraries need. In particular, the Sybase libraries need custom `LD_LIBRARY_PATH` environment settings. `LD_LIBRARY_PATH` is interpreted by the OS's `ld.so` dynamic linker, but only at process initialization. Setting `LD_LIBRARY_PATH` from within a process will not affect the library search path of the process itself, only for subsequent childs the process launches. To expound on the problem, see the following transcript (`env -i` clears the environment): $ env -i PERL5LIB=.local/lib/perl5 perl -MSybase::Simple Can't load '.local/lib/perl5/x86_64-linux-thread-multi/auto/Sybase/CTl +ib/CTlib.so' for module Sybase::CTlib: libsybct_r64.so: cannot open s +hared object file: No such file or directory at /usr/lib64/perl5/Dyna +Loader.pm line 193. at .local/lib/perl5/Sybase/Simple.pm line 19. Compilation failed in require at .local/lib/perl5/Sybase/Simple.pm lin +e 19. BEGIN failed--compilation aborted at .local/lib/perl5/Sybase/Simple.pm + line 19. Compilation failed in require. BEGIN failed--compilation aborted. $ [download] The `PERL5LIB` environment variable is the only one that is set in the empty environment that launches the perl binary (to find the `Sybase::Simple` module). And indeed, in an empty environment, the XS `.so` blob that implements the actual database access layer in `Sybase::CTlib` (on which `Sybase::Simple` depends) refers to libraries that can't be resolved (and this is how these Sybase libraries work by design): $ env -i ldd .local/lib/perl5/x86_64-linux-thread-multi/auto/Sybase/CT +lib/CTlib.so linux-vdso.so.1 (0x00007ffcebdce000) libsybct_r64.so => not found libsybcs_r64.so => not found libsybblk_r64.so => not found libperl.so.5.32 => /lib64/libperl.so.5.32 (0x00007f4970c00000) libc.so.6 => /lib64/libc.so.6 (0x00007f4970800000) libm.so.6 => /lib64/libm.so.6 (0x00007f4970b25000) libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f4970fc3000) /lib64/ld-linux-x86-64.so.2 (0x00007f4971032000) $ [download] The code I arrived at from your linked comment to 'steal' the environment from a subshell (with the added requirement of merging with any existing `%ENV` values), is: `#!/usr/bin/perl BEGIN { my $shellcode = <<__KOEKOEK__; . sybsdk/SYBASE.sh perl -MData::Dumper -e 'print Dumper [\%ENV]' __KOEKOEK__ %ENV = (%ENV => @{eval qx{/bin/sh -c "$shellcode"}}); }; print $ENV{LD_LIBRARY_PATH}; require Sybase::Simple;` [download] But this still fails (even though the environment is modified correctly (the first line is the content of `$ENV{LD_LIBRARY_PATH}`): $ env -i PERL5LIB=.local/lib/perl5 perl ./do_shell_source.pl ~/sybsdk/DataAccess64/ODBC/lib:~/sybsdk/DataAccess/ODBC/lib:~/sybsdk/O +CS-20_0/lib:~/sybsdk/OCS-20_0/lib3p64:~/sybsdk/OCS-20_0/lib3p: Can't load '.local/lib/perl5/x86_64-linux-thread-multi/auto/Sybase/CTl +ib/CTlib.so' for module Sybase::CTlib: libsybct_r64.so: cannot open s +hared object file: No such file or directory at /usr/lib64/perl5/Dyna +Loader.pm line 193. at .local/lib/perl5/Sybase/Simple.pm line 19. Compilation failed in require at .local/lib/perl5/Sybase/Simple.pm lin +e 19. BEGIN failed--compilation aborted at .local/lib/perl5/Sybase/Simple.pm + line 19. Compilation failed in require at ./do_shell_source.pl line 14. $ [download] If we modify the script to just dump the `%ENV` hash, we see that all environment variables were indeed set correctly in the parent process from our detour in the `qx{}` subshell: `#!/usr/bin/perl BEGIN { my $shellcode = <<__KOEKOEK__; . sybsdk/SYBASE.sh perl -MData::Dumper -e 'print Dumper [\%ENV]' __KOEKOEK__ %ENV = (%ENV => @{eval qx{/bin/sh -c "$shellcode"}}); }; use DDP; p %ENV;` [download] $ env -i PERL5LIB=.local/lib/perl5 perl ./do_shell_source.pl { _ "/usr/bin/perl", INCLUDE "~/sybsdk/OCS-20_0/include:", LD_LIBRARY_PATH "~/sybsdk/DataAccess64/ODBC/lib:~/sybsdk/DataAcc +ess/ODBC/lib:~/sybsdk/OCS-20_0/lib:~/sybsdk/OCS-20_0/lib3p64:~/sybsdk +/OCS-20_0/lib3p:", LIB "~/sybsdk/OCS-20_0/lib:", PATH "~/sybsdk/tools/bin:~/sybsdk/OCS-20_0/bin:/usr/l +ocal/bin:/usr/bin", PERL5LIB ".local/lib/perl5", PWD "/tmp", SAP_JRE8 "~/sybsdk/shared/SAPJRE-8_1_062_64BIT", SAP_JRE8_64 "~/sybsdk/shared/SAPJRE-8_1_062_64BIT", SHLVL 0, SYBASE "~/sybsdk", SYBASE_OCS "OCS-20_0", SYBROOT "~/sybsdk" } $ [download] And this corresponds to what's inside the (Sybase provided) shell script file for their proprietary libraries that is meant to be sourced in the shell's environment: $ cat sybsdk/SYBASE.sh # # SAP Product Environment variables # SAP_JRE8="~/sybsdk/shared/SAPJRE-8_1_062_64BIT" export SAP_JRE8 SAP_JRE8_64="~/sybsdk/shared/SAPJRE-8_1_062_64BIT" export SAP_JRE8_64 SYBASE_OCS="OCS-20_0" export SYBASE_OCS INCLUDE="~/sybsdk/OCS-20_0/include":$INCLUDE export INCLUDE LIB="~/sybsdk/OCS-20_0/lib":$LIB export LIB PATH="~/sybsdk/OCS-20_0/bin":$PATH export PATH # # Replace lib, lib3p, and lib3p64 with devlib, devlib3p, and devlib3p6 +4 when debugging # LD_LIBRARY_PATH="~/sybsdk/OCS-20_0/lib:~/sybsdk/OCS-20_0/lib3p64:~/syb +sdk/OCS-20_0/lib3p":$LD_LIBRARY_PATH export LD_LIBRARY_PATH SYBASE="~/sybsdk" export SYBASE LD_LIBRARY_PATH="~/sybsdk/DataAccess/ODBC/lib":$LD_LIBRARY_PATH export LD_LIBRARY_PATH LD_LIBRARY_PATH="~/sybsdk/DataAccess64/ODBC/lib":$LD_LIBRARY_PATH export LD_LIBRARY_PATH SYBROOT="~/sybsdk" export SYBROOT PATH="~/sybsdk/tools/bin":$PATH export PATH $ [download] So, coming back around again: a re-exec is still required in this case (due to the semantics and interaction of `LD_LIBRARY_PATH` and the system's dynamic linker). Moreover, because a Perl script file is first compiled before it is executed, any statements that fire at compile time (like `use`) need to be deferred to runtime (using `require`) when `LD_LIBRARY_PATH` potentially needs adjustments. Otherwise, the `use Sybase::Simple;` statements gets triggered before the re-exec through a shell that first sets `LD_LIBRARY_PATH`, causing compilation to abort due to `DynaLoader` getting an error from the OS's `ld.so` when trying to resolve libraries a module's XS `.so` file is linked to.	[reply] [d/l] [select]
Re^4: Is require still required? by NERDVANA (Priest) on Feb 12, 2024 at 19:50 UTC
Are you aware of the patchelf tool? It lets you update the RPATH embedded within the .so files so that they can find other .so files without using LD_LIBRARY_PATH. You could run it on the perl Sybase/CTlib/CTlib.so and then perl would always be able to find the sybase libs. You could even run it on the fly as long as CTLib.so was writable and you modified it prior to loading it :-) Your re-exec option is probably cleaner since you're using the official way to configure the sybase lib location, and won't get lost if you install a new version of the module, but I just wanted to let you know about another fun option.	[reply]
Re^4: Is require still required? by mpeppler (Vicar) on Feb 12, 2024 at 08:09 UTC
If the location is fairly stable it should be possible to set the LD_LIBRARY_PATH via ld.so.conf (or similar). You'd still need the $SYBASE env variable so that things like the interfaces file can be found, but it would make things a lot simpler. Of course if you need to be able to run different versions of the Sybase client libs then I suppose setting this at run-time is necessary... Michael PS - I didn't think anyone was still using Sybase::Simple :-)	[reply]
Re^4: Is require still required? by LanX (Saint) on Feb 06, 2024 at 23:18 UTC
Wow, that sounds indeed weird. But if you have a working solution, I wouldn't want to interfere further. C and dynamic linkers are not really my field of expertise :) Cheers Rolf _{(addicted to the Perl Programming Language :) see Wikisyntax for the Monastery}	[reply]
Re^4: Is require still required? by bliako (Abbot) on Feb 12, 2024 at 12:19 UTC
that's where wrappers (shell scripts) come handy.	[reply]
Re^5: Is require still required? by LanX (Saint) on Feb 12, 2024 at 12:50 UTC