seann has asked for the wisdom of the Perl Monks concerning the following question:

I updated from Fedora Core 35 to Fedora Core 38 over the weekend. This usually updates my Perl distribution. Here's what I'm running now: 
proxy::</home/seann> rpm -qi perl mod_perl httpd
Name        : perl
Epoch       : 4
Version     : 5.36.1
Release     : 497.fc38
Architecture: x86_64
Install Date: Sat 26 Aug 2023 06:43:01 AM MDT
Group       : Unspecified
Size        : 0
License     : GPL+ or Artistic
Signature   : RSA/SHA256, Tue 16 May 2023 05:19:26 AM MDT, Key ID 809a
+8d7ceb10b464
Source RPM  : perl-5.36.1-497.fc38.src.rpm
Build Date  : Tue 16 May 2023 04:52:03 AM MDT
Build Host  : buildvm-x86-11.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://www.perl.org/
Bug URL     : https://bugz.fedoraproject.org/perl
Summary     : Practical Extraction and Report Language
Description :
Perl is a high-level programming language with roots in C, sed, awk an
+d shell
scripting. Perl is good at handling processes and files, and is especi
+ally
good at handling text. Perl's hallmarks are practicality and efficienc
+y.
While it is used to do a lot of different things, Perl's most common
applications are system administration utilities and web programming.

This is a metapackage with all the Perl bits and core modules that can
+ be
found in the upstream tarball from perl.org.

If you need only a specific feature, you can install a specific packag
+e
instead. E.g. to handle Perl scripts with /usr/bin/perl interpreter,
install perl-interpreter package. See perl-interpreter description for
+ more
details on the Perl decomposition into packages.
Name        : mod_perl
Version     : 2.0.12
Release     : 6.fc38
Architecture: x86_64
Install Date: Sat 26 Aug 2023 06:43:00 AM MDT
Group       : Unspecified
Size        : 6694134
License     : ASL 2.0
Signature   : RSA/SHA256, Thu 19 Jan 2023 09:57:36 PM MST, Key ID 809a
+8d7ceb10b464
Source RPM  : mod_perl-2.0.12-6.fc38.src.rpm
Build Date  : Thu 19 Jan 2023 04:05:27 PM MST
Build Host  : buildvm-x86-08.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://perl.apache.org/
Bug URL     : https://bugz.fedoraproject.org/mod_perl
Summary     : An embedded Perl interpreter for the Apache HTTP Server
Description :
Mod_perl incorporates a Perl interpreter into the Apache web server,
so that the Apache web server can directly execute Perl code.
Mod_perl links the Perl run-time library into the Apache web server an
+d
provides an object-oriented Perl interface for Apache's C language
API.  The end result is a quicker CGI script turnaround process, since
no external Perl interpreter has to be started.

Install mod_perl if you're installing the Apache web server and you'd
like for it to directly incorporate a Perl interpreter.
Name        : httpd
Version     : 2.4.57
Release     : 1.fc38
Architecture: x86_64
Install Date: Sat 26 Aug 2023 06:33:29 AM MDT
Group       : Unspecified
Size        : 61953
License     : ASL 2.0
Signature   : RSA/SHA256, Tue 11 Apr 2023 06:52:41 AM MDT, Key ID 809a
+8d7ceb10b464
Source RPM  : httpd-2.4.57-1.fc38.src.rpm
Build Date  : Tue 11 Apr 2023 05:33:52 AM MDT
Build Host  : buildhw-x86-01.iad2.fedoraproject.org
Packager    : Fedora Project
Vendor      : Fedora Project
URL         : https://httpd.apache.org/
Bug URL     : https://bugz.fedoraproject.org/httpd
Summary     : Apache HTTP Server
Description :
The Apache HTTP Server is a powerful, efficient, and extensible
web server.

[download]
I have a web site that uses mod_perl and was working fine on FC35 and now hangs when using "SetHandler perl-script". I have reverted to "SetHandler cgi-script" for now. When Apache hangs, I run the following shell script looking for hung threads: 
#!/bin/sh

for pid in $(pgrep -u apache httpd); do
   echo Process $pid
   pstack $pid | grep ^#0 | grep -v _wait
done

[download]
This produces the following output: 
Process 6848
#0  0x00007fed4f59e384 in accept () from target:/lib64/libc.so.6
Process 6857
#0  0x00007fed4f591b77 in select () from target:/lib64/libc.so.6
#0  0x00007fed4f58b0fa in read () from target:/lib64/libc.so.6
Process 6858
#0  0x00007fed3c079843 in THX_ck_rv2xv () from target:/usr/lib64/perl5
+/vendor_perl/auto/Lexical/Var/Var.so
#0  0x00007fed3c0796b0 in THX_name_key () from target:/usr/lib64/perl5
+/vendor_perl/auto/Lexical/Var/Var.so
#0  0x00007fed3c079591 in THX_name_key () from target:/usr/lib64/perl5
+/vendor_perl/auto/Lexical/Var/Var.so
#0  0x00007fed4f53d167 in __strncmp_sse2 () from target:/lib64/libc.so
+.6
#0  0x00007fed4f591b77 in select () from target:/lib64/libc.so.6
#0  0x00007fed4f58b0fa in read () from target:/lib64/libc.so.6
Process 6859
#0  0x00007fed4f591b77 in select () from target:/lib64/libc.so.6
#0  0x00007fed4f58b0fa in read () from target:/lib64/libc.so.6
Process 6861
#0  0x00007fed4f591b77 in select () from target:/lib64/libc.so.6
#0  0x00007fed4f58b0fa in read () from target:/lib64/libc.so.6

[download]
I've been trying to track down the THX_name_key() call in source code but I cannot find any reference to it on the Internet. Here's a stack dump from one of the process threads: 
Thread 24 (Thread 0x7fed431e96c0 (LWP 6959) "/usr/sbin/httpd"):
#0  0x00007fed3c0795e0 in THX_name_key () from target:/usr/lib64/perl5
+/vendor_perl/auto/Lexical/Var/Var.so
#1  0x00007fed3c0798aa in THX_ck_rv2xv () from target:/usr/lib64/perl5
+/vendor_perl/auto/Lexical/Var/Var.so
#2  0x00007fed4e2630df in Perl_newUNOP () from target:/lib64/libperl.s
+o.5.36
#3  0x00007fed4e2b94f9 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#4  0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#5  0x00007fed4e36b12b in Perl_pp_require () from target:/lib64/libper
+l.so.5.36
#6  0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#7  0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#8  0x00007fed4e27f944 in Perl_call_list () from target:/lib64/libperl
+.so.5.36
#9  0x00007fed4e26e158 in S_process_special_blocks () from target:/lib
+64/libperl.so.5.36
#10 0x00007fed4e26f3c6 in Perl_newATTRSUB_x () from target:/lib64/libp
+erl.so.5.36
#11 0x00007fed4e270511 in Perl_utilize () from target:/lib64/libperl.s
+o.5.36
#12 0x00007fed4e2ba6e6 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#13 0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#14 0x00007fed4e36b12b in Perl_pp_require () from target:/lib64/libper
+l.so.5.36
#15 0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#16 0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#17 0x00007fed4e27f944 in Perl_call_list () from target:/lib64/libperl
+.so.5.36
#18 0x00007fed4e26e158 in S_process_special_blocks () from target:/lib
+64/libperl.so.5.36
#19 0x00007fed4e26f3c6 in Perl_newATTRSUB_x () from target:/lib64/libp
+erl.so.5.36
#20 0x00007fed4e270511 in Perl_utilize () from target:/lib64/libperl.s
+o.5.36
#21 0x00007fed4e2ba6e6 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#22 0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#23 0x00007fed4e36d231 in Perl_pp_entereval () from target:/lib64/libp
+erl.so.5.36
#24 0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#25 0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#26 0x00007fed4e27f944 in Perl_call_list () from target:/lib64/libperl
+.so.5.36
#27 0x00007fed4e26e158 in S_process_special_blocks () from target:/lib
+64/libperl.so.5.36
#28 0x00007fed4e26f3c6 in Perl_newATTRSUB_x () from target:/lib64/libp
+erl.so.5.36
#29 0x00007fed4e270511 in Perl_utilize () from target:/lib64/libperl.s
+o.5.36
#30 0x00007fed4e2ba6e6 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#31 0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#32 0x00007fed4e36b12b in Perl_pp_require () from target:/lib64/libper
+l.so.5.36
#33 0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#34 0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#35 0x00007fed4e27f944 in Perl_call_list () from target:/lib64/libperl
+.so.5.36
#36 0x00007fed4e26e158 in S_process_special_blocks () from target:/lib
+64/libperl.so.5.36
#37 0x00007fed4e26f3c6 in Perl_newATTRSUB_x () from target:/lib64/libp
+erl.so.5.36
#38 0x00007fed4e270511 in Perl_utilize () from target:/lib64/libperl.s
+o.5.36
#39 0x00007fed4e2ba6e6 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#40 0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#41 0x00007fed4e36b12b in Perl_pp_require () from target:/lib64/libper
+l.so.5.36
#42 0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#43 0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#44 0x00007fed4e27f944 in Perl_call_list () from target:/lib64/libperl
+.so.5.36
#45 0x00007fed4e26e158 in S_process_special_blocks () from target:/lib
+64/libperl.so.5.36
#46 0x00007fed4e26f3c6 in Perl_newATTRSUB_x () from target:/lib64/libp
+erl.so.5.36
#47 0x00007fed4e270511 in Perl_utilize () from target:/lib64/libperl.s
+o.5.36
#48 0x00007fed4e2ba6e6 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#49 0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#50 0x00007fed4e36b12b in Perl_pp_require () from target:/lib64/libper
+l.so.5.36
#51 0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#52 0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#53 0x00007fed4e27f944 in Perl_call_list () from target:/lib64/libperl
+.so.5.36
#54 0x00007fed4e26e158 in S_process_special_blocks () from target:/lib
+64/libperl.so.5.36
#55 0x00007fed4e26f3c6 in Perl_newATTRSUB_x () from target:/lib64/libp
+erl.so.5.36
#56 0x00007fed4e270511 in Perl_utilize () from target:/lib64/libperl.s
+o.5.36
#57 0x00007fed4e2ba6e6 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#58 0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#59 0x00007fed4e36b12b in Perl_pp_require () from target:/lib64/libper
+l.so.5.36
#60 0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#61 0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#62 0x00007fed4e27f944 in Perl_call_list () from target:/lib64/libperl
+.so.5.36
#63 0x00007fed4e26e158 in S_process_special_blocks () from target:/lib
+64/libperl.so.5.36
#64 0x00007fed4e26f3c6 in Perl_newATTRSUB_x () from target:/lib64/libp
+erl.so.5.36
#65 0x00007fed4e270511 in Perl_utilize () from target:/lib64/libperl.s
+o.5.36
#66 0x00007fed4e2ba6e6 in Perl_yyparse () from target:/lib64/libperl.s
+o.5.36
#67 0x00007fed4e369e2b in S_doeval_compile () from target:/lib64/libpe
+rl.so.5.36
#68 0x00007fed4e36d231 in Perl_pp_entereval () from target:/lib64/libp
+erl.so.5.36
#69 0x00007fed4e317958 in Perl_runops_standard () from target:/lib64/l
+ibperl.so.5.36
#70 0x00007fed4e27c5d2 in Perl_call_sv () from target:/lib64/libperl.s
+o.5.36
#71 0x00007fed4e795cde in modperl_callback () from target:/etc/httpd/m
+odules/mod_perl.so
#72 0x00007fed4e797bf0 in modperl_callback_run_handlers () from target
+:/etc/httpd/modules/mod_perl.so
#73 0x00007fed4e797fd7 in modperl_callback_per_dir () from target:/etc
+/httpd/modules/mod_perl.so
#74 0x00007fed4e79a449 in modperl_response_handler_cgi () from target:
+/etc/httpd/modules/mod_perl.so
#75 0x00005651504882da in ap_run_handler ()
#76 0x0000565150491d76 in ap_invoke_handler ()
#77 0x00005651504ce0c8 in ap_process_async_request ()
#78 0x00005651504ce2e7 in ap_process_request ()
#79 0x00007fed4e74ff02 in h2_c2_hook_process.lto_priv () from target:/
+etc/httpd/modules/mod_http2.so
#80 0x00005651504986da in ap_run_process_connection ()
#81 0x00007fed4e75925e in slot_run () from target:/etc/httpd/modules/m
+od_http2.so
#82 0x00007fed4f516907 in start_thread () from target:/lib64/libc.so.6
#83 0x00007fed4f59c870 in clone3 () from target:/lib64/libc.so.6

[download]
I don't know if I am on the right track for not trying to track down this problem. I just can't figure out where to start. I've been pouring over the mod_perl documentation looking for coding style that I need to fix to make it mod_perl compatible. I've found and fixed a few things, and haven't uncovered what is hanging mod_perl on Fedora Core 38. If I run enough https requests for the site through Apache I end up with a memory leak. If anyone has any suggestions on how to find THX_name_key() or THX_ck_rv2xv(), please let me know.

Replies are listed 'Best First'.
Re: mod_perl: Where can I find THX_name_key()?
by hv (Prior) on Aug 29, 2023 at 03:10 UTC

    The function THX_name_key is defined in the XS part of Lexical::Var here: Var.xs.

    I would be inclined to start by adding diagnostics in perl code to understand what it's trying to do before delving into the C code, eg by hacking Lexical::Var to warn a Carp::longmess when it is loaded, and when its import method is called.

[reply]
[d/l]
      Thanks for the pointer. Still not sure why Google wouldn't pull this up. I found a solution and will post it shortly.
[reply]
Re: mod_perl: Where can I find THX_name_key()?
by seann (Novice) on Aug 29, 2023 at 19:49 UTC
    Upgrading from Fedora 35 to Fedora 38 made the following changes: 
       Apache 2.4.54 -> Apache 2.4.57
   mod_perl 2.0.12 -> mod_perl 2.0.12 (no change)
   Perl 5.34.1 -> Perl 5.36.1

    [download]
    I downgraded back to Apache 2.4.54 and mod_perl was still hanging when multiple mod_perl requests for the same cgi-script came in. mod_perl worked fine when a single cgi-script request came in by itself.

    I attempted to roll back to Perl 5.34.1 and the dnf package manager wanted to roll back a huge number of rpms, so I looked for another solution. Perl 5.38.0 was released on July 3, 2023 and there is a Fedora 39 rpm available for that, so I upgraded to Perl 5.38.0 rpms on my test box and now mod_perl is working fine again with the existing codebase.

    My guess is there was a defect in the THX_name_key() or THX_ck_rv2xv() routines in my environment that caused an infinite loop when simultaneous mod_perl requests hit the web server. A single mod_perl request worked fine. I have no way to confirm this definitively, it's just my humble opinion. I was unable to identify anything in the release notes for 5.38.0 that might have been the root cause. Maybe a deeper inspection into the source code might reveal the answer, however I am not really qualified to do that.

    Thanks to the folks that took the time to respond to my post.

[reply]
[d/l]
      Take a look at https://metacpan.org/dist/Lexical-Var/changes. There were a huge number of bug fixes in Lexical::Var released on Mar 24, including several mentions of thread safety. If Redhat's packaging for 5.36 ecosystem didn't include that new version, that would explain why.

      BTW, any time you change perl interpreter versions, you *should* see a massive number of packages upgraded or downgraded because every perl module needs re-installed for that version of perl.

      (technically the pure-perl modules don't, but their Makefiles might behave differently depending on what other module versions are available, so even pure-perl ones ought to be re-installed generally)

[reply]
Re: mod_perl: Where can I find THX_name_key()?
by seann (Novice) on Aug 29, 2023 at 00:47 UTC
    Here's a follow up. This hung process is consuming 2 CPU cores worth of time. 
    top - 18:43:36 up 20:15,  2 users,  load average: 4.00, 4.01, 4.00
Tasks: 244 total,   1 running, 243 sleeping,   0 stopped,   0 zombie
%Cpu(s): 99.8 us,  0.2 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si
+,  0.0 st
MiB Mem :   3649.3 total,   2021.0 free,    485.4 used,   1142.8 buff/
+cache
MiB Swap:   7999.0 total,   7694.3 free,    304.7 used.   2853.8 avail
+ Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+
+ COMMAND                                                     
   6858 apache    20   0 2017692 131412  14080 S 199.7   3.5  91:09.18
+ /usr/sbin/httpd                                             
   4030 seann     20   0   18160   4456   4352 S   0.3   0.1   0:10.85
+ sshd                                                        
   7671 root      20   0  225244   3712   2944 R   0.3   0.1   0:00.02
+ top                                                         
      1 root      20   0  168760   9392   6016 S   0.0   0.3   1:56.96
+ systemd                                                     
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.03
+ kthreadd

    [download]
    and strace shows no system call activity. 
    proxy::</home/seann> strace -p 6858
strace: Process 6858 attached
read(5,

    [download]
[reply]
[d/l]
[select]
Re: mod_perl: Where can I find THX_name_key()?
by Anonymous Monk on Aug 29, 2023 at 08:12 UTC
    Does a hello world mod perl work?
[reply]

Back to Seekers of Perl Wisdom