Strange infinite loop somewhere in libc.so while running FCGI server.

syber has asked for the wisdom of the Perl Monks concerning the following question:

I've discovered that from time to time my fcgi processes running under Catalyst are falling into infinite loop between requests.

Here is the GDB backtrace of running hanged process.

#0  0x0000000800c227c9 in _pthread_mutex_init_calloc_cb () from /lib/l
+ibc.so.7
#1  0x0000000800c26dce in free () from /lib/libc.so.7
#2  0x00000008006daad0 in Perl_sv_clear () from /usr/local/lib/perl5/5
+.10.1/mach/CORE/libperl.so
#3  0x00000008006dac96 in Perl_sv_free2 () from /usr/local/lib/perl5/5
+.10.1/mach/CORE/libperl.so
#4  0x00000008006c3e76 in Perl_hv_free_ent () from /usr/local/lib/perl
+5/5.10.1/mach/CORE/libperl.so
#5  0x00000008006c448e in S_hfreeentries () from /usr/local/lib/perl5/
+5.10.1/mach/CORE/libperl.so
#6  0x00000008006c6b3b in Perl_hv_undef () from /usr/local/lib/perl5/5
+.10.1/mach/CORE/libperl.so
#7  0x00000008006da97c in Perl_sv_clear () from /usr/local/lib/perl5/5
+.10.1/mach/CORE/libperl.so
#8  0x00000008006dac96 in Perl_sv_free2 () from /usr/local/lib/perl5/5
+.10.1/mach/CORE/libperl.so
#9  0x00000008006d2c40 in S_visit () from /usr/local/lib/perl5/5.10.1/
+mach/CORE/libperl.so
#10 0x00000008006d2ca1 in Perl_sv_clean_objs () from /usr/local/lib/pe
+rl5/5.10.1/mach/CORE/libperl.so
#11 0x000000080067bd71 in perl_destruct () from /usr/local/lib/perl5/5
+.10.1/mach/CORE/libperl.so
#12 0x0000000000400bcc in main ()
[download]

OS: FreeBSD 7.2

What the hell is going on? Unfortunaly OS core was built without "-g" so i cannot determine which line is the stopper in _pthread_mutex_init_calloc_cb

Can someone help?

Comment on Strange infinite loop somewhere in libc.so while running FCGI server. Download Code

Replies are listed 'Best First'.
Re: Strange infinite loop somewhere in libc.so while running FCGI server. by Illuminatus (Curate) on Mar 23, 2011 at 19:45 UTC
The function _pthread_mutex_init_calloc_cb is called from within malloc_spin_lock. This might hang if a thread in the same process space either terminated abnormally (leaving the mutex locked), or it might hang if allocated memory had been corrupted somehow. The last (less likely) possibility is that either FreeBSD or perl has a bug. How easy would it be to upgrade to an 8.x version of the OS to try it out? fnord	[reply]
Re^2: Strange infinite loop somewhere in libc.so while running FCGI server. by syber (Initiate) on Mar 23, 2011 at 23:44 UTC
Hmmm, the perl on that server was built without threads support (standard installation). So i doubt there were threads in that process :-) Also i discovered that the problem was happening when fcgi process reached max requests and tried to finish working. So perl just was cleaning up the memory. And you're right as far as i can remember the problem is occuring only on FreeBSD 7.X, not on 8.X. Thanks for advice, i'll try to upgrade the server to 8.X and monitor what will happen. :-)	[reply]