comment on

Hi,

I'm looking for advice on debugging an error in a cgi script. First, some background info: the script in this case is accepting utf8 data in many languages, including Japanese. Several Japanese users are experiencing occasional errors where they get no response from the server. At first we were inclined to blame the network, but now we know that other language users on the same network do not see the errors. Also, when the users swap machines the problems follow the Japanese users exclusively. And when the user goes back and re-types the same info into the form, it works on the next submission.

From the apache logs and from my own logging, I'm inclined to believe that this might be an apache error. The access_log shows this:

clientip - - [20/Oct/2006:16:53:56 +0000] "POST /...myscript.pl HTTP/1
+.1" 104 617 0
[download]

this breaks down as:

LogFormat "%h %t \"%r\" %>s %b %T"
%...h:          Remote host
%...t:          Time, in common log format time format
%...r:          First line of request
%>..s:          Status
%...b:          Bytes sent, excluding HTTP headers.
%...T:          seconds taken to serve the request.
[download]

Aside from the date/time, the line is always the same. The first interesting thing to me was the 104 status code. Then I noticed that 617 bytes is the same size as the standard "500 internal server error" page. There is nothing in the error_log when these errors occur.

I write a log file at the start of my script, but when this 104 error happens the log file does not get written. So maybe this is an apache problem more than a perl problem? The 104 code would at least make me think that apache is getting a little confused.

My boss is still focused on ruling out the network. He wants the user to try yet another client machine, bypassing the proxy server. If that also fails, I want to have a next step in mind. We already verified (with a local proxy on the client's machine) that the request looked ok. Is there a way in apache to make it log the entire request before any processing occurs? Then we could definitely rule out the network.

How would you debug this problem? Could I attach strace to every apache process -- and if so, how could I make sense of the massive amounts of resulting data? Is there some apache option that might help here?

I found only 1 similar problem when searching google for the 104 status code: http://mail-archives.apache.org/mod_mbox/httpd-users/200302.mbox/%3C04aa01c2d76b$e890a020$08071bce@WS4%3E But there was no resolution in that thread.

Since I don't know what's relevant as far as versions, here's all the info I could think of mentioning:

[root@xxx4 debugging]# uname -a
Linux xxx4.xxx.net 2.4.20-021stab028.17.777-enterprise #1 SMP Tue Jul 
+19 19:31:27 MSD 2005 i686 i686 i386 GNU/Linux

[root@xxx4 debugging]# /usr/sbin/httpd -l
Compiled in modules:
  core.c
  prefork.c
  http_core.c
  mod_so.c

[root@xxx4 debugging]# /usr/sbin/httpd -V
Server version: Apache/2.0.51
Server built:   Feb 22 2005 19:17:50
Server's Module Magic Number: 20020903:9
Architecture:   32-bit
Server compiled with....
 -D APACHE_MPM_DIR="server/mpm/prefork"
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="run/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_LOCKFILE="run/accept.lock"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

[root@xxx4 debugging]# /usr/sbin/httpd -v
Server version: Apache/2.0.51
Server built:   Feb 22 2005 19:17:50

perl -V
Summary of my perl5 (revision 5.0 version 8 subversion 1) configuratio
+n:
  Platform:
    osname=linux, osvers=2.2.12-20, archname=i586-linux
    uname='linux mars 2.2.12-20 #1 mon sep 27 10:25:54 edt 1999 i586 u
+nknown '
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultipl
+icity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cc', ccflags ='-fno-strict-aliasing -D_LARGEFILE_SOURCE -D_FIL
+E_OFFSET_BITS=64 -I/usr/include/gdbm',
    optimize='-O2',
    cppflags='-fno-strict-aliasing -I/usr/include/gdbm'
    ccversion='', gccversion='egcs-2.91.66 19990314/Linux (egcs-1.1.2 
+release)', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=1
+2
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='off_t',
+ lseeksize=8
    alignbytes=4, prototype=define
  Linker and Libraries:
    ld='cc', ldflags =' -L/usr/local/lib'
    libpth=/usr/local/lib /lib /usr/lib
    libs=-lnsl -lndbm -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lposix
    perllibs=-lnsl -ldl -lm -lcrypt -lutil -lc -lposix
    libc=/lib/libc-2.1.2.so, so=so, useshrplib=false, libperl=libperl.
+a
    gnulibc_version='2.1.2'
  Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-rdynami
+c'
    cccdlflags='-fpic', lddlflags='-shared -L/usr/local/lib'

Characteristics of this binary (from libperl):
  Compile-time options: USE_LARGE_FILES
  Built under linux
  Compiled at Feb 18 2004 10:50:23
  @INC:
    /usr/my_perl/lib/5.8.1/i586-linux
    /usr/my_perl/lib/5.8.1
    /usr/my_perl/lib/site_perl/5.8.1/i586-linux
    /usr/my_perl/lib/site_perl/5.8.1
    /usr/my_perl/lib/site_perl
[download]

Thanks,
Joe

Update: added readmore tags

In reply to debugging apache response status 104 for cgi script by blahblahblah

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.