in reply to debugging apache response status 104 for cgi script

I'd use a network sniffer like wireshark, ethereal or tcpdump and capture the packets of interest on the server, possibly into a ring buffer. tail -f $access_log | grep 104 should then suffice to stop capturing in a timely manner to have the whole client-server conversation in the buffer.

BTW, are you using strict and warnings in your perl code? (obligatory question :-)

--shmem

_($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                              /\_¯/(q    /
----------------------------  \__(m.====·.(_("always off the crowd"))."·
");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}

Replies are listed 'Best First'.
Re^2: debugging apache response status 104 for cgi script
by blahblahblah (Priest) on Oct 24, 2006 at 20:45 UTC
    Thanks for the advice. It took some effort, but we were able to get ethereal going to monitor the server traffic. There is a definite pattern to the failures. Every one of them coincides with a TCP RST command, like this:
    client > server HTTP POST server > client ACK client > server RST
    I don't know enough about networking to interpret what that means, but I can see a one-to-one correlation between those RST's and the 104 status codes in our log.

    My best guess is that some firewall/antivirus/router/whatever in between their citrix and our server is examining the content of the traffic and deciding to kill the session because the Japanese UTF-8 data looks malicious.

    Does that seem like a reasonable theory? Anyone else know TCP well enough to offer other theories?

    Thanks,
    Joe

      Your theory is reasonable, but I can't tell whether it's accurate without having the logs. Next would be a network log at the client machine, which will be even more difficult than the server side sniffing, I guess. But now you should focus on determining the guilty party. Is it really the original client sending the RST, or is it some other component?

      Some hints.. you talk of Several Japanese users in the OP, do they come along the same network path? A traceroute to the client could tell you. The ethereal gui allows you to look at the packet's content and at various header fields. Are the sequence numbers like what is expected? Does the ACK from the server carry any payload? It shouldn't, and specially there shouldn't be any UTF-8 data present. What's the time difference between the ACK and the client RST? Maybe the client is running into a timeout?

      Again, without having the capture it's difficult to tell.

      --shmem

      _($_=" "x(1<<5)."?\n".q·/)Oo.  G°\        /
                                    /\_¯/(q    /
      ----------------------------  \__(m.====·.(_("always off the crowd"))."·
      ");sub _{s./.($e="'Itrs `mnsgdq Gdbj O`qkdq")=~y/"-y/#-z/;$e.e && print}