Hanging when reading from pipe-y open

ViceRaid has asked for the wisdom of the Perl Monks concerning the following question:

I've inherited a script that uses curl to connect over HTTPS and get the result of a CGI. This script has worked fine for a long time, but it's suddenly started hanging. I've isolated it to:

open(CURL, qq[curl -s -N "https://www.foo.com/cgi-bin/do.cgi" |]) 
    or die;
# the script hangs here
my $read = sysread(CURL, $line, 1);
close(CURL);
[download]

When I run the corresponding curl command in my shell, it prints out the content I want, but doesn't finish - the CGI doesn't seem to return an EOF that curl recognises. The original script had 'my @lines = <CURL>', which also hung. However, I thought I could circumvent this by using sysread and just reading enough until I have what I want, but I still have the problem.

Thanks in advance
Alex

Comment on Hanging when reading from pipe-y open Download Code

Replies are listed 'Best First'.
Re: Hanging when reading from pipe-y open by matija (Priest) on Mar 30, 2004 at 12:11 UTC
As long as curl doesn't close it's output, all the ways of reading are going to be stuck in the reading loop. Your best bet is to throw away the curl call, and replace it with LWP::UserAgent. Note that you need to install the right SSL libraries in order for LWP::UserAgent to handle `https:` connections.	[reply]
Re: Re: Hanging when reading from pipe-y open by ViceRaid (Chaplain) on Mar 30, 2004 at 12:19 UTC
Thanks. What I don't understand is if the shell can see (some of) of the output before the program closes its output stream (or not, as the case may be), why I can't similarly access the IO stream so far in perl. For good measure, the `-N` argument is supposed to make curl not buffer its output. As you suggest, I would much prefer to use LWP and Crypt::SSLeay, but am temporarily blocked by, erm, human factors, in the shape of a intemperate sysadmin. Thanks	[reply]
Re: Hanging when reading from pipe-y open by Abigail-II (Bishop) on Mar 30, 2004 at 13:11 UTC
Well the -N might cause curl not to buffer its output, the shell may. Try removing the quotes, that ought to cause Perl to exec() curl directly instead of passing it off to the shell. I can't remember on top of my head whether unblocking mode of file descriptors gets inherited after fork and exec, but you may want to set `$\| = 1` before your open. Abigail	[reply]