kmp has asked for the wisdom of the Perl Monks concerning the following question:

I have been trying to use cURL to get some info from a website, but having no luck with it. Most of the code I am using is copied right from the e-gold automatoin specs but I still can't get it to work. I've never used cURL before so I'm not sure if it's something I am doing wrong or maybe a problem with my server? 
$curlargs  = "PassPhrase=$pp";
$curlargs .= "&AccountID=$accountfrom";
$sysstring = "curl -s -d ";
$sysstring .= '"';
$sysstring .= $curlargs;
$sysstring .= '"';
$sysstring .= " https://www.e-gold.com/acct/balance.asp";
# read everything into one buffer
undef $/;
open(curls, "$sysstring|");
while(<curls>){$return = $_} 
close curls;

print "Content-Type: text/html\n\n";
print "$return";

[download]

Replies are listed 'Best First'.
Re: cURL help
by elusion (Curate) on Feb 16, 2003 at 17:13 UTC
[reply]
      I tried this bit of code pretty much right from the cpan LWP page and got the following errors in my error log. 
      #!usr/bin/perl -w

# Create a user agent object
  use LWP::UserAgent;
  $ua = LWP::UserAgent->new;
  $ua->agent("MyApp/0.1 ");

  # Create a request
  my $req = HTTP::Request->new(GET => 'http://www.e-gold.com/unsecure/
+metaldata.asp?LATEST=1&GOLD=1');
  $req->content_type('application/x-www-form-urlencoded');
#  $req->content('LATEST=1&GOLD=1');

  # Pass request to the user agent and get a response back
  my $res = $ua->request($req);

  # Check the outcome of the response
  if ($res->is_success) {
      print $res->content;
  } else {
      print "Bad luck this time\n";
  }

      [download]
      failed to open log file fopen: Permission denied Sun Feb 16 16:22:39 2003 error client 68.58.80.240 Premature end of script headers: /usr/local/psa/home/vhosts/spendmaker.biz/cgi-bin/test3.cgi
[reply]
[d/l]

        First you need to figure out why your web server isn't allowed to write to it's logs, but that isn't a perl issue. Second you need to try running that script from the command line and figure out why it's dying early. The first problem I see is on the very fist line (should be #!/usr/bin/perl -w).

[reply]
      Will that work with SSL?
[reply]
        Sorry, didn't see the 's' there. No, I don't think LWP::Simple supports SSL, but it'll work with LWP (the more complex version of LWP::Simple). Support in LWP is transparent, it seems. It's not too much harder to use either.

        elusion : http://matt.diephouse.com

[reply]
Re: cURL help
by Ryszard (Priest) on Feb 16, 2003 at 17:50 UTC
    If you add LWP::Simple to HTML::TableExtract you've got a great deal of flexibility when parsing web pages.

    To make page parsing a little easier:

    local*FH;
open (FH, '>/tmp/somefile.txt');
$te->parse($content); # $content is your HTML page from LWP::Simple
 foreach my $ts ($te->table_states) {
  print FH "Table (", join(',', $ts->coords), "):\n";
  foreach my $row ($ts->rows) {
     print FH join(',', @$row), "\n";
  }
}
close FH;

    [download]
    Which is lifted pretty much as is from the HTML::TableExtract documentation. I output it to a file so i can look at the output with the flexibility of my favourite text editor (vi).

    I should point out the above code example will print out the co-ords of the tables in your page and the content of each cell.

[reply]
[d/l]

Back to Seekers of Perl Wisdom