dumbadam has asked for the wisdom of the Perl Monks concerning the following question:

I'm not sure if anyone can help.

I wrote a script that automatically posts pictures to picposts. This is something I use everyday and makes the process automated so that I can do other things once I set it up and type it in.
Anyway, it worked perfect for awhile; Just the other day I tried it and it stated a message on there "my" screen stating: "ive been banned" i emailed them; they say im not banned.
Heres the clintcher: I type in all my "usual" info exactly the same info as my script AND IT WORKS perfectly.
My question is: im obvisouly using LWP:useragent; http:headers; http:request and CGI to be able to accomplish this. Sending in all my own info makes there script think im actually doing it manually. I dont have time to do this everyday without my script.

I need to know from all you Professionals: What could "maybe" they have put into there "cgi-bin" files to prevent me from posting thru my script BUT, allow it to work if i do it manually.
I dont belive its cookies and iam already sending in http_referrer;http_host;server_name to there script. I m sorry for being long winded!

Heres an example but not the actual LONG script there is lots of missing pieces of this code I left out:
#!/usr/bin/perl


use CGI;
use HTTP::Headers;
use HTTP::Request;
use LWP::UserAgent;
$q = new CGI;

$posttourl = "http://www.theposturlhere.com/cgi-bin/post.cgi";
$refererurl = "http://www.theposturlrefererhere.com/cgi-bin/post.cgi";


$q = new CGI( {
    pic_url=>"$pictureurl",
    description =>$picturedes,
    banner_url=>$bannerurl,
    banner_link=>$siteurl,
    banner_text=>$bannertext,
    email=>$email,
    nickname=>$poster,
    category=>$postcat,
    action=>"done",
});

$formdata = $q->query_string;

$h = new HTTP::Headers(
    Accept=>'*/*',
    Referer=> $refererurl,
    user_agent=>'Mozilla/4.5',
    virtual_host=>'216.and so on,
    server_name=>'www.thesite.com',
    script_name=>'/post.cgi',
    remote_host=>'www.thesiteagain.com',
    http_user_agent=>'Mozilla/4.5',
    Content_Type=>'application/x-www-form-urlencoded'
);

$req = new HTTP::Request(
    "POST",
    $posttourl,
    $h
);

$req->content($formdata);

$ua = new LWP::UserAgent;
$ua->agent( "Mozilla/4.5" );


$response = $ua->request($req);

print $q->header;#blah blah
print $q->start_html(-script=>$JSCRIPT);

print $response->content;

print $q->end_html;

[download]
Edited by boo_radley : Closed assorted tags.

Edit kudra, 2001-09-23 Removed colours for themesafe viewing

  • Comment on LWP::UserAgent; HTTP::Headers; HTTP::Request; CGI; automated scripts
  • Download Code

Replies are listed 'Best First'.
Re: LWP::UserAgent; HTTP::Headers; HTTP::Request; CGI; automated scripts
by tachyon (Chancellor) on Sep 23, 2001 at 17:18 UTC

    If you have one method that works and one that appears the same but does not work then obviously the two methods differ in a vital detail.

    Solution: find the difference. All you need to do is capture the output of your script (what it sends) and your manual method (what it sends) and compare the two say using diff. Ideally you want to capture exactly what goes down your modem/lan to be absolutely sure. Find the difference. Avoid speculation. Answer your own question.

    It probably relates to your environment variables being different but you can find this out and I can only speculate. If you don't know how to dig into the internals of you system to do the capture a simple CGI script that spews the $ENV hash when you call it may well do the trick. Just call it both ways - it will ignore all your data any just spew the environment variables for you - if they don't match I'd start there.

    Update

    Here is a quick script that should do the trick:

    Update2

    Added sub to escape the HTML properly as noted by merlyn Thought I may as well make it vomit forth all the CGI params as well just for good measure. 

    #!usr/bin/perl -w
use strict;
$|++;
use CGI;
my $q = new CGI;
my @params = $q->param;
my ($environment, $params);

$environment .= "  <tr><td>".escapeHTML($_)."</td><td>".escapeHTML($EN
+V{$_})."</td></tr>\n" for keys %ENV;
$params .= "  <tr><td>".escapeHTML($_)."</td><td>".escapeHTML($q->para
+m($_))."</td></tr>\n" for @params;

print <<HTML;
Content-type: text/html

<html>
<head>
  <title>Spew Entire CGI Environment</title>
</head>
<body>
<h1>Environment Variables</h1>
<table border='2'>
  <tr><td bgcolor="#C0C0C0">ENV VAR</td><td bgcolor="#C0C0C0">Value</t
+d></tr>
$environment
</table>
<h1>CGI Parameters</h1>
<table border='2'>
  <tr><td bgcolor="#C0C0C0">Param</td><td bgcolor="#C0C0C0">Value</td>
+</tr>
$params
</table>
</body>
</html>
HTML

sub escapeHTML {
      local $_ = shift;
      # make the required escapes
      s/&/&amp/g;
      s/"/&quot;/g;
      s/</&lt;/g;
      s/>/&gt;/g;
      # make the whitespace escapes - not required within pre tags
      s/\t/&nbsp;&nbsp;&nbsp;&nbsp;/g;
      s/( {2,})/"&nbsp;" x length $1/eg;
      # make the brower bugfix escapes;
      s/\x8b/&#139;/g;
      s/\x9b/&#155;/g;
      # make the PERL MONKS escapes (if desired)
      s/\[/&#091;/g;
      s/\]/&#093;/g;
      # change newlines to <br> if desired - not needed with pre tags
      # s/\n/<br>/g;
    return $_;
}

    [download]

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[reply]
[d/l]
      That script fails to properly escape environment variables that contain HTML-sensitive characters, a common mistake of course. I'd stick with something this simple: 
      #!/bin/sh
echo content-type: text/plain
echo
printenv

      [download]
      Yeah, and if you're not on a system with a shell or printenv, you can do it in Perl if you insist: 
      #!/usr/bin/perl
print "Content-type: text/plain\n\n";
print "$_\t$ENV{$_}\n" for sort keys %ENV;

      [download]
      See how much simpler that is? And no problems if there's HTML involved.

      Sometimes, HTML is overkill.

      -- Randal L. Schwartz, Perl hacker

[reply]
[d/l]
[select]

        You are of course correct that I should have escaped the env variables for HTML display so I have added a sub and a credit. You will of course have to view source to get useful output from your example as it will be solidly uninteligible in a browser window where it will appear as one long string. It is short and sweet though :-)

        cheers

        tachyon

        s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[reply]
      Your update added an implementation for one escapeHTML which is completely redundant with the one (even named the name!) provided by CGI.pm!

      Simply change your calls to escapeHTML with CGI::escapeHTML, or import it with use CGI qw(escapeHTML), then remove the entire last half of your program, and you're done!

      -- Randal L. Schwartz, Perl hacker

[reply]

        As you state an escapeHTML routine is provided by CGI.pm. For the purpose which is intended (ie display plain text in a browser window in the original form) CGI.pm's routine is inadequate.

        I make this red flag to bull statement because it does not actually escape text in such a way as to render text with multiple spaces or newlines in a form that imitates the original text. Any sequences of spaces will render as a single space if you use $q->escapeHTML(). The snippet presented does the required &nbsp; subs. It offers the opportunity to convert \n to its HTML equivalent of <BR> which is potentially required. It also adds Perl Monks posting comptibility by escaping the [ and ] which thus far seems to have strangely been overlooked in CGI.pm :-)

        cheers

        tachyon

        s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

[reply]
Re: LWP::UserAgent; HTTP::Headers; HTTP::Request; CGI; automated scripts
by merlyn (Sage) on Sep 23, 2001 at 12:42 UTC
    Just a thought, but I bet your web-brower's UserAgent is not exactly Mozilla/4.5. They're probably doing a more specific match.

    -- Randal L. Schwartz, Perl hacker

[reply]
Re: LWP::UserAgent; HTTP::Headers; HTTP::Request; CGI; automated scripts
by archen (Pilgrim) on Sep 23, 2001 at 21:04 UTC
    yeah, I've had a few scripts which have been hung up because of their user agent, although sometimes I think the referrer is also catching them. I actually made a guestbook script which has been catching the user agents of everyone has been signing just so I could see what browser people used - strange that when I wanted to figure out what browser agent to use I had a catalog already at my disposal. Lately I've been mimicking IE5 
    Mozilla/4.0 (compatible; MSIE 5.0; Windows 98; DigExt

    [download]
    As far as I can tell, you've thought of everything I would have, except perhaps some hidden field, or something done with JavaScript or an applet.
[reply]
[d/l]
      I think the hidden field theory bears further investigation. There may well be a session ID buried in there. You may need to download the form page each time and capture that field so you can return it.
[reply]
Re: LWP::UserAgent; HTTP::Headers; HTTP::Request; CGI; automated scripts
by dumbadam (Novice) on Sep 23, 2001 at 13:35 UTC
    I tried Mozilla/4.0 (compatible; MSIE 6.0; Windows 98; Win 9x 4.90) but still no dice.
    What would you guys do to keep a automated script from working like what I descibed above.
[reply]
      Actually, here's my scheme (major conspiracy theory here). You have a page which loads a form. At the top is a generic advertizement which might be an advertizement (but might be an image as well). The page uses  <img src="/generic/p0rn-advertizement.cgi"> . The advertizement has a dual purpose. One is that it prints the advertizement gif. The second is that it records the user agent and other info for the GET request which is stored in a list on the server. Then for each corresponding POST it looks up that information in the list for the GET request for the advertizement. Most people couldn't enter stuff in a form fast enough for the browser to not even make the GET request, and the only time you wouldn't retrieve the image is if you turned off image loading (unlikely) or you were using a script which of course would ignore loading images. Man, that's so evil I'm surprised I thought if it.
[reply]
[d/l]
Re: LWP::UserAgent; HTTP::Headers; HTTP::Request; CGI; automated scripts
by dumbadam (Novice) on Sep 24, 2001 at 10:22 UTC
    sorry would this do the same?
    foreach $var_name ( sort keys %ENV ) {
print "<p><b>$var_name</b><br>";
print $ENV{$var_name};
}

    [download]
[reply]
[d/l]
Re: LWP::UserAgent; HTTP::Headers; HTTP::Request; CGI; automated scripts
by Anonymous Monk on Sep 24, 2001 at 10:06 UTC
    This is Adam, every time I compile the above I get this:
    Missing $ on loop variable at /www/po.cgi line 11.

    Is it these lines? 
    $environment .= "  <tr><td>".escapeHTML($_)."</td><td>".escapeHTML($EN
+V{$_})."</td></tr>\n" for keys %ENV;
$params .= "  <tr><td>".escapeHTML($_)."</td><td>".escapeHTML($q->para
+m($_))."</td></tr>\n" for @params;

    [download]
[reply]
[d/l]

Back to Seekers of Perl Wisdom