Posting form data from within a virtual browser

katgirl has asked for the wisdom of the Perl Monks concerning the following question:

This is going to take some explaining!

The place where I work has a firewall which stops me from posting on my favourite forum (Sorry, meant 2nd favourite ;) so I wrote a script that will pull the data from the forum and display it through my site, sidestepping the headers that define the forum as a forum. All ok so far?

I've managed to get the links to work (the site has relative links) but posting data to the forum is impossible - I just go back to the start page I made. (I'm not bothered about the pictures not working)

Please could you tell me:

Why I can't post (probably overlooking something obvious)
How to fix the above

(btw - I'm already working on the absolute links, so no need to tell me off about that!)


#!/usr/bin/perl -w
BEGIN {
  $|++;
}
use strict;
use lib "/usr/local/www/virtual/katgirl/modules";
use LWP::Simple qw(!head);
use CGI::Carp qw(fatalsToBrowser);
use CGI qw/:standard/;
my $url = param("url");
my $surfurl = param("surfurl");
my $surfur1 ||= $url;
#if(!$surfurl){$surfurl = $url; }
my @file;
print header,
        start_html('web getter');

if($url){

    if($url =~ m/action/ig){
        @file = post($url);
    }else{
        @file = get($url);
    }

    foreach my $line(@file){
        #chomp($line);
        $line =~ s!href=\"!href=\"?surfurl=$surfurl&url=$surfurl\/!gi;
        $line =~ s!src=\"!src=\"$surfurl\/!gi;
        $line =~ s!action=\"!action=\"?surfurl=$surfurl&url=$surfurl\/
+!gi;
        print $line;
    }

}else{

      print p("Where do you want to go today?"),
      start_form,
      textfield(-default =>"http://", -name=> "url"),
      submit(-name=> "GO!"),
      end_form;

}

print end_html;
[download]

Comment on Posting form data from within a virtual browser Download Code

Replies are listed 'Best First'.
Re: Posting form data from within a virtual browser by dws (Chancellor) on Jul 17, 2003 at 16:03 UTC
1. Why I can't post (probably overlooking something obvious)? Does the forum depend on Cookies? If so, you'll need to set up a cookie jar so that your redirector can capture them and issue them back with your post. Something like the following is the general idea: `use HTTP::Request::Common qw(GET POST); use HTTP::Cookies; use LWP::UserAgent; ... my $ua = new LWP::UserAgent(); my $cookies = new HTTP::Cookies(file => 'cookies.dat', autosave => 1); $ua->cookie_jar($cookies); # We now have a persistent cookie jar associated with the # user agent. my $req = GET "$url"; # if the response included a cookie, it'll be saved in the # jar, and resent along with subsequent GETs or POSTs to # the same domain that issued the cookie.` [download] See HTTP::Cookie for more details. Assuming the forum requires that you log in, you'll need to manually simulate the login steps so that you can capture a login cookie. WWW::Mechanize can help there.	[reply] [d/l]
Re: Posting form data from within a virtual browser by sgifford (Prior) on Jul 17, 2003 at 16:02 UTC
One thing that looks missing is passing along the POST parameters to the URL you're fetching, unless the `post` sub does that. I see that you're trying to rewrite all links to add a `surfurl` parameter to the URL. If the request is a POST instead of GET request, you may have to use a different CGI function (`url_param`) to get to the parameters from the URL. An approach you might consider is to use a Web proxy that rewrites the headers, then configuring your browser to use the proxy. Proxies are designed for this sort of thing, and IIRC there are lots of modules that do the bulk of the work.	[reply] [d/l] [select]
Re: Posting form data from within a virtual browser by mpeppler (Vicar) on Jul 17, 2003 at 16:20 UTC
In addition to the other comments - looking at LWP::Simple I don't see a `post()` subroutine in that module. So unless that is defined elsewhere in the code that you aren't showing you'll have to build something that gets the POST arguments from the browser and resends them to the real URL. I tend to use LWP::UserAgent for this, like so: `my $ua = new LWP::UserAgent; my $req = new HTTP::Request POST => "http://$url"; $req->content_type('application/x-www-form-urlencoded'); $req->content("param=data to send to the client"); # note - should be + URL encoded my $res = $ua->request($req); # Check the outcome of the response if ($res->is_success) { return $res->content; } else { return $res->error_as_HTML; }` [download]	[reply] [d/l] [select]