Hya Monks, lemme say that this is the most se.. site for all perl issues. I love it. Today i have a question on how to prepare a special thing. well days ago i needed a script - to get some data: some guys here helped me alot and now the script (see below) runs - since we have set the right parameter in the line my $ua = LWP::RobotUA->new('my-robot/0.1', 'me@foo.com'); but should i change some thing in the script in order to run the script against this site here. a forum-sitemap.php http://www.kite2fly.com/forum/sitemap.php http://www.mynak.com/forum/sitemap.php http://www.dizign.de/forum/sitemap.php well i think taht we need minor change in lwp - I need ideas how to accomplish do i have to change the agent here - or is it impossible to run automatically against (on) a forum site. imagine - a site like this here forum-sitemap.php (note - this sites are only examples!! i am not interested in those sites - but in the technique to run a useragent against a site like this) is much much more easier to take as a base-site: http://www.kite2fly.com/forum/sitemap.php http://www.mynak.com/forum/sitemap.php http://www.dizign.de/forum/sitemap.php any ideas were greatly appreciated metabo 

#!e:/Server/xampp/perl/bin/perl.exe -w 
use strict;
use CGI::Carp qw(fatalsToBrowser warningsToBrowser);
use CGI;
my $cgi = CGI->new();
print $cgi->header();
warningsToBrowser(1); # dies ist wichtig und muss nach dem Header komm
+en!
use warnings;

use LWP::RobotUA;
use HTML::LinkExtor;
use HTML::TokeParser;
use URI::URL;

use Data::Dumper; # for show and troubleshooting

my $url = "http://www.mysite.com/forums/";
my $lp = HTML::LinkExtor->new(\&wanted_links);
my $ua = LWP::RobotUA->new('my-robot/0.1', 'me@foo.com'); 
my $lp = HTML::LinkExtor->new(\&wanted_links); 



print "Content-type: text/html\n\n"; 
print "Surfer variablen ua PRINT: $ua \n"; 
print "Surfer variablen lp PRINT: $lp \n"; 

my @links; 
get_threads($url); 

foreach my $page (@links) { # this loops over each link collected from
+ the index 
   my $r = $ua->get($page); 
   if ($r->is_success) { 
      my $stream = HTML::TokeParser->new(\$r->content) or die "Parse e
+rror in $page: $!"; 
      # just printing what was collected 
      print Dumper get_thread($stream); 

print "Content-type: text/html\n\n"; 
print "surfer variablen stream PRINT: $stream \n"; 

     } else {
        warn $r->status_line;
     }
}

sub get_thread {
    my $p = shift;
    my ($title, $name, @thread);
        while (my $tag = $p->get_tag('a','span')) {
        if (exists $tag->[1]{'class'}) {
            if ($tag->[0] eq 'span') {
                if ($tag->[1]{'class'} eq 'name') {
                    $name = $p->get_trimmed_text('/span');
                } elsif ($tag->[1]{'class'} eq 'postbody') {
                    my $post = $p->get_trimmed_text('/span');
                    push @thread, {'name'=>$name, 'post'=>$post};
                }
            } else {
                if ($tag->[1]{'class'} eq 'maintitle') {
                    $title = $p->get_trimmed_text('/a');
                }
            }
        }
    }
    return {'title'=>$title, 'thread'=>\@thread};
}

sub get_threads {
    my $page = shift;
    my $r = $ua->request(HTTP::Request->new(GET => $url), sub {$lp->pa
+rse($_[0])});
    # Expand URLs to absolute ones
    my $base = $r->base;
    return [map { $_ = url($_, $base)->abs; } @links];
}

sub wanted_links {
    my($tag, %attr) = @_;
    return unless exists $attr{'href'};
    return if $attr{'href'} !~ /^viewtopic\.php\?t=/;
    push @links, values %attr;
}

[download]
well what do you think - what should i do here - to change the code in order to run against a sitemap http://www.kite2fly.com/forum/sitemap.php http://www.mynak.com/forum/sitemap.php http://www.dizign.de/forum/sitemap.php

In reply to user-agent question [light one for the quick perler] by perl_lover_girl

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.