(jeffa) Re: Reading PerlMonks offline

Hello. I wrote this script a few months ago - it is ugly and nasty and doesn't even grab the author's name. But it works ;) Just pipe the output to a file and load that file in your browser.

use LWP;
use HTTP::Request::Common;

use strict;
use constant URL => 'http://perlmonks.org/index.pl';

$|++;

my $node = shift;
print "USAGE: $0 [node_id]\n" and exit unless $node;

my $ua = LWP::UserAgent->new;
$ua->agent('node_grabber/1.0 (' . $ua->agent .')');

my $request  = POST(URL, Content => [ node_id => $node ] );
my $response = $ua->request($request) or die "can't download id $node"
+;
my $html     = $response->content();
my ($date)   = $html =~ /a>\s*on\s*([^<]+?)<\/font/i;
my ($title)  = $html =~ /<title>([^<]+)<\/title>/;

my $chunk;
my @ends = (
   qr|<BR>\s*<hr\s*\/>|,
   qr|<BR>\s+<BR><font size=2><I>|,
   qr|<CENTER>\s+back to <A HREF="|,
   qr|<BR>go see more <A HREF="|,
   qr|<center><TABLE width=|,
); 

foreach (@ends) {
   ($chunk) = $html =~ /<INPUT type=hidden name=op value=vote>(.*)(?:$
+_)/ms;
   last if $chunk;
}

unless ($chunk) {
   ($chunk) = $html =~ /<!--\s+-->(.*)(?:<!--\s+-->)/ms;
}

print "couldn't parse it :(\n" and exit unless $chunk;
print "$title [$date]\n", $chunk;
[download]

I orginally tried to solve this problem with HTML::TokeParser but failed miserably. So, i just used some very fragile regexes instead. If anyone feels the need to improve this, then please be my guest. :)

P.S. an RSS feed that grabbed just a node sure would be nice. :)

UPDATE (April 2,2002): Kanji just /msg'ed me about displaytype=xml (i could have sworn that this was broken some time ago). This will make the above code not only overkill, but just plain silly. I'll be working on a replacement in the meantime this morning. And there it is below ---> ;)

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

Comment on (jeffa) Re: Reading PerlMonks offline Download Code

Replies are listed 'Best First'.
XML Node Grabber (was (jeffa) 2Re: Reading PerlMonks offline) by jeffa (Bishop) on Apr 02, 2002 at 16:01 UTC
This is soooo much better. It should last a lot longer without maintenance as well. use LWP; use XML::Simple; use HTTP::Request::Common; use strict; use constant URL => 'http://perlmonks.org/index.pl'; my $node = shift; print "USAGE: $0 [node_id]\n" and exit unless $node; my $ua = LWP::UserAgent->new; $ua->agent('node_xml_grabber/1.0 (' . $ua->agent .')'); my $request = POST(URL, Content => [ node_id => $node, displaytype => 'xml', ]); my $response = $ua->request($request) or die "no download id $node"; my $content = $response->content(); my $xml = XMLin($content) or die "xml error!"; my $date = scalar localtime($xml->{ucreatetime}->{content}); print <<EOF; $xml->{title}->{content} by [$xml->{author_user}->{content}] on $date $xml->{doctext}->{content} EOF [download] Big thanks to Kanji :) jeffa L-LL-L--L-LL-L--L-LL-L-- -R--R-RR-R--R-RR-R--R-RR B--B--B--B--B--B--B--B-- H---H---H---H---H---H--- (the triplet paradiddle with high-hat)	[reply] [d/l]

Replies are listed 'Best First'.

XML Node Grabber (was (jeffa) 2Re: Reading PerlMonks offline)
by jeffa (Bishop) on Apr 02, 2002 at 16:01 UTC

use LWP;
use XML::Simple;
use HTTP::Request::Common;

use strict;
use constant URL => 'http://perlmonks.org/index.pl';

my $node = shift;
print "USAGE: $0 [node_id]\n" and exit unless $node;

my $ua = LWP::UserAgent->new;
$ua->agent('node_xml_grabber/1.0 (' . $ua->agent .')');

my $request  = POST(URL, Content => [
   node_id     => $node,
   displaytype => 'xml',
]);

my $response = $ua->request($request) or die "no download id $node";
my $content  = $response->content();
my $xml      = XMLin($content) or die "xml error!";
my $date     = scalar localtime($xml->{ucreatetime}->{content});

print <<EOF;

$xml->{title}->{content} by [$xml->{author_user}->{content}]
on $date

$xml->{doctext}->{content}
EOF
[download]

Kanji

jeffa

L-LL-L--L-LL-L--L-LL-L--
-R--R-RR-R--R-RR-R--R-RR
B--B--B--B--B--B--B--B--
H---H---H---H---H---H---
(the triplet paradiddle with high-hat)

[reply]
[d/l]