anush81 has asked for the wisdom of the Perl Monks concerning the following question:

Hi, I have a Java servlet open and I need to screen scrap it every 10 mins. Can anyone help me with it? I tried using LWP::UserAgent and it gave me "400 URL must be absolute" error. Anusha

Replies are listed 'Best First'.
Re: Screen scraping
by theorbtwo (Prior) on Aug 11, 2009 at 06:37 UTC

    Have you tried giving it an absolute URL? "I got an error" is not a strange state of affairs. If you read it, with the assumption that it is meant to be useful, and looking up terms you don't understand, then you will often find that the error message tells you everything you need to solve it.


    Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).

Re: Screen scraping
by Anonymous Monk on Aug 11, 2009 at 06:40 UTC
    #!/usr/bin/perl -- use strict; use warnings; use WWW::Mechanize; my $ua = WWW::Mechanize->new( autocheck => 1, ); $ua->get('nice try'); __END__ Error GETing nice%20try: URL must be absolute at - line 9
Re: Screen scraping
by anush81 (Initiate) on Aug 11, 2009 at 07:19 UTC
    At this moment, I am struck at another point. I am screen scraping a GUI. This is the code I have till now. #!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; my $url='file:///C:/Program%20Files/abcde/index.html'; my $obj = LWP::UserAgent->new; $obj->credentials( '192.0.0.0', '12345', 'admin' => 'admin' ); my $resp = $obj->get($url); print "SUCCESS ", $resp->content, "\n\n"; I need to navigate to the middle of the menu. Is it possible to do that in Perl? Thanks, Anusha
        The problem is I have an User Interface in servlet. I have my Menu bar (Navigation toolbar) like
        | |-"System" |-"Alarms" |-"Trunks" |-"Resources" |-"Networking".
        I want to navigate to "Resources" and that will display a screen. I have to grap values in that screen. This is what I am intending to do... Any help??? Thanks, Anusha