Here's your first bit of help: I've added use strict; and use warnings; near the top and fixed the resultant errors/warnings. I've run your code through perl tidy (a few times) and fixed up some other errors introduced. I fixed a few quoting and commenting issues introduced possibly by cut-n-paste errors, but definitely by your broken database connection string, which I fixed. I've also terminated your while and for loops (near the end) which may or may not be the right spot to terminate them - I can't actually fully run your code since I don't have all the modules installed nor a database handy at the moment.

I have run it once and passed it a base URL, it spit out a few of the links on the page, so I suppose it's doing something properly.

Try running this modified version and see what happens. If you make any changes, please format it for readability before posting again.

#!/usr/bin/perl use strict; use warnings; use LWP::UserAgent; use HTML::LinkExtor; use URI::URL; use DBI(); my $url = <>; # for instance #my $depth = 0; my @link = (); my $ua = LWP::UserAgent->new; # Set up a callback that collect li +nks my @a = (); sub callback { my( $tag, %attr ) = @_; return if $tag ne 'a'; push( @a, values %attr ); } # Make the parser.Unfortunately, we don't know the base yet (it migh +t be #diffent from $url) my $p = HTML::LinkExtor->new( \&callback ); my $res = $ua->request( HTTP::Request->new( GET => $url ), sub { $p->parse( $_[0] ) } ) ; # Expand all image URLs to absolute ones my $base = $res->base; @a = map { $_ = url( $_, $base )->abs; } @a; # Print them out print join( "\n", @a ), "\n"; my $dbh = DBI->connect( "DBI:mysql:database=gatxp;host=\"\"", "", "" ) +; #$dbh->do(" CREATE TABLE newlinks( md5 INTEGER(100) not null " # ."primary key, webpage VARCHAR(80) not null) "); $dbh->do(" INSERT INTO newlinks VALUES( 'MD5', '0', '$base', '1' ) "); foreach $a (@a) { $dbh->do(" INSERT INTO newlinks VALUES( '', '1', '$a', '0' ) "); } my $sth = $dbh->prepare('SELECT * FROM newlinks') or die " Couldn't prepare statement : " . $dbh->errstr; $sth->execute(); while( my $ref = $sth->fetchrow_hashref() ) { my $link = $ref->{'webpage'}; foreach $link (@link) { my $usa = LWP::UserAgent->new; $p = HTML::LinkExtor->new( \&callback ); my $res = $usa->request( HTTP::Request->new( GET => $link ), sub { $p->parse( $_[0] ) } ); $base = $res->base; @link = map { $_ = url( $_, $base )->abs; } @link; # Print them + out print "$$link \n "; } } $sth->finish(); $dbh->disconnect();

HTH



--chargrill
s**lil*; $*=join'',sort split q**; s;.*;grr; &&s+(.(.)).+$2$1+; $; = qq-$_-;s,.*,ahc,;$,.=chop for split q,,,reverse;print for($,,$;,$*,$/)

In reply to Re^5: How to extract links from a webpage and store them in a mysql database by chargrill
in thread How to extract links from a webpage and store them in a mysql database by syedahmed.uos

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.