Beefy Boxes and Bandwidth Generously Provided by pair Networks
Think about Loose Coupling

Link Parser, something to be desired?

by koolgirl (Hermit)
on May 29, 2009 at 21:09 UTC ( #766957=perlquestion: print w/replies, xml ) Need Help??

koolgirl has asked for the wisdom of the Perl Monks concerning the following question:

Hello Fellow Monks,

Long time since I've posted, lo and behold this time I have a question dealing with an actual program I've been asked to write, rather than an exercise. I'm writing a parser to take any random web page, and deaden all the links. This is the code:

#!usr/bin/perl use strict; my $file = "wiki.html"; my $file2 = ""; my $regex = "href\s*=\s*\"?.*\""; my $sub = "href=\"#\""; my $match; my $before; my $after; my $line; open(IN, "$file") || die $!; open(OUT, ">$file2") || die $!; while (<IN>) { chomp($line = $_); if ($line =~ /$regex/) { print "Got a Match\n"; $before = $`; $match = $&; $after = $'; $line =~ s/href\s*=\s*\"?.*\"/$sub/g; print OUT $line . $after . "\n"; #print OUT $' . "\n"; } else { # do nothing } # end if } # end while close(IN); close(OUT);

On this line:

$line =~ s/href\s*=\s*\"?.*\"/$sub/g;
I originally had
$line =~ s/$match/$sub/g;
But that would leave certain links unchanged. Mind you, my original regex catches everything I'm looking for, I think, however this substitution operator was my problem, and I ended up having to put the original expression in place of the variable that contained the match. Theoretically this makes no sense to me, and I'm thinking if I understood the reason behind this I might better understand the problem I had.

Also, please note, I saved the before and after of all the matches for later code I need to put together, not just to waste memory ;). Anyway, can anyone explain why I couldn't use $match in the line I pointed out? I'm not trying to get anyone to do my work for me, I'm only trying to understand what the problem was I just created a hack to fix.

Thanks :)

UPDATE:Thanks everyone for your very helpful replies, through which I have gained some great resources both for this project and future parsing projects. I would like to make one quick point, however, it turns out that this really is just a quick "one-off" (which YourMother pointed out as a slight possibility), so it wasn't really necessary to be dealing with a module, it isn't a program I'm planning to maintain for a long period of time, just something I wanted to be able to do quickly for a live application demo, something I could verify easily with my own two eyes, and also something that while perhaps better done with an alternative method, it is certainly enhancing my Perl skills, programming skills and introducing me to the world of parsing, therefore being more beneficial for me, in the long run, to do it in this manner, rather than those mentioned, IMHO, of course.

That being said, I am nonetheless planning to go and explore all of the modules mentioned by my fellow monks, momentarily much farther advanced in the Force (or rather, the Source) than myself, expanding the learning aspect of this project to even greater limits, so as I said in the beginning, Thank You very much to all of the monks who have taken their very precious time and effort to offer me a push in the right direction ;) ( also, fixed a typo )

Replies are listed 'Best First'.
Re: Link Parser, something to be desired?
by planetscape (Chancellor) on May 29, 2009 at 23:43 UTC
Re: Link Parser, something to be desired?
by zwon (Abbot) on May 29, 2009 at 22:50 UTC


    It seems doesn't work.

    my $regex = "href\s*=\s*\"?.*\"";

    I think it should be

    my $regex = qr{href\s*=\s*".*?"};
    Note qr// and question mark position. So perhaps your code should look something like this:
    use strict; use warnings; # this is also useful my $regex = qr{href\s*=\s*".*?"}; my $sub = "href=\"#\""; while(<DATA>){ s/$regex/$sub/g; print; } __DATA__ afjalsdfj href="asfasdfa" afdsas href="akjshfakjsd" href = "ajsfhaklj"

    Update: and here's another approach using HTML::Parser:

Re: Link Parser, something to be desired?
by Your Mother (Archbishop) on May 30, 2009 at 06:13 UTC

    XML::LibXML example. Terse and robust. Don't reach for regexes for HTML unless it's an instant one-off you'll be able to verify by eye. Parsers require little more effort and are much more reliable.

    use strict; use warnings; use XML::LibXML; my $parser = XML::LibXML->new(); $parser->keep_blanks(1); $parser->recover_silently(1); # There are other parse methods: string, fh. my $dom = $parser->parse_html_file(shift || die "give a file\n"); $_->setAttribute("href", "#") for $dom->findnodes('//a[@href]'); print $dom->serialize(1);
Re: Link Parser, something to be desired?
by ig (Vicar) on May 29, 2009 at 23:17 UTC

    Escapes within double quotes can be tricky. I suspect what you want is one of the following:

    $regex = "href\\s*=\\s*\".*?\""; $regex = 'href\s*=\s*".*?"';

    It might help you to print $regex and $match, to see what you are getting.

    update: You can use $match in the substitution (at least it worked fine in my tests) but it won't do what you expect if $regex isn't what you expect. Also, since $match is one particular match, adding the global qualifier to the substitution using $match isn't likely to replace multiple instances.

    You have indicated that you have more to do than just substitute the href values in the anchors. Without knowing more about what you are doing it is hard to make an appropriate recommendation but something like the following might be closer to what you want:

    use strict; use warnings qw( FATAL all ); my $regex = 'href\s*=\s*".*?"'; print "regex = $regex\n"; foreach my $line (<DATA>) { chomp($line); while ($line =~ /$regex/) { my $match = $&; $line =~ s/$match/xxx/; } } __DATA__ No anchors here <a href="asdf">go there</a> <a href="fdsa">go here</a>

      Escapes within double quotes can be tricky.

      qr// makes it easy.

      $regex = qr/href\s*=\s*".*?"/;

      while ($line =~ /$regex/) { my $match = $&; ... }
      can be better written as
      while (my ($match) = $line =~ /($regex)/) { ... }
      since it avoids globals and $&, which slows down matches throughout your program.

Log In?

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://766957]
Approved by zwon
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others about the Monastery: (7)
As of 2022-08-08 07:23 GMT
Find Nodes?
    Voting Booth?

    No recent polls found