Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Trying to fetch web author name where I get the name of the author from my meta tags. 
<META NAME="HTML.author" CONTENT="Joe Smith">

[download]
or another page might have:
<META NAME="Company.HTML.author" CONTENT="Carl Jones">

[download]
or a page could have:
<META NAME="web.author" CONTENT="Mike Simmons">

[download]


Here is my attempt at it but I am not having any luck here: 
if($line =~ /<meta name = "\*.HTML.author" Content = (.+)>/i)
   {
         print "FILE = $File::Find::name   Author = $1\n";
   }

[download]

Replies are listed 'Best First'.
Re: Fetching meta tag info
by PodMaster (Abbot) on Nov 12, 2002 at 15:32 UTC
    You have to use the right tool for the job, here is how with one of them: 
    use HTML::TokeParser::Simple;

my $p = new HTML::TokeParser::Simple ( \ qq{
    <META NAME="Company.HTML.author" CONTENT="Carl Jones">
    <META NAME="Company.HTML.author" CONTENT="Carl Jones">
    <META NAME="web.author" CONTENT="Mike Simmons">
});

while(defined( my $t = $p->get_tag('meta') ) ) {
    my $at = $t->return_attr;
    if($$at{name} =~ /author/i) {
      print "$$at{name}=$$at{content}\n";
    }
}
__END__
Company.HTML.author=Carl Jones
Company.HTML.author=Carl Jones
web.author=Mike Simmons

    [download]
    update:
    That's cause HTML::TokeParser::Simple is sooo easy, i'm able to write gems like these in under a minute ;)

    ____________________________________________________
    ** The Third rule of perl club is a statement of fact: pod is sexy.

[reply]
[d/l]

      Darn! You beat me to it! Here is what I came up with, using HTML::TokeParser, using Perl & LWP as a reference:

      #!/usr/bin/perl -w
use strict;

use HTML::TokeParser;

$/="\n\n";

while( my $html=<DATA>)
  { my $author= get_author( $html);
    print "Author: $author\n";
  }
exit;

sub get_author
  { my $html= shift;
    my $stream= HTML::TokeParser->new( \$html);
    while( my $token= $stream->get_token)
      { my $token_type= shift @$token;
        next unless $token_type eq 'S';
        my( $tag, $attribute_hashref, $attribute_order_arrayref, $sour
+ce)= @$token;
        next unless $tag eq 'meta';
        if( grep { m{\.author$} } values %$attribute_hashref)
          { return $attribute_hashref->{content}; }
      }
    return 'unknown';
  }

__DATA__
<html>
  <head>
    <META NAME="HTML.author" CONTENT="Joe Smith">
    <META NAME="whatever" CONTENT="foo">
  </head>
  <body><p>dummy</p></body>
</html>

<html>
  <head><meta name="Company.HTML.author" content="Carl Jones"></head>
  <body><p>dummy</p></body>
</html>

<html>
  <head><META NAME="web.author" CONTENT="Mike Simmons"></head>
  <body><p>dummy</p></body>
</html>

<html>
  <head>
    <META NAME="whatever" CONTENT="foo">
  </head>
  <body><p>dummy</p></body>
</html>

      [download]
[reply]
[d/l]
Re: Fetching meta tag info
by robartes (Priest) on Nov 12, 2002 at 15:35 UTC
    Just to clarify why your regexp is broken: you escaped the asterisk with a backslash, so you are looking for a literal asterisk in $line, which isn't there.

    Your regexp is quite brittle however: it fails to match your third example and any variation in the amount of whitespace in the meta tag will defeat it. You're better off using any of the examples given in the other answers.

    CU
    Robartes-

[reply]
[d/l]
Re: Fetching meta tag info
by mirod (Canon) on Nov 12, 2002 at 16:36 UTC

    I think it is worth adding to this thread a quick explanation on why regular expressions are a very risky way to process HTML (as well as XML).

    HTML is a very complex standard, the rules about what is a tag, how they can be ommited and infered, about whitespaces etc... are really complex.

    Here are a list of proper meta tags that "naive" or even quite advanced regular expressions would fail to match (all are correctly parsed by HTML::TokeParser and I guess by HTML::TokeParser::Simple):

    <meta  NAME="HTML.author" CONTENT="Joe Smith">    <!-- 2 spaces before
+ NAME -->
<meta NAME=HTML.author CONTENT="Joe Smith">       <!-- no " around the
+ value of NAME -->
<meta NAME="HTML.author" CONTENT='Joe Smith'>     <!-- you can use ' i
+nstead of " -->
<meta CONTENT="Joe Smith" NAME="HTML.author">     <!-- the order of at
+tributes is not significant -->
<meta NAME="HTML.author"
      CONTENT="Joe Smith">                        <!-- tags can span a
+ccross several lines-->
<meta NAME="HTML.author" CONTENT="Joe Smith" />   <!-- XHTML style -->

    [download]

    I am sure other examples could be found.

    And of course you might grab things that look like a meta tag but are not, like:

    <!-- <meta NAME="HTML.author" CONTENT="Joe Smith"> --> <!-- yep, a comment -->
[reply]
[d/l]
[select]
      I think it is worth adding to this thread a quick explanation on why regular expressions are a very risky way to process HTML (as well as XML).

      <many excellent examples SNIPPed>

      Al the above is absolutely true for arbitrary webpages - as in the problem you repsonded to.

      However one of the more common LWP etc. tasks is to retrieve and search for/extract information from the web front ends of query tools (such as Pubmed or Scirus (or google although google doesn't like LWP)) or other template built webpages (such as this one). This is where the recent micro-optimisation comments apply. In such a case regexes or even simpler index($html, $string) statements are much faster and consume fewer resources. The difference can mean that your script can run OK on an ancient PC versus requiring oodles of processing power/memory.

      Moreover for that kind of operation navigating the complex hashes and hashes of arrays of hashes etc. produced by HTML::TokeParser::* and relatives is probably harder to maintain in the event of layout changes than the regex method. Consider what happens when a TABLE is replaced by a set of positioned DIVs and ULs for example, but all you want is to see if "Fred Bloggs" appears anywhere in the results section.

      Summary: If you want to get a small amount of information from a particular tag or tags from arbitrary HTML then TokeParser is definitely the way to go. On the other hand if you want to extract large amounts of content from a series of similar pages then regexes are not a bad way to do things.

      Dingus

      Enter any 47-digit prime number to continue.
[reply]
Re: Fetching meta tag info
by neilwatson (Priest) on Nov 12, 2002 at 15:28 UTC
[reply]
[d/l]
      It it possible to have spaces besides equel sign, and quotes isn't mandatory in some cases. So, I suggest the following:
      if($line =~ /<meta name\s*=\s*"?.*?author"?\s+content\s*=\s*"?([^>"]*)
+"?/i)

      [download]
[reply]
[d/l]
        Thanks!

        ? I am Lost on what the numberous question marks are doing in this req expression?
[reply]
Re: Fetching meta tag info
by gnangia (Scribe) on Nov 12, 2002 at 20:08 UTC
    You can use HTTP::Headers or HTML::Parser Modules to extract the data. Using Regular expressions can get very messy and is prone to errors.
[reply]

Back to Seekers of Perl Wisdom