For the parsing part I would use HTML::TokeParser::Simple, wrote by our brother Ovid; here it is an example from the documentation:

use HTML::TokeParser::Simple;
 my $p = HTML::TokeParser::Simple->new( $somefile );

 while ( my $token = $p->get_token ) {
     # This prints all text in an HTML doc (i.e., it strips the HTML)
     next unless $token->is_text;
     print $token->as_is;
 }
[download]

Wow. That is sweet. I'll post my implementation of it when I get a chance. Still, any thoughts on the multiple-file issue, or do you think this will address that too?

• Hmm... instead of using <> to run through the files, you could do something like:

  foreach my $in_fname (@ARGV) {
    ...
  }
  [download]

• As for a regex, you could do:

    $input_txt =~ s/\<script.*?\<\/script\>//igs; # remove code
    $input_txt =~ s/\<.*?\>//gs; # remove html tags
    $input_txt =~ s/\&nbsp;/ /gs; # &nbsp -> space
    $input_txt =~ s/\s{2,}/ /gs; # Remove extra spaces
    print $input_txt;
  [download]

  
  It's not valid HTML to have <> elements inside of other HTML tags (they should be entity-encoded as in:

    <a href="" onClick="alert('&lt; foo! &gt;');">
    not:
    <a href="" onClick="alert('< foo! >');"> Invalid HTML!
  [download]

  However, if they're there, then you can do the regex inside a while loop:

    while( $input_txt =~ s/\<.*?\>//gs ) {};
  [download]

• Using modules is probably better. Every time you reinvent the wheel, somewhere, a monk sheds a tear...

instead of using <> to run through the files, you could do something like:

That's excellent. I'll give that a shot. But can you tell me why the camel book says to do it this way if it doesn't really work?

For the regex part, you're right that that would work for standard HTML, but I have to parse JSP pages too, and they are allowed to nest tags like that. I'll try your while loop and see how that works, though.

As for the sadness of monks, I apologize. The reason I did it this way was to familiarize myself with syntax, standard perl functions, and little gotchas that are inherent in any language. Once I feel proficient with this skill, I'll be happy to keep you guys smiling :)

The syntax

while( <> ) {
}
[download]

Will 'automagically' run through every line of each of the files the user specified on the command line. It is the equivalent of:

  foreach $ARGV (@ARGV) {
    open( THISFILE, $ARGV ) || next;
    while( $_ = <THISFILE> ) {
    }
  }
[download]

Which is why your script was executing many times per file. UNIX programs traditionally don't automatically create new files from their input, they just slurp in all the input files they're given and then print output directly to STDOUT so it can be redirected. Consider:

  $ grep 'foo' thisfile.txt thisOtherFile.txt
[download]

Which will run through all the files it's given and print matching lines. Perl allows the programmer to do this kind of task easily with the <> syntax.

if($#ARGV < 0){
    die "You did not specify any files to process! : $!\n";
}
[download]

oh, i thought that would give me info about what caused it to die. Now that I looked it up again, do you think $@ is really what I should use?

No. Why would it be? There's no system error. There's merely a usage error.

You didn't make a system call that failed ($!) and there's no caught exception ($@). Your program didn't get called with any arguments, which your code detects itself, and there's no more information to provide about the source of the error--you know exactly what the source of the error is!

Now I prefer to write that code as:

die "No files to process\n" unless @ARGV;

... mostly because I prefer the boolean check of the number of elements in @ARGV to checking the number of the last array index.

My new source is below. This works fine on HTML, but not so great on the JSP. I'll get to work on that part myself. Thanks to everyone!


use strict;
use HTML::TokeParser::Simple;

if($#ARGV < 0){
    die "You did not specify any files to process! : $@\n";
}

foreach my $infname (@ARGV)
{
    my $outfname = $infname.".txt";
    
    my $inputtxt = HTML::TokeParser::Simple->new($infname);
    my $outputtxt = "";

    #this section removes the code   
    while(my $token = $inputtxt->get_token){
        next unless $token->is_text;
        $outputtxt.= $token->as_is;
    }
    
    #this section removes whitespace
    $outputtxt =~ s/&nbsp;/ /g;  #HTML special space char
    $outputtxt =~ s/\s\s\s//mg;  #tabs (mostly) and newlines
    
    open (OUTPUT, ">>$outfname") or die "$outfname could not be opened
+.: $@ *_* $!\n";
 
    print OUTPUT $outputtxt;

    close $infname;
    close $outfname;
}
[download]