#!/usr/bin/perl -w·
use strict;
use HTML::TokeParser::Simple; 

my $html = q(
<option value="STAFE">STAFE - 900 - BEN PROG - Food Assistance</option
+>
<option value="STAM7">STAM7 - 900 - BEN PROG - Med Asst - Lynchbrg</op
+tion>
<option value="STAM8">STAM8 - 900 - BEN PROG - Med Asst - Marion</opti
+on>
<option value="STAM9">STAM9 - 900 - BEN PROG - Med Asst - Petrsbrg</op
+tion>
<option value="STAMA">STAMA - 900 - BEN PROG - Medical Assistance</opt
+ion>
<option value="STATA">STATA - 900 - BEN PROG - Economic Assistance</op
+tion>
<option value="STATR">STATR - 900 - BEN PROG - Training Development</o
+ption>);
my $p = HTML::TokeParser::Simple->new(\$html);
while(my $token = $p->get_token){
  if($token->is_text){
  my $text = $token->return_text;
  next unless $text =~ /\S/;
  print "[$text]\n";
  }
}
[download]

produces as output:

[STAFE - 900 - BEN PROG - Food Assistance]
[STAM7 - 900 - BEN PROG - Med Asst - Lynchbrg]
[STAM8 - 900 - BEN PROG - Med Asst - Marion]
[STAM9 - 900 - BEN PROG - Med Asst - Petrsbrg]
[STAMA - 900 - BEN PROG - Medical Assistance]
[STATA - 900 - BEN PROG - Economic Assistance]
[STATR - 900 - BEN PROG - Training]
[ Development]
[download]

I have no idea why the "Development" ends up on it's own line. This is the smallest sample from my data that gave these results -- adding more data "moves" the problem, but the problem still exists. Taking out the space between "Training" and "Development" in the data makes the new compound word the one that goes to its own line.

It's acting as if some buffer length is interfering, but it isn't just the token length (making a longer text token will change the position of the problem, but it doesn't necessarily hit the longest token -- in fact, in this sample set, it continues to hit the last text token somewhere.)

I've skimmed the docs for HTML::Parser, (I've used 3.25 and 3.26) HTML::TokeParser(2.24) , and HTML::TokeParser::Simple(1.4). I've tried in on different boxes. (one aging SuSE box with 5.6.0, one Debian unstable with 5.8.0) any ideas?