Re^3: easy HTML::TokeParser help request

If you're dead set on tokeing it, build a state machine:

start in looking_for_full until you see a div with class full, when you transition to looking_for_content
when you see a div with class content in state looking_for_content, transition to looking_for_anchors
when you see an anchor in looking_for_anchors, save the href attribute
when you see a </div> in looking_for_anchors, go back to looking_for_full

Additional: Little note on implementation: you'd have a $state variable which keeps track of which state you're in (start with my $state = 'looking_for_full';). You'd then have a while( my $t = $stream->get_token ) { ... } loop, inside of which you'd implement the above behaviors. Any non-interesting token for the current state would be ignored (e.g. just next back to fetch the next token).

Comment on Re^3: easy HTML::TokeParser help request Select or Download Code

Replies are listed 'Best First'.
Re^4: easy HTML::TokeParser help request by 2ge (Scribe) on Aug 03, 2006 at 14:21 UTC
Thanks for nice explanation, Fletch. I think this is how it should be done. But...In real life is easier to make regexp for that content I want to parse, so when doesn't exist other simpler solution for HTML::TokeParser, I have to pick up regexs. Also, which parser is better ? HTML::TreeBuilder or ? I want learn only one, which is able to parse these relative easy things and has no other glitches...	[reply]
Re^5: easy HTML::TokeParser help request by Fletch (Bishop) on Aug 03, 2006 at 14:42 UTC
If you really have a fixed format that you can guarantee isn't going to change (e.g. this is a one-off throw away program to convert old data into a new format), sure go ahead and use regexen. Otherwise you'll find out n months down the road that you're going to spend the time again re-implementing it when the HTML changes because the designer got a new version of Dreampage 06 X. As for which parser is better: depends. Which is better, a Ferrari or a heavy duty pickup? Try moving a couple palates of bricks with the former, or winning a race with the later. In my personal experience the answer is: depends. :) I used to use TokeParser more than TreeBuilder (writing my own RSS feeds before sites provided them themselves), but more often that's now the other way around. As you can see, for a task with more context sensitivity (`foo` elements 2 levels down inside `bar` elements) it's more scaffolding from the programmer to do things with TokeParser than with a tree. But there's other types of tasks (extract any `foo` elements with class `zorch`) that'll probably be simpler to think of in the TokeParser manner. If you haven't looked at it you should also take a gander at HTML::TokeParser::Simple which provides an even nicer token interface.	[reply]
Re^6: easy HTML::TokeParser help request by 2ge (Scribe) on Aug 04, 2006 at 12:20 UTC
I think, when we are talking about changing format, in many times there is also needed change also program, not only regexp. Now it is ok, assume it will not change. Thanks for nice answer about comparing TokeParser and TreeBuilder, that is really enough for me and I see the difference. Maybe there should be some other module putting those two propertie together (take token and have also tree stored somewhere). Dont know if it is possible. The main thing is - problem is solved, and I hope this node will help also to other monks!	[reply]