in reply to RFC: URI::URL::Detail

I wanted to propose HTML::LinkExtract as a name, but then I found HTML::LinkExtor and HTML::LinkExtractor - maybe you can build it on top of one of those modules, or maybe extend them, or maybe they already do what you want?

"Clean" a URL so it can be used as a string ( in say Regular expressions or MySql insert statements ).

I don't think you need that. For regular expressions you just use /\Q$url\E/ or quotemeta, and for SQL inserts you should use placeholders anyway, no need to escape or clean anything.

HTML::TreeBuilder - Overkill?

It is never overkill to use a proper HTML parser for such a task. I don't know if that's the best for this task, but it should certainly work.

Replies are listed 'Best First'.
Re^2: RFC: URI::URL::Detail
by tmharish (Friar) on Aug 07, 2009 at 14:09 UTC
    Thanks moritz.
    I wanted to propose HTML::LinkExtract as a name, but then I found HTML::LinkExtor and HTML::LinkExtractor - maybe you can build it on top of one of those modules, or maybe extend them, or maybe they already do what you want?

    I believe HTML::LinkExtract does a fair bit of what I am proposing, but I would like to provide for some additional functionality and subtly different access methods.

    I find that these access methods make the writing of a crawler slightly easier.

    Maybe it could be "HTML::LinkExtor::Simple" or "HTML::LinkExtor::Spider". I am not sure if it should be altogether different and be named "HTML::Spider::LinkExtor".

    Also I am not sure I understand the difference between extend and build on top of - If I am looking at say "HTML::LinkExtor::Simple" should I get in touch with the author of "HTML::LinkExtor"?

    With regard to the rest (Clean and TreeBuilder) - it makes perfect sense, Thanks for that.

[reply]
      Also I am not sure I understand the difference between extend and build on top of

      That wasn't very precise of me. What I meant was either to inherit from the classes and add your own methods (extending), or use it as a backend but using your own API instead (building on top).

      If I am looking at say "HTML::LinkExtor::Simple" should I get in touch with the author of "HTML::LinkExtor"?

      Yes.

[reply]

In Section Meditations