in reply to Re^2: Module hierarchy help
in thread Module hierarchy help

I guess without knowing a bit more about the kind of data you seek to retrieve and how you intend to parse it, I can't give you very good advice.

But it seems to me as though you are wanting to repeatedly execute a small handful of fairly basic tasks:

  1. Connect to a website
  2. Find some information
  3. Extract that information
  4. Format the extracted information
  5. Save to a database

It seems to me that all of those methods could be part of your Robot::Base. Admittedly, you might have to subclass in some cases, but it would seem to me that the website URL, the regular expression(s) or other identifying tag names for locating the proper information, the formatting rules and the SQL statement handles could all be part of your configuration, so that you are executing the same code for each website with different configuration parameters.

Heck, the best way might be to try it out for a half-dozen of the sites, and see if you can abstract the common code and come up with a solution that is general enough -- if not, then you may end up writing distinct code for each of the 50 data sources. If you end up doing it that way, though, I would hate to be the poor guy who comes along after you and has to maintain it. :)

Fifty subclasses is beyond my limit of how granular I am willing to get ... what do you do if someone asks you to go to 100 websites, or 1000? When you find yourself writing so much code for essentially the same tasks, you have to ask yourself, "Isn't there some general solution that would save me from so much duplicate effort?". I usually ask myself that question if I have more than five or six subclasses at a particular level.


No good deed goes unpunished. -- (attributed to) Oscar Wilde

Replies are listed 'Best First'.
Re^4: Module hierarchy help
by 2ge (Scribe) on Jul 27, 2006 at 14:30 UTC

    Thanks for long reply,

    yes, ideal situation would be have those informations in one config. But I really can't imagine such a config, I will give you explanation: we have 50 different websites - I will compare 2 of them.

    On 1st website is something like catalog, I have to click on category links, later click to sub-category, there should be paging (or not), clicking to detail, there should be more pages with details, parse all info I need, and save it to DB.

    On 2nd website I have to simulate queries, so it is totally different as first, I get some list, clicking on that list, get other list, clicking and so on...

    So creating general config like this and making all those routines in Baseclass seems nearly same to me. Problem is, there is really no same navigation nowhere and data is always stored other as on any page. I'd like to see how such a config should look like, never seen that. I know this would be ideal, but...Could you show me such a config, please ?

[reply]

      Have ye considered the wonders that WW::Mechanize would do for your coding? It's a module to simulate a browser for purposes of interacting on a website. With the help of HTTP::Recorder, you can even do it once manually, and have the module build the WWW:Mechanize script you need for your apps.

      Based upon what you've written, I'd strongly consider those options, and doing some research upon them. The docs I've linked to above, plus Googling for the module names, will lead to a wealth of resources.

      Think that might help remove a layer or two of work and code for you?

      ----Asim, known to some as Woodrow.

[reply]
        thanks, I am already using WWW::Mechanize(::Cached), doing by hand, no need for HTTP::Recorder. This is not about downloading itself (BASE class for sure), but which links to visit, every web has another hierarchy and thats the problem. It is something like you want to parse news from webs: yahoo news, bbc news, google news and so on.
[reply]

In Section Seekers of Perl Wisdom