I guess without knowing a bit more about the kind of data you seek to retrieve and how you intend to parse it, I can't give you very good advice.

But it seems to me as though you are wanting to repeatedly execute a small handful of fairly basic tasks:

  1. Connect to a website
  2. Find some information
  3. Extract that information
  4. Format the extracted information
  5. Save to a database

It seems to me that all of those methods could be part of your Robot::Base. Admittedly, you might have to subclass in some cases, but it would seem to me that the website URL, the regular expression(s) or other identifying tag names for locating the proper information, the formatting rules and the SQL statement handles could all be part of your configuration, so that you are executing the same code for each website with different configuration parameters.

Heck, the best way might be to try it out for a half-dozen of the sites, and see if you can abstract the common code and come up with a solution that is general enough -- if not, then you may end up writing distinct code for each of the 50 data sources. If you end up doing it that way, though, I would hate to be the poor guy who comes along after you and has to maintain it. :)

Fifty subclasses is beyond my limit of how granular I am willing to get ... what do you do if someone asks you to go to 100 websites, or 1000? When you find yourself writing so much code for essentially the same tasks, you have to ask yourself, "Isn't there some general solution that would save me from so much duplicate effort?". I usually ask myself that question if I have more than five or six subclasses at a particular level.


No good deed goes unpunished. -- (attributed to) Oscar Wilde

In reply to Re^3: Module hierarchy help by ptum
in thread Module hierarchy help by 2ge

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":



  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.