I know this is PERLmonks but if you are adamant about not writing this thing in it's entirety, why not cron + short shell script running wget? wget offers many switches that can make your life easier such as checking timestamps and file sizes before downloading. And if you can do it via SSH+RSA keys, there is rsync which is even better on only downloading new or modified files. Another possibility I can think of is CFEngine. It can download and execute files across countless systems. Hopefully they've fixed some of their file-transfer problems since I last used them.

As far as slicing-n-dicing(tm) your data into your database. Unless you already have a program to do this, you will probably need to write a script yourself. Use DBI or if you are unfamiliar with it and in a hurry, just have your script dump the required SQL to a file or the sql client directly.

Bottom line is, and you basically said it yourself, the systems out there are overkill for your needs because software tends to be generalized to meet the needs of many. Since your needs are narrow and specialized it's tough to find a system that fits and you either have to wrangle one of these "big" systems to your needs, or put together your own.

Good Luck

In reply to Re: cron plus? by pzbagel
in thread cron plus? by glwtta

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.