I have a local table with thousands of records, each one of these records is associated with a .jpg file on a remote system that is available via ftp only. Each of these records has a unique identifier that is several digits (list_no).

The ftp site is divided into 1000 directories which correspond to the last three digits of the list_no. For example,
list_no         ftp directory
20039467    photos/467/20039467.jpg
20039512    photos/512/20039512.jpg


The information in the table gets updated every night, so the images that I need to pull changes daily as well.

I am using Net::FTP to grab the files. What I'd really like to do is only grab the files that I need - which is to say that if I downloaded it yesterday, skip it today.

My thought was to query out the last three digits and the list_no from the table and push them into a hash, keyed on the last three digits. Then loop through the keys of the hash, ftp to that directory and then grab all of the files that are associated with that hash - first doing a check to see if the file already exists locally.

The problem I have is I can't seem to get past the concept of this process. Is this the best approach and how exactly should I build/populate the hash?

In reply to efficient ftp of numerous files by Anonymous Monk

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.