Depending on what your actual needs are, your situation could lend itself very neatly to a parallel approach, provided that reading the files actually is your bottleneck.
As you know the fixed structure of the files, you can simply spawn 24 copies of the processing program which will read their respective hourly directories and process those. This should be faster than iterating the whole directory tree and processing items as they are found.
Also potentially fast might be your OS equivalent of launching open my $entries, "ls -1R $basedir |" and reading from <$entries>, as that allows the reading of files in a separate program.
You could also consider having a process on the remote machine scan the directories and add new entries to a database and then accessing the database from your other machine for processing, but that risks files getting "forgotten" if they never enter the database or get moved to a different location in a manual process.
Also consider whether you really want to iterate over your complete tree of incoming data folders - judicious use of $File::Find::prune can let File::Find skip a whole year if you can be certain that there can be no new items to process there.
In reply to Re^3: File::Find won't iterate through CIFS share withouth "dont_use_nlink"
by Corion
in thread File::Find won't iterate through CIFS share withouth "dont_use_nlink"
by reinaldo.gomes
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |