Win has asked for the wisdom of the Perl Monks concerning the following question:

I have a system that allows for pooling of request files. The request files are read and the data contained in them informs activation of a relational database system. This system already allows for specification of overnight processing, priority setting and node targeting. I have the idea that it would be nice to allow the setting of the earliest time period that the request can be read.

I have the idea that there could be two approaches to this. Either a tree structure of folders could be created, whereby the upper node is year and branches form the months and then the days and then hour. Or separate folders could tag all the time details in the folder name. If I use this second approach what convention do I use? Do I need to consider any Perl modules that use specific time specification string conventions?

I also need to consider the efficiency of the algorithm. I don't want my program to have to repeatedly check the same folder for a whole year. Maybe I need a combination of the two aforementioned methods, where folders get copied across on the day of processing.


Update: To codify the problem:
foreach ( @directories ) # foreach of the time stamped directories { $directory_name = $_; # I need to take the directory name here # I need to process the directory name here to give me a time tha +t the directory is labelled with my $date_time = ; # .... and to assign a value to $date_time my $now = ; Not sure how to get this value if ($date_time => $now ) # If the date has been passed { my $directory_to_move = $input_directory_CC."".$directory_name +; return $directory_to_move; } }

Replies are listed 'Best First'.
Re: Approaches to time stamping request files
by merlyn (Sage) on Apr 30, 2007 at 10:43 UTC
    It's not clear what you're optimizing for, or that you're not suffering from premature optimization.

    If the speed by which an "earliest request" to be found is a problem, I'd put the data into a database, not a file. Then I'd simply create an index on the timestamp column, making for very speedy searches.

    Sure, the filesystem can be used as a database, but sometimes, a database makes sense. {grin}

Re: Approaches to time stamping request files
by TomDLux (Vicar) on Apr 30, 2007 at 19:56 UTC

    How many files will you have? How large will each file be? Are you simply grabbing the entire file, or do you want subsets of fields and records from a file? Will you be doing comparisons between files? ... i.e. Find the offices previously managed by managers who went on to lead sales teams with top 10% performance?

    --
    TTTATCGGTCGTTATATAGATGTTTGCA

      These files are request files that come in from users of an interactive analytical database system. In theory there could be hundreds coming in every day (even thousands). If I develop this 'minimum time of activation' idea I will be able to automate daily checks that ensure consistency of results coming back.

      Not very large files, they only hold the specification for activations of a stored procedure.