Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles

UPDATE: As CountZero points out below, there is a problem with my do {...} blocks, so please check his proposal Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles which should run as intended.

It sounds like your data is sorted ascending with respect to time. Under this assumption I propose to use the following idea of a script:

use strict;
use warnings;

open my $fileA, ... or die;
open my $fileB, ... or die;
# same with the other files

my $lineB = '';
while( my $lineA = <$fileA> ) {
  chomp $lineA;
  next unless $lineA =~ m/^\d\.+(\d{2}):(\d{2}):(\d{2})\.+(\d{2})(\d{2
+})(\d{2})\d{2}.abc/;  
  my $dateA = ... ; # construct date from $1, $2 etc
  do {
    chomp $lineB;
    next unless $lineB =~ m/^\d\.+(\d{2}):(\d{2}):(\d{2})\.+(\d{2})(\d
+{2})(\d{2})\d{2}.abc/;  
    my $dateB = ... ; # construct date ...
    next if $dateB < $dateA; # use some date and time library for thes
+e comparisons
    last if $dateB > $dateA; # ...
    $lineA .= $lineB;
  } while( $lineB = <$fileB> );
  # same with fileC etc...
  print "$lineA\n";
}
[download]

Comment on Re: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles Select or Download Code

Replies are listed 'Best First'.
Re^2: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles by CountZero (Bishop) on Sep 22, 2013 at 13:57 UTC
Did you try your code? One cannot use `next` or `last` inside a `do { ... }` block. As the docs say : "contrary to popular belief do{} blocks, which do NOT count as loops". It is not a loop, so you cannot use loop-control commands. You can double the curly braces and put a bare block inside the `do { ... }` block. Bare blocks are loops that execute once. The loop-control commands will work ... somewhat, but `last` will not do as you might expect: it will exit the inner (bare) block, only to fall into the outer (do) block and hence has the same effect as `next`. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James My blog: Imperial Deltronics	[reply] [d/l] [select]
Re^3: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles by hdb (Monsignor) on Sep 22, 2013 at 14:06 UTC
I did not try my code as it is only a sketch of how to start. Thanks for pointing out this flaw, I was not aware of it. I did put the while at end to avoid the duplicate checking of the times. You have solved this directly in your code by adding another check.	[reply]
Re^4: matching datetimestamps and concatenating data where timestamps match from multiple large datafiles by CountZero (Bishop) on Sep 22, 2013 at 14:41 UTC
Yes, the duplicate checks do not make me very happy, but I cannot see another easy way out. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James My blog: Imperial Deltronics	[reply]