in reply to Working with multiple records with different dates

G'day rruser,

Here's a solution using the builtin modules Time::Piece and Time::Seconds. It only requires a single script (instead of your three) and doesn't create an additional file concatenating all the data from the original three files.

With the data you posted, "COMPANY      BCYX 156095 L  2013,11,30   2013,12,08" has the greatest time spread (8 days). For my testing, I duplicated that record in file2 and file3 and added a unique record to file1 which also had a spread of 8 days. Here's the data I worked with:

$ cat pm_1071030_file1.txt COMPANY ABCD 764200 E 2013,12,13 2013,12,19 COMPANY BCDX 156167 L 2013,11,29 2013,12,03 COMPANY BCYX 165230 L 2013,12,13 2013,12,19 COMPANY BCYX 156095 L 2013,11,29 2013,12,07 $ cat pm_1071030_file2.txt COMPANY ABCD 764200 E 2013,12,13 2013,12,19 COMPANY BCDX 156167 L 2013,12,28 2013,12,31 COMPANY BCYX 156095 L 2013,11,30 2013,12,08 $ cat pm_1071030_file3.txt COMPANY ABCD 764200 E 2013,12,13 2013,12,17 COMPANY BCDX 156167 L 2013,11,30 2013,12,03 COMPANY BCYX 165230 L 2013,12,13 2013,12,17 COMPANY BCYX 156095 L 2013,11,30 2013,12,08 COMPANY BCYX 156095 L 2013,11,30 2013,12,08

Here's the script:

#!/usr/bin/env perl use strict; use warnings; use autodie; use Time::Piece; use Time::Seconds; my $format = '%Y,%m,%d'; my $free_days = 4; my @wanted_record_data = (0, []); for my $file (map { "pm_1071030_file${_}.txt" } 1 .. 3) { open my $fh, '<', $file; while (<$fh>) { my ($start, $end) = (split)[4, 5]; my $t0 = Time::Piece->strptime($start, $format); my $t1 = Time::Piece->strptime($end, $format); my $diff = ($t1 - $t0)->days; next if $diff <= $free_days; push @{$wanted_record_data[1]}, $_ if $diff == $wanted_record_ +data[0]; @wanted_record_data = ($diff, [$_]) if $diff > $wanted_record_ +data[0]; } } print "Greatest time spread = $wanted_record_data[0] days.\n"; print "Records with this time spread:\n"; my %seen; print for grep { ! $seen{$_}++ } @{$wanted_record_data[1]};

Here's the output:

Greatest time spread = 8 days. Records with this time spread: COMPANY BCYX 156095 L 2013,11,29 2013,12,07 COMPANY BCYX 156095 L 2013,11,30 2013,12,08

-- Ken

Replies are listed 'Best First'.
Re^2: Working with multiple records with different dates
by rruser (Acolyte) on Jan 21, 2014 at 23:01 UTC

    Thanks Ken that gets me closer, I didn't take into account the longest time spread may be different for different records.

    example COMPANY ABCD 764200 E 2013,12,13 2013,12,19 COMPANY BCYX 156095 L 2013,11,30 2013,12,08

    The first record has 6 days and the second 8 days. I need both records with any duplicates or shorter time span dates removed. I'm not sure if I am explaining this well. I need each unique record so if it exists only in 1 file then I use that date. If records exists in multiple files then the one with the longest time frame.

    Any suggestions, I sure appreciate your time and expertise

      "I'm not sure if I am explaining this well."

      I don't think you are.

      You say "The first record has 6 days and the second 8 days.". None of the three files you show have records like that. I wondered if you meant "file" instead of "record", but that leaves no mention of the the third file (which is the one with an 8 day time spread).

      My best guess is that you should process the files separately (perhaps storing interim data something like '$data_for_file{$file} = [ @wanted_record_data ]') and then extract the records you want after all files have been read.

      -- Ken