Re^3: copying records from one file to another with filter.

A few observations and comments:

You are reading in the entire file and saving it in an array of arrays, but then don't do anything with it. (Well, other than use the size of the array as a never terminating while() condition.) For that situation your loop probably should be

for (@input)

rather than

while (@input)

You are searching for a YYYY-MM-DD pattern when your example data is in DD-MMM-YYYY format. You'll never find anything.

You should probably use the 3 argument form of open and lexical file handles. What you have isn't wrong particularly, but isn't considered best practices.

The way you are calculating 30 days is easy but fragile. You may be better off using one of the date calculation packages. It may be good enough here though, so take that with a grain of salt.

Here's a minor reworking of your posted script to be more efficient (and work, for certain values of "work").

#!/usr/bin/perl
use strict;
use warnings;

my @months = ( qw/Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec/); 

# source file directory
my $srcdir = '../source';

# source file name
my $srcfile = $srcdir . '/vol.dat';

my $desdir = '../target';

my $desfile = $desdir . '/total_volume.csv';

# Open source file.
open my $read, '<', $srcfile or die "Can't open the DAT file $!\n";

open my $write, '>', $desfile or die "Can't open the CSV file. $!\n";

print $write "Date,Total_Volume\n";

my ( $year, $month, $day ) = ( localtime( time() - 2592000 ) )[ 5, 4, 
+3 ];

$year += 1900;

my $startdate = "$day-".$months[$month]."-$year";

while (<$read>) {
    if ( /^$startdate/ .. /\x4/ ) {
        chomp;
        my (@items) = split /,/;
        printf $write "%s,%d\n", $items[0], $items[1] + $items[2];
    }
}

# probably not necessary but not a bad idea
close $read  or die "Couldn't close the DAT file: $!\n"; 
close $write or die "Couldn't close the CSV file: $!\n";
[download]

Comment on Re^3: copying records from one file to another with filter. Download Code

Replies are listed 'Best First'.
Re^4: copying records from one file to another with filter. by avanta (Beadle) on Jan 13, 2010 at 01:58 UTC
First of all, thanks a lot of the help. Also Im really sorry i didnt updated the source file format. Actually, I was done with the code(with some glitches). Havent used ur code yet, but will do in another script. New Source format: 2009-12-22,174865.6853,171 2009-12-23,158423.1442,155 2009-12-24,146650.9855,143 2009-12-25,127228.4832,124 2009-12-26,179032.6644,175 2009-12-27,179770.0221,176 2009-12-28,153049.9829,149 2009-12-29,159508.811,155 2009-12-30,75322.9348,143 2009-12-31,184494.3142,124 2010-01-01,88085.89262,87 2010-01-02,157525.6179,204 2010-01-03,213673.8187,214 2010-01-04,190080.1713,198 2010-01-05,139624.0644,155 2010-01-06,159684.3982,180 2010-01-07,159508.811,91 2010-01-08,75322.9348,130 2010-01-09,174867.4572,207 2010-01-10,206403.5704,86 2010-01-11,121876.6863,154 2010-01-12,89091.60969,209 2010-01-13,159684.3982,180 2009-01-14,153049.9829,149 [download] New code (which works): #!/usr/bin/perl use strict; use warnings; # source file directory. my $srcdir = "../source"; # source file name. my $srcfile = $srcdir."/vol.dat"; # Open source file in READ mode. open (READ, "< $srcfile") \|\| die "Can't find the DAT file\n"; ################## Start date time i.e. the first day of the last 30 d +ays, is calculated by this code. my $epochToday1 = time; $epochToday1 = $epochToday1 - 2592000; my ($year1, $month1, $day1) = (localtime($epochToday1))[5,4,3]; $month1++; $year1+=1900; my $startdate = $year1."-".$month1."-".$day1; ######################## ################## End date time i.e. the current day, is calculated b +y this code. my $epochToday2 = time; my ($year2, $month2, $day2) = (localtime($epochToday2))[5,4,3]; $month2++; $year2+=1900; my $enddate = $year2."-".$month2."-".$day2; ######################## # Destination Directory. my $desdir = "../target"; #destination File name. my $desfile = $desdir."/total_volume_last30days.csv"; #Open target file in Write mode. open (WRITE, "> $desfile") \|\| die "Can't find the CSV file.\n"; # Print the headers for the Report in CSV. my @headers = ("Date",",","Total_Volume"); print WRITE @headers,"\n"; my $printout; #declaration of variable which prints in the Tar +get file. while (<READ>) { my $start = $startdate; my $end = $enddate; if ( /^$start/ .. /^$end/ ) #range to be checked and writt +en to Target file. { chomp; my (@items) = split (/,/,$_); my $tot = $items[1]+ $items[2]; #sum of incoming an outgoin +g bytes. $printout .= "$items[0],$tot \n"; #variable which store +s the Data to be written in target file. } } print WRITE $printout; #write $printout va +lues to Target file. close READ \|\| die "Couldn't close the DAT file"; #close input f +ile. close WRITE \|\| die "Couldn't close the CSV file"; #close target + file. exit 0; [download] In this new code I have used today's date as ending parameter in the range operator, which is what I actually require. And the glitch which I was talking about is that if the input contains a date next to today's date, the output file also gets that data. but as u can see I have used $end as $enddate which is current system date, How can I check this? Apart from this I wish to create another or edit this same script which gives the current month data. For that the idea which I was using was hardcode the $day1 and remove that 30 days of seconds which I was subtracting in $startdate as '01' but just changing that creates an error `Use of uninitialized value $printout in print at total_volume_last30.p +l line 67, <READ> line 122.` [download] If you can suggest some other way it would be grateful of you. Thanks AvantA, .	[reply] [d/l] [select]
Re^5: copying records from one file to another with filter. by thundergnat (Deacon) on Jan 13, 2010 at 13:56 UTC
If the data you show is the actual format, you're going to need to make sure all your dates are in the correct format. The dates in the data have leading zeros in single digit months and days but your search terms do not. That's likely to cause an issue. Build your search string with a sprintf function rather than just concatenating numbers. If you need to do more complex date calculations, you are better of using one of the modules designed to work with date data. Date::Calc and Date::Manip are two well known modules that can help with that. If you are getting uninitialized value warnings, you need to check why. Look at the warning. It tells you where the problem occurred. My suspicion is the there is a blank line on line 122 in the input file, which means there wasn't any data to split, so the variables being printed were uninitialized. If that is a problem, just check for blank lines before splitting or empty variables before printing.	[reply]