UPDATE:
looking for advice on how to build some logic into my process so that I don't "duplicate" data entry. My current process uses file:copy:recursive to copy all the output to a temp directory from there, I concatenate the files using this code then from there a stored procedure is called that dumps the csv into the database, I was wondering what would be recommended way of ensuring my data is not duplicated.
UPDATE: Ah so it was just doing what I told it to do instead of what I meant.
thanks, this is part of a larger process and I have been wracking my brain trying to figure out how to update only the newer records into my database, at present it will take anything I give it, and I want the process to be able to grab only the new information so when I go to load it it gives me only what is currently generated since the last time it ran, what would be the best way and most idiot proof way to go about building in this logic to my process?
I have about 50 directories, which contain files with the current date.txt as their names, they contain comma delimited data, I would like to combine all the files into one huge csv file but I am not sure how to put them into just one file, I would also like to set a limit going forward so that only new files are added to the csv I would also like to check that each line contains 9 values using magic numbers.
I have code to remove the newlines and split on the commas to make each file a csv but I would like to have them all become one file and I am not sure how to do that. I just tried to open a myfile.txt within the loop but it only gives me the last file's content I was curious why it did not work, what is happening here
Update opening file in the right (write) mode helps tremendously thanks to all for their input.
#!/usr/bin/perl
use strict;
use warnings;
my $dir = ".";
my $root= "c:\\mlx\\myfile.txt";
lista_dirs($dir, $dir);
sub lista_dirs {
my ($dir, $dirname) = @_;
my (@dirs, @files);
opendir DIR, "$dir";
my @dircontent = grep { /[^\.]/ } readdir(DIR);
closedir DIR;
@dirs = ();
@files = ();
foreach(@dircontent) {
if(-d "$dir/".$_) {
push @dirs, $_;
} else {
push @files, $_ if($_ =~ /\.txt$/);
}
}
foreach my $d(@dirs) {
lista_dirs("$dir/".$d, $d);
}
# IMPORTANT NOTE: if your files aren't too big, you can read every
+thing at once as shown in the code below. # if note, you can open
+the input file for reading and a temp file for wirting. Read the file
+ line by line, # replace \n with the , and write to the temp file
+. Close both files when you're done, and then 'rename'
# the temp file to the input filename
foreach my $file (@files) {
$/ = '';
open FILE, "$dir/$file";
my $content = <FILE>;
close FILE;
$content =~ tr/\n/,/;
open FILE, ">$dir/$file";
print FILE $content;
open FILE2, ">>$root";
print FILE2 $content;
close FILE;
close FILE2;
}
return;
}