concatenate directories and files into one csv

grashoper has asked for the wisdom of the Perl Monks concerning the following question:

UPDATE: looking for advice on how to build some logic into my process so that I don't "duplicate" data entry. My current process uses file:copy:recursive to copy all the output to a temp directory from there, I concatenate the files using this code then from there a stored procedure is called that dumps the csv into the database, I was wondering what would be recommended way of ensuring my data is not duplicated. UPDATE: Ah so it was just doing what I told it to do instead of what I meant. thanks, this is part of a larger process and I have been wracking my brain trying to figure out how to update only the newer records into my database, at present it will take anything I give it, and I want the process to be able to grab only the new information so when I go to load it it gives me only what is currently generated since the last time it ran, what would be the best way and most idiot proof way to go about building in this logic to my process? I have about 50 directories, which contain files with the current date.txt as their names, they contain comma delimited data, I would like to combine all the files into one huge csv file but I am not sure how to put them into just one file, I would also like to set a limit going forward so that only new files are added to the csv I would also like to check that each line contains 9 values using magic numbers. I have code to remove the newlines and split on the commas to make each file a csv but I would like to have them all become one file and I am not sure how to do that. I just tried to open a myfile.txt within the loop but it only gives me the last file's content I was curious why it did not work, what is happening here Update opening file in the right (write) mode helps tremendously thanks to all for their input.

 
#!/usr/bin/perl

use strict;
use warnings;

my $dir = ".";
my $root= "c:\\mlx\\myfile.txt";
lista_dirs($dir, $dir);

sub lista_dirs {
    my ($dir, $dirname) = @_;
    my (@dirs, @files);

    opendir DIR, "$dir";
    my @dircontent = grep { /[^\.]/ } readdir(DIR);
    closedir DIR;

    @dirs  = ();
    @files = ();

    foreach(@dircontent) {
        if(-d "$dir/".$_) {
            push @dirs, $_;
        } else {
            push @files, $_ if($_ =~ /\.txt$/);
        }
    }

    foreach my $d(@dirs) {
        lista_dirs("$dir/".$d, $d);
    }

    # IMPORTANT NOTE: if your files aren't too big, you can read every
+thing at once as shown in the code below.    # if note, you can open 
+the input file for reading and a temp file for wirting. Read the file
+ line by line,     # replace \n with the , and write to the temp file
+. Close both files when you're done, and then 'rename' 
    # the temp file to the input filename
    foreach my $file (@files) {
        $/ = '';
        open FILE, "$dir/$file";
        my $content = <FILE>;
        close FILE;
   
        $content =~ tr/\n/,/;
        open FILE, ">$dir/$file";
        print FILE $content;
        open FILE2, ">>$root";
        print FILE2 $content;
        close FILE;
        close FILE2;
    }
    return;
}
[download]

Comment on concatenate directories and files into one csv Download Code

Replies are listed 'Best First'.
Re: concatenate directories and files into one csv by jethro (Monsignor) on Apr 28, 2009 at 16:40 UTC
You open the file for reading instead of appending. `open FILE2,">>","$root";`	[reply] [d/l]
Re: concatenate directories and files into one csv by mr_mischief (Monsignor) on Apr 28, 2009 at 16:41 UTC
from open: If MODE is '>', the file is truncated and opened for output, being created if necessary. If MODE is '>>', the file is opened for appending, again being created if necessary. You're opening and truncating when you want opening and appending. This, BTW, is being done iteratively rather than recursively as was mentioned in the CB.	[reply]
Re: concatenate directories and files into one csv by almut (Canon) on Apr 28, 2009 at 16:42 UTC
...but it only gives me the last file's content. `foreach my $file (@files) { ... open FILE, ">$dir/$file"; ... }` [download] Open the output file once outside of the loop, or open it in append mode (`">>"`), if you want to accumulate content... Update: Actually, it seems that FILE2 is supposed to be the output file for the accumulated content... (thanks jethro!) The comment with respect to opening it once or in append mode still applies, though.	[reply] [d/l] [select]
Re: concatenate directories and files into one csv by ikegami (Patriarch) on Apr 28, 2009 at 17:33 UTC
If you have a problem, perhaps you should check the error code returned by the function that's not doing what you want it to do. one would figure that goes without saying, but I've told you this before.	[reply]
Re: concatenate directories and files into one csv by GrandFather (Saint) on Apr 28, 2009 at 22:27 UTC
Rules for using open: Always use the 3 parameter version of open Always use locally scoped lexical file handles Always check the result The template is: `open my $inFile, '<', $filename or die "Failed to open $filename: $!\n +";` [download] Omit the newline in the die if you want to show the line number where the open failed. True laziness is hard work	[reply] [d/l]