UPDATE: looking for advice on how to build some logic into my process so that I don't "duplicate" data entry. My current process uses file:copy:recursive to copy all the output to a temp directory from there, I concatenate the files using this code then from there a stored procedure is called that dumps the csv into the database, I was wondering what would be recommended way of ensuring my data is not duplicated. UPDATE: Ah so it was just doing what I told it to do instead of what I meant. thanks, this is part of a larger process and I have been wracking my brain trying to figure out how to update only the newer records into my database, at present it will take anything I give it, and I want the process to be able to grab only the new information so when I go to load it it gives me only what is currently generated since the last time it ran, what would be the best way and most idiot proof way to go about building in this logic to my process? I have about 50 directories, which contain files with the current date.txt as their names, they contain comma delimited data, I would like to combine all the files into one huge csv file but I am not sure how to put them into just one file, I would also like to set a limit going forward so that only new files are added to the csv I would also like to check that each line contains 9 values using magic numbers. I have code to remove the newlines and split on the commas to make each file a csv but I would like to have them all become one file and I am not sure how to do that. I just tried to open a myfile.txt within the loop but it only gives me the last file's content I was curious why it did not work, what is happening here Update opening file in the right (write) mode helps tremendously thanks to all for their input. 
 
#!/usr/bin/perl

use strict;
use warnings;

my $dir = ".";
my $root= "c:\\mlx\\myfile.txt";
lista_dirs($dir, $dir);

sub lista_dirs {
    my ($dir, $dirname) = @_;
    my (@dirs, @files);

    opendir DIR, "$dir";
    my @dircontent = grep { /[^\.]/ } readdir(DIR);
    closedir DIR;

    @dirs  = ();
    @files = ();

    foreach(@dircontent) {
        if(-d "$dir/".$_) {
            push @dirs, $_;
        } else {
            push @files, $_ if($_ =~ /\.txt$/);
        }
    }

    foreach my $d(@dirs) {
        lista_dirs("$dir/".$d, $d);
    }

    # IMPORTANT NOTE: if your files aren't too big, you can read every
+thing at once as shown in the code below.    # if note, you can open 
+the input file for reading and a temp file for wirting. Read the file
+ line by line,     # replace \n with the , and write to the temp file
+. Close both files when you're done, and then 'rename' 
    # the temp file to the input filename
    foreach my $file (@files) {
        $/ = '';
        open FILE, "$dir/$file";
        my $content = <FILE>;
        close FILE;
   
        $content =~ tr/\n/,/;
        open FILE, ">$dir/$file";
        print FILE $content;
        open FILE2, ">>$root";
        print FILE2 $content;
        close FILE;
        close FILE2;
    }
    return;
}

[download]

In reply to concatenate directories and files into one csv by grashoper

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.