to divide/split a file to small files.

Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

hello there, I have a big chunk of data in a file that has got 7 columns(0-7). I would need the first and the third column in some case and the first and the fifth column in other cases. I have no problem parsing the columns. But I need to write them into 24 files depending on the second column,which is from 1-24.

The actual file is something like this:

10234,1,34234,342,453,34535,3453,3464
12123,1,23432,4353,44645,45654,45645,657
2123345,2,35435,3453,454,56567,56756,567567
3234353,2,34534,4564,567567,56757,56575,24234
33453,3,45464,4564564,45645,645645,54564,56456
[download]

I would like to have the required columns of the first two lines in a file and the next two in a separate file and so on for the rest(max 24).
Is there a module to do this? If not how could we open multiple files to write? Is there any easy method to do this?Any suggestions or ideas would be very helpful. thanks in advance.
PS. The files are comma separated and quite big- running upto 5442187 lines in the main file.

Comment on to divide/split a file to small files. Download Code

Replies are listed 'Best First'.
Re: to divide/split a file to small files. by jethro (Monsignor) on Mar 07, 2011 at 17:47 UTC
You can use arrays as filehandles, so any number of files can simply be handled in a loop. The only thing to observe is how to use the filehandles with print: `open ($f[$x],'>','test$x.txt') or die ... ... print { $f[$y] } "whataniceview\n";` [download] You can read about it with 'perldoc -f print' and 'perldoc -f open'	[reply] [d/l]
Re: to divide/split a file to small files. by toolic (Bishop) on Mar 07, 2011 at 18:31 UTC
I've been meaning to figure out how to open an array of filehandles, so here's my opportunity. I Super Searched where title contains all of "open", "handles" and found Opening Multiple FileHandles w/ Simpler Code and FileHandle. The following will create 3 output files, named 1.txt, 2.txt, 3.txt. You can change the 3 to 24 in the code. `use warnings; use strict; use FileHandle; my @filenames = map { $_ . '.txt' } 1 .. 3; my @handles = map { FileHandle->new($_, '>') } @filenames; while (<DATA>) { my @cols = split /,/; $handles[$cols[1]-1]->print($_); } __DATA__ 10234,1,34234,342,453,34535,3453,3464 12123,1,23432,4353,44645,45654,45645,657 2123345,2,35435,3453,454,56567,56756,567567 3234353,2,34534,4564,567567,56757,56575,24234 33453,3,45464,4564564,45645,645645,54564,56456` [download]	[reply] [d/l]
Re: to divide/split a file to small files. by Corion (Patriarch) on Mar 07, 2011 at 17:58 UTC
Also see part - split up files according to column value, which you could either try to reuse or adapt to your needs	[reply]
Re: to divide/split a file to small files. by locked_user sundialsvc4 (Abbot) on Mar 09, 2011 at 14:30 UTC
Yet another approach would be to simply fire off `grep` command(s), each one directing its output to a different file. If you felt particularly brave, you could even fire 24 copies of that command to run in parallel with one another. (They would not smash each other nearly as much as you might suppose, since the OS would undoubtedly realize that “the same file” is open for reading by, say, 24 different processes.) To my way of thinking, the trade-off is between “the efficiency of the computer’s job-description when it runs this,” vs. “how much time do you want to spend to ‘get ’er done?’ ” The nice thing is, any ol’ computer these days will more-or-less shrug its shoulders indifferently at what you see to be a “large” file. So, you have several reasonable alternatives. `Be nice to me, for you can be replaced by a very small shell script.`
Re: to divide/split a file to small files. by jellisii2 (Hermit) on Mar 11, 2011 at 13:38 UTC
Completely untested, but I'd start with something like this: `while (<infile>) { my @line = split /,/, $_; [case code here for determining what data you need] open(FH,">$_[1].txt"); print FH [data you need in said file] }` [download] If your file names are predetermined by some other method, set up a hash with 1-24 as your keys, and use them in the open statement.	[reply] [d/l]