It is completely unclear what separates the columns (above, ':')). This detail does matter. I would give us say, the first 7 columns x 3 rows and put that data within <code></code> tags. Then show us your "best go" at this problem so far in Perl. Can you give more info about the size of this input file? How many rows? I suspect that the entire input file will fit comfortably in memory and that generating the 70 or so output files can proceed in a straightforward way. There are a number of techniques to do this. Your question is still too general to get a concrete answer other than "heck yes, Perl can do it!". Oh, also mention if performance is of any concern at all? I don't expect that to be an issue here because most of the time will be taken by I/O, generating the plethora of output files.row_nameA:col2:col3:col4:col5:col6...col200 row_nameB:col2:col3:col4:col5:col6...col200
I guess there is the additional question, at least for my own curiosity: why are you doing this? Your application just seems odd enough (200 cols, 3 cols per file, about 70 output files), that perhaps there is a better way to do whatever it is that you are trying to do. This might be what is known as an X-Y problem.
Update:
Below is some code for one way to do this, there are other ways.
The code reads the input file and makes a 2-D array of the data. I presume that this amount of data will "fit" into memory without problems. If the line format is complex, then perhaps a .CSV module will be needed to parse each line? At each iteration of generating a new file, column 1 (the name) is reused and then the next left most 3 columns of data are consumed (the @data array "shrinks"). The loop ends when only column 1 remains of the original data.
#!/usr/bin/perl use strict; use warnings; my $number_of_cols_per_file =3; my @data; #this is a 2-D Array while (my $line = <DATA>) { chomp $line; my (@cols) = split (':', $line); push @data, \@cols; } my $file_num=1; while (@{$data[0]} > 1) # Any Columns after the name_column left? { # generate the next file # This print would change to a "file open" statement for # file_num, n... print "File Number = ",$file_num++,"\n"; foreach my $row_ref (@data) { my $row_name=$row_ref->[0]; my @data_cols = splice (@$row_ref,1,$number_of_cols_per_file); print join(":", $row_name, @data_cols), "\n"; } } =Prints: File Number = 1 row_nameA:col2a:col3a:col4a row_nameB:col2b:col3b:col4b File Number = 2 row_nameA:col5a:col6a row_nameB:col5b:col6b =cut __DATA__ row_nameA:col2a:col3a:col4a:col5a:col6a row_nameB:col2b:col3b:col4b:col5b:col6b
In reply to Re: Split a large text file by columns
by Marshall
in thread Split a large text file by columns
by tc
| For: | Use: | ||
| & | & | ||
| < | < | ||
| > | > | ||
| [ | [ | ||
| ] | ] |