roadtest has asked for the wisdom of the Perl Monks concerning the following question:

Background: Window Excel supports maximum 256 columns or 64K rows. Many of our CSV files were generated with 30~60 rows and 1000~2500 columns. So it can't be fed to excel directly.

I would like to invert row and columns as following:

#!/usr/bin/perl my @data; while (<DATA>) { chomp; push @data,[split /,/]; } for my $z ( 0 .. $#data ) { for my $v ( 0 .. $#{$data[$z]} ) { print "$data[$v][$z],"; } print "\n"; } __DATA__ Date,a,b,c,d,e,f Mon-1,11,12,13,14,15,16 Mon-2,21,22,23,24,25,26 Tue-1,31,,33,34,35,36 Tue-2,41,42,,44,45,46 Wed-1,51,52,53,54,55,56 Wed-2,61,62,63,64,65,66

Everything appears working fine

When I run it against the csv file, it can't convert all columns to row. I am puzzled by the behavior. It may be related to the csv content. Here is a small portion of one csv file

===

,/dir/dir1/file1,/dir/dir1/dir3,/dir/dir1/file2,/dir/dir1/file3,/dir/dir1/dir2,.....(up to 1000-2000 diretory or filename)

May-1,34,44,44,443,68,93,,,,,993,9,2,,,,13343,...(up to 1000-2000 columns)

May-2,343,343,,343,343,,,,4343,5,56,,,,,,........

May-3,3030...

....

May-30....

===

I like to see output like:

May-1,May-2,May-3,May-.......,May-30 /dir/dir1/file1,3,34,,,,34,....... /dir/dir1/dir3,..... /dir/dir1/file2,.... /dir/dir1/file3,... /dir/dir1/dir2.... ..up to 1000-2000 rows..
Thanks for your comments,

carl

Replies are listed 'Best First'.
Re: Invert CSV
by GrandFather (Saint) on May 30, 2011 at 22:13 UTC

    Could it be that among those many rows and columns some of the data contains a comma? If that is the case your split just ain't going to do the job for you - you will get too many rows in the result and some of them will have too few columns.

    You really should be using Text::CSV or Text::xSV to do the heavy lifting of parsing and generating CSV files.

    Oh, and as an aside, always use strictures (use strict; use warnings; - see The strictures, according to Seuss).

    True laziness is hard work
      Thanks, I like "The strictures, according to Seuss":-)
Re: Invert CSV
by BrowserUk (Patriarch) on May 30, 2011 at 19:57 UTC
    it can't convert all columns to row.

    What happens when "it cannot..."?

    Error messages? Out-of-memory? Other?


    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
      No, I don't see any error messages. It turns out there is wrong logic in my program.
Re: Invert CSV
by ig (Vicar) on May 30, 2011 at 20:00 UTC

    It looks to me like you have your iterations over rows and columns ($z and $v) reversed. Maybe something like the following would work better.

    use strict; use warnings; my @data; while (<DATA>) { chomp; push(@data, [ split /,/ ]); } my $cols = $#{$data[0]}; my $rows = $#data; foreach my $col (0..$cols) { foreach my $row (0..$rows) { print "$data[$row][$col],"; } print "\n"; } __DATA__ 1.1,1.2,1.3,1.4,1.5 2.1,2.2,2.3,2.4,2.5 3.1,3.2,3.3,3.4,3.5

    Are you sure every row has the same number of fields?

      Yes, you are right. I made logic mistake. I should loop column number first, then loop row number inside it. My testing data happens to be a square matrix, same columns and row numbers, which hides the logic error.

      After correcting it, all CSV files are processed correctly.

      Cheers,

Re: Invert CSV
by rgcosma (Beadle) on May 30, 2011 at 20:10 UTC
    Not a Perl answer, but since MS Office 2007 and OpenOffice 3, this is no longer an issue. Also, inverting = select the rows and columns you want to switch, cut, paste special, transpose
      Also, inverting = select the rows and columns you want to switch, cut, paste special, transpose
      Of course, if your tools cannot deal with that many columns, you won't even get to the stage where you can select...

      Thanks for posting that. I didn't realize that 2007 and OpenOffice could now handle that many columns. I just tried it with 10,000 columns.

      Thanks for your information. Microsoft is doing right thing this time:-)
Re: Invert CSV
by Tux (Canon) on May 30, 2011 at 21:56 UTC

    If I understand correctly, what you need is a PIVOT. Here is a way to do so:

    $ cat pivot.pl #!/pro/bin/perl use strict; use warnings; use Text::CSV_XS; use Spreadsheet::Read; my $ss = ReadData (shift)->[1]; my $csv = Text::CSV_XS->new ({ binary => 1, auto_diag => 1, eol => "\r +\n" }); $csv->print (*STDOUT, [@{$ss->{cell}[$_]}[1..$ss->{maxrow}]]) for 1 .. + $ss->{maxcol}; $ cat test.csv a,b,c,d,e,f 1,2,3,4,5,6 2,3,4,5,6,7 3,4,5,6,7,8 4,5,6,7,8,9 $ perl pivot.pl test.csv a,1,2,3,4 b,2,3,4,5 c,3,4,5,6 d,4,5,6,7 e,5,6,7,8 f,6,7,8,9 $

    Enjoy, Have FUN! H.Merijn
      Thanks, I will consider to install these modules if similar requests keep coming:-)