in reply to reading columns from a flat file

You could write a short perl script (or even a one-liner) that would read the inverted input file and re-invert it to stdout. You can then use this to read and convert the file and read it's output directly using a piped open (See perlopentut)

The following example processes a simple whitespace delimited file and inverts it. You'll need to adapt flip.pl to your formats. Also, it assumes that every line will have the same number of columns as the first. 

P:\test>type test.dat
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5
1 2 3 4 5

P:\test>type flip.pl
#! perl -slw
use strict;
my @cols;

push @cols, [ split ] while $_ = <>;

for my $n ( 0 .. $#{ $cols[ 0 ] } ) { 
    print join ' ', map{ $_->[ $n ] } @cols; 
}

P:\test>perl -le "
    open F, qq[ flip test.dat |]; 
    print join '|', split while $_ = <F>; 
"

1|1|1|1|1|1|1|1
2|2|2|2|2|2|2|2
3|3|3|3|3|3|3|3
4|4|4|4|4|4|4|4
5|5|5|5|5|5|5|5

[download]
Note: The one-liner manually wrapped and uses win32 quoting.
Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail

Replies are listed 'Best First'.
Re: Re: reading columns from a flat file
by nosbod (Scribe) on Mar 19, 2004 at 11:23 UTC
    yes, if i'm reading this correcly then this is performing the pivot and is going to have to be done either:

    on the whole input file when each new individual is called,

    or once the first time the $ob->get_next method is called and then stored in an object variable for further calls for the next id

    The question is which though? The size could be large and i guess this will be the decider. I was wondering whether there might be another way

[reply]

      The idea was that by moving the inversion into a sperate process, your main program can just read the 'correct' format from a file handle (as shown) and the memory consumption wouldn't be a burdon on your main process. Nor would you need to change the main program, except to use the special form of open for the errent file.

      How big is big?

      Other than re-reading and re-spliting every line once for every column in the file, there isn't another way.

      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "Think for yourself!" - Abigail
[reply]
[d/l]
        yes, i guess that would be the way to go. I can't think of a better way

        There could be 10,000 ids each having up to 200,000 elements

[reply]

In Section Seekers of Perl Wisdom