in reply to Re: reading columns from a flat file
in thread reading columns from a flat file

yes, if i'm reading this correcly then this is performing the pivot and is going to have to be done either:

on the whole input file when each new individual is called,

or once the first time the $ob->get_next method is called and then stored in an object variable for further calls for the next id

The question is which though? The size could be large and i guess this will be the decider. I was wondering whether there might be another way

  • Comment on Re: Re: reading columns from a flat file

Replies are listed 'Best First'.
Re: Re: Re: reading columns from a flat file
by BrowserUk (Patriarch) on Mar 19, 2004 at 11:54 UTC

    The idea was that by moving the inversion into a sperate process, your main program can just read the 'correct' format from a file handle (as shown) and the memory consumption wouldn't be a burdon on your main process. Nor would you need to change the main program, except to use the special form of open for the errent file.

    How big is big?

    Other than re-reading and re-spliting every line once for every column in the file, there isn't another way.

    Examine what is said, not who speaks.
    "Efficiency is intelligent laziness." -David Dunham
    "Think for yourself!" - Abigail
[reply]
[d/l]
      yes, i guess that would be the way to go. I can't think of a better way

      There could be 10,000 ids each having up to 200,000 elements

[reply]
        There could be 10,000 ids each having up to 200,000 elements

        Egads! Is there a possibility of just writing a script that takes these unwieldy inverted flat-files extracts the data in correct order once, and inserts it into a proper database for greater scalability? DBI with DBD::SQLite might be one (of several) good choices for the database stuff. Do the conversion once (or whenever you get a new inverted file), and then just be done with it. ...and enjoy the flexibility and scalability of a database.


        Dave

[reply]

In Section Seekers of Perl Wisdom