in reply to Re: Re: reading columns from a flat file
in thread reading columns from a flat file

The idea was that by moving the inversion into a sperate process, your main program can just read the 'correct' format from a file handle (as shown) and the memory consumption wouldn't be a burdon on your main process. Nor would you need to change the main program, except to use the special form of open for the errent file.

How big is big?

Other than re-reading and re-spliting every line once for every column in the file, there isn't another way.

Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"Think for yourself!" - Abigail

Replies are listed 'Best First'.
Re: Re: Re: Re: reading columns from a flat file
by nosbod (Scribe) on Mar 19, 2004 at 12:11 UTC
    yes, i guess that would be the way to go. I can't think of a better way

    There could be 10,000 ids each having up to 200,000 elements

[reply]
      There could be 10,000 ids each having up to 200,000 elements

      Egads! Is there a possibility of just writing a script that takes these unwieldy inverted flat-files extracts the data in correct order once, and inserts it into a proper database for greater scalability? DBI with DBD::SQLite might be one (of several) good choices for the database stuff. Do the conversion once (or whenever you get a new inverted file), and then just be done with it. ...and enjoy the flexibility and scalability of a database.


      Dave

[reply]

In Section Seekers of Perl Wisdom