Re: Re: Re: Re: reading columns from a flat file

yes, i guess that would be the way to go. I can't think of a better way

There could be 10,000 ids each having up to 200,000 elements

Comment on Re: Re: Re: Re: reading columns from a flat file

Replies are listed 'Best First'.
Re: Re: Re: Re: Re: reading columns from a flat file by davido (Cardinal) on Mar 20, 2004 at 05:23 UTC
There could be 10,000 ids each having up to 200,000 elements Egads! Is there a possibility of just writing a script that takes these unwieldy inverted flat-files extracts the data in correct order once, and inserts it into a proper database for greater scalability? DBI with DBD::SQLite might be one (of several) good choices for the database stuff. Do the conversion once (or whenever you get a new inverted file), and then just be done with it. ...and enjoy the flexibility and scalability of a database. Dave	[reply]

Replies are listed 'Best First'.

Re: Re: Re: Re: Re: reading columns from a flat file
by davido (Cardinal) on Mar 20, 2004 at 05:23 UTC

There could be 10,000 ids each having up to 200,000 elements

Egads! Is there a possibility of just writing a script that takes these unwieldy inverted flat-files extracts the data in correct order once, and inserts it into a proper database for greater scalability? DBI with DBD::SQLite might be one (of several) good choices for the database stuff. Do the conversion once (or whenever you get a new inverted file), and then just be done with it. ...and enjoy the flexibility and scalability of a database.

Dave

[reply]