Re^4: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

As I think about it, the best option might be to insert everything in a database (with columns fileID and intValue), then--after it's all been added to the database--loop through each fileID and add its values to the mega-file (but delete rows as they're inserted to save space).

FFR, the speed of this means nothing . . . it's all about disk space conservation and the speed at which it can be read back in.

Comment on Re^4: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?)

Replies are listed 'Best First'.
Re^5: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?) by davido (Cardinal) on Jul 24, 2004 at 06:15 UTC
If speed meant NOTHING you'd do it by hand. The point is that inserting into a flat file means rewriting the entire file each time, unless you store updates and do them in groups. Imagine, if your 2mb flat file grows by 4 bytes each iteration, you're moving around 2mb of data each time you try to add 4 bytes. That's a 2,000,000 bytes of data reading and rewriting for each 4 byte insertion. Speed has got to mean something. I think your solution may be a good one. If building up the dataset is a one-time deal, do it in a database, and then transfer the completed product to a flat file where it can be read quickly. Dave	[reply]
Re^6: Combining Ultra-Dynamic Files to Avoid Clustering (Ideas?) by rjahrman (Scribe) on Jul 24, 2004 at 07:19 UTC
Sorry if I came off as a bit ignorant. :) I agree that moving that data around like that is not an option. I just meant that if building this file take twice as long, but yields a file 10% smaller, it's worth it in this particular case. The file will only be built occasionally (like every other month).	[reply]