Re^7: Working with large amount of data

the main point remains: -why would you process a TB file on a wimp "laptop" machine? -where does this file come from? -why wouldn't this be a DB report?

The OP has made no mention of "laptop".
None of you trucking business :)
It would take 10x the disk storage and 20x longer just to load this data into the DB.
Never mind how long it would take to process the query, returning a billion counts, serialised through a socket or pipe, to an application program.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.

"Science is about questioning the status quo. Questioning authority".

In the absence of evidence, opinion is indistinguishable from prejudice.

RIP PCW It is as I've been saying!(Audio until 20090817)

Comment on Re^7: Working with large amount of data

Replies are listed 'Best First'.
Re^8: Working with large amount of data by Marshall (Canon) on Sep 21, 2009 at 02:22 UTC
1) I just meant small machine, actually smaller than my many years old desktop! 1GB user memory is considered "small". A user who has a 1TB log file would normally be a commercial user. A commercial system will have more than 1 GB for user in this type of case. 2) No offense intended re: where did file come from? That meant from what application, etc. Not any personal info needed. Maybe this thing is from an input to a commercial DB? I dunno know. Maybe a report from this DB is more appropriate. 3) I don't know what #3 means. In any event, no offense was intended!	[reply]
Re^9: Working with large amount of data by BrowserUk (Patriarch) on Sep 21, 2009 at 02:29 UTC
3) I don't know what #3 means. Simply that before one can produce a DB report, the data has to exist in a DB. And getting the data into a DB is going to take more diskspace and time than producing the required counts directly from the flat file. Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice. RIP PCW It is as I've been saying!(Audio until 20090817)	[reply]