in reply to Re^5: Working with large amount of data
in thread Working with large amount of data

I have updated previous posts. Windows NTFS can have a file >4GB.

However interesting these details are, the main point remains:
-why would you process a TB file on a wimp "laptop" machine?
-where does this file come from?
-why wouldn't this be a DB report?

  • Comment on Re^6: Working with large amount of data

Replies are listed 'Best First'.
Re^7: Working with large amount of data
by BrowserUk (Patriarch) on Sep 21, 2009 at 01:23 UTC
    the main point remains: -why would you process a TB file on a wimp "laptop" machine? -where does this file come from? -why wouldn't this be a DB report?
    1. The OP has made no mention of "laptop".
    2. None of you trucking business :)
    3. It would take 10x the disk storage and 20x longer just to load this data into the DB.

      Never mind how long it would take to process the query, returning a billion counts, serialised through a socket or pipe, to an application program.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      1) I just meant small machine, actually smaller than my many years old desktop! 1GB user memory is considered "small". A user who has a 1TB log file would normally be a commercial user. A commercial system will have more than 1 GB for user in this type of case.
      2) No offense intended re: where did file come from? That meant from what application, etc. Not any personal info needed. Maybe this thing is from an input to a commercial DB? I dunno know. Maybe a report from this DB is more appropriate.
      3) I don't know what #3 means.

      In any event, no offense was intended!

[reply]
        3) I don't know what #3 means.

        Simply that before one can produce a DB report, the data has to exist in a DB. And getting the data into a DB is going to take more diskspace and time than producing the required counts directly from the flat file.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom