in reply to Re^4: Working with large amount of data
in thread Working with large amount of data

Update: This is (irrelevant) the article that was originally cited in the post above!

Sorry, but I'm not sure how an article describing the 4GB limit of files on FAT32 partitions is relevant to a claim that NTFS can only handle 2GB files?

For the real skinny, go to the source. Besides which I used files far greater than 2GB on my old thinkpad running NT4/NTFS. From memory the largest I ever created was 32GB, but when I first got my current machine I used contig to create a 500GB contiguous file just for the fun of it. I don't have enough diskspace to repeat the process, but I did just create one 16GB.

Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
"Science is about questioning the status quo. Questioning authority".
In the absence of evidence, opinion is indistinguishable from prejudice.
RIP PCW It is as I've been saying!(Audio until 20090817)
  • Comment on Re^5: Working with large amount of data

Replies are listed 'Best First'.
Re^6: Working with large amount of data
by Marshall (Canon) on Sep 21, 2009 at 00:43 UTC
    I have updated previous posts. Windows NTFS can have a file >4GB.

    However interesting these details are, the main point remains:
    -why would you process a TB file on a wimp "laptop" machine?
    -where does this file come from?
    -why wouldn't this be a DB report?

[reply]
      the main point remains: -why would you process a TB file on a wimp "laptop" machine? -where does this file come from? -why wouldn't this be a DB report?
      1. The OP has made no mention of "laptop".
      2. None of you trucking business :)
      3. It would take 10x the disk storage and 20x longer just to load this data into the DB.

        Never mind how long it would take to process the query, returning a billion counts, serialised through a socket or pipe, to an application program.

      Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
      "Science is about questioning the status quo. Questioning authority".
      In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
        1) I just meant small machine, actually smaller than my many years old desktop! 1GB user memory is considered "small". A user who has a 1TB log file would normally be a commercial user. A commercial system will have more than 1 GB for user in this type of case.
        2) No offense intended re: where did file come from? That meant from what application, etc. Not any personal info needed. Maybe this thing is from an input to a commercial DB? I dunno know. Maybe a report from this DB is more appropriate.
        3) I don't know what #3 means.

        In any event, no offense was intended!

[reply]

In Section Seekers of Perl Wisdom