in reply to Re^3: Working with large amount of data
in thread Working with large amount of data

At the "end of the day", if the file size (pointers) in the OS's file system are limited to 32 bits, then you can only do what 32 bits can do. 2**32-1=4,294,967,296.

Anyway 2GB vs 4 GB is WAY off from 1 TB! Update: Windows NTFS file pointers are NOT limited to 32 bits, even on a 32 bit systems. So the above is "right", but still "wrong".

  • Comment on Re^4: Working with large amount of data

Replies are listed 'Best First'.
Re^5: Working with large amount of data
by BrowserUk (Patriarch) on Sep 21, 2009 at 00:25 UTC

    Update: This is (irrelevant) the article that was originally cited in the post above!

    Sorry, but I'm not sure how an article describing the 4GB limit of files on FAT32 partitions is relevant to a claim that NTFS can only handle 2GB files?

    For the real skinny, go to the source. Besides which I used files far greater than 2GB on my old thinkpad running NT4/NTFS. From memory the largest I ever created was 32GB, but when I first got my current machine I used contig to create a 500GB contiguous file just for the fun of it. I don't have enough diskspace to repeat the process, but I did just create one 16GB.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      I have updated previous posts. Windows NTFS can have a file >4GB.

      However interesting these details are, the main point remains:
      -why would you process a TB file on a wimp "laptop" machine?
      -where does this file come from?
      -why wouldn't this be a DB report?

[reply]
        the main point remains: -why would you process a TB file on a wimp "laptop" machine? -where does this file come from? -why wouldn't this be a DB report?
        1. The OP has made no mention of "laptop".
        2. None of you trucking business :)
        3. It would take 10x the disk storage and 20x longer just to load this data into the DB.

          Never mind how long it would take to process the query, returning a billion counts, serialised through a socket or pipe, to an application program.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom