in reply to Re: Working with large amount of data
in thread Working with large amount of data

I agree with this! I have a relatively old Win XP machine and on this "wimp machine" a user can use 2GB of the physical memory (aside from what WinXP uses).

I would also ask what file system is being used to read this TB monster file?Update: this limit is not right: (My NTFS is limited to 2GB file size). This part is right: What really blows my mind is that this is a log file! Log file of WHAT?

  • Comment on Re^2: Working with large amount of data

Replies are listed 'Best First'.
Re^3: Working with large amount of data
by BrowserUk (Patriarch) on Sep 20, 2009 at 23:49 UTC
    (My NTFS is limited to 2GB file size).

    Huh? NTFS has been able to handle individual files up to a gnat's todger under 16TB forever. (At least since NT4 days; and I believe since 3.51). Even FAT16 & FAT32 can handle 4GB.

    Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
    "Science is about questioning the status quo. Questioning authority".
    In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]
      At the "end of the day", if the file size (pointers) in the OS's file system are limited to 32 bits, then you can only do what 32 bits can do. 2**32-1=4,294,967,296.

      Anyway 2GB vs 4 GB is WAY off from 1 TB! Update: Windows NTFS file pointers are NOT limited to 32 bits, even on a 32 bit systems. So the above is "right", but still "wrong".

[reply]

        Update: This is (irrelevant) the article that was originally cited in the post above!

        Sorry, but I'm not sure how an article describing the 4GB limit of files on FAT32 partitions is relevant to a claim that NTFS can only handle 2GB files?

        For the real skinny, go to the source. Besides which I used files far greater than 2GB on my old thinkpad running NT4/NTFS. From memory the largest I ever created was 32GB, but when I first got my current machine I used contig to create a 500GB contiguous file just for the fun of it. I don't have enough diskspace to repeat the process, but I did just create one 16GB.

        Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error.
        "Science is about questioning the status quo. Questioning authority".
        In the absence of evidence, opinion is indistinguishable from prejudice.
[reply]

In Section Seekers of Perl Wisdom