harangzsolt33 has asked for the wisdom of the Perl Monks concerning the following question:

I recently saved the contents of a 128gb SSD which had Sparky Linux on it as a single file on my large 2tb NTFS hard drive. Since the Linux partition was taking up only the first 40GB of the hard drive, I wanted to shorten the file. So, I opened a text editor (Keep in mind that I am using Windows XP Pro SP2 32-bit with TinyPerl 5.8), and I type the following: 

#!/usr/bin/perl -w

use strict;
use warnings;

# shorten file to 50GB:
truncate "K:\\LINUX\\SparkyLinux.bin", 50000000000;

[download]

And I was surprised, because nothing happened! And truncate doesn't even have a return value about whether it succeeded or not. Well, it was evident that it failed. But why??? I then booted into Sparky Linux 7.2 which is a Debian based 64-bit Linux with Perl 5.36, and I typed the following and it worked perfectly:

truncate "/media/owner/DATA 2TB/LINUX/SparkyLinux.bin", 50000000000;

Now, I have used truncate() function before in Windows on much smaller files and it worked. So, I'm wondering if maybe the truncate() function does not work in Windows XP when we're dealing with larger files? I tried to write this same code in other languages, and those codes failed too. Weird...

I have been aware that QBASIC 1.1 in DOS has a limitation of 2GB when it comes to reading/writing binary files, and it's because they store the file pointer in a signed long, which goes from -2 billion to +2 billion. That forces a limit on how far you can move the file pointer for read and write operations. The LOF() function which returns the file size in QBASIC is also affected by issue, because it returns a negative value every time you're checking a file's size that is greater than 2GB. Looks like these older operating systems can't deal with big numbers correctly. :/

Replies are listed 'Best First'.
Re: file truncate not working on 128GB file in Windows
by LanX (Saint) on Feb 06, 2024 at 05:35 UTC
    Hard to tell...

    Apparently you didn't

    • check the return code
    • didn't use seek beforehand
    • didn't tell us if an exception occured

    From truncate

      • truncate FILEHANDLE,LENGTH
      • truncate EXPR,LENGTH

      Truncates the file opened on FILEHANDLE, or named by EXPR, to the specified length.

      Raises an exception if truncate isn't implemented on your system. Returns true if successful, undef on error.

      The behavior is undefined if LENGTH is greater than the length of the file.

      The position in the file of FILEHANDLE is left unchanged. You may want to call seek before writing to the file.

      Portability issues: "truncate" in perlport.

    > I tried to write this same code in other languages, and those codes failed too

    Probably a problem for Windows XP (!) to handle a Linux FS.

    So not really a Perl problem.

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    see Wikisyntax for the Monastery

[reply]
      ... to handle a Linux FS.

      [download]

      it's just a binary file

[reply]
[d/l]
      That explains something. I didn't realize that truncate returns undef if it fails. I thought it was supposed to return a zero or empty string. Oh, well, it's in the manual. Doh! Why didn't I see that? Anyway, I thought that was weird. I wonder what's the limit where truncate will not fail. I know that earlier versions of Windows XP before Service Pack 2 had a limitation of 120GB. They couldn't access physical storage space above the 120gb limit, which is also really unexplainable. weird. Well, I think, it's time for me to transition to Linux. I've used Windows all my life, but it's becoming increasingly uncomfortable and I have to live with so many limitations...
[reply]
        Did you see perlport#truncate ?

        > (Win32) If a FILEHANDLE is supplied, it must be writable and opened in append mode (i.e., use open(my $fh, '>>', 'filename') or sysopen(my $fh, ..., O_APPEND|O_RDWR). If a filename is supplied, it should not be held open elsewhere.

        Probably that's an indication what's going wrong with TinyPerl. And are you sure the file isn't opened otherwise?

        Cheers Rolf
        (addicted to the Perl Programming Language :)
        see Wikisyntax for the Monastery

[reply]
[d/l]
[select]
Re: file truncate not working on 128GB file in Windows
by sectokia (Friar) on Feb 08, 2024 at 03:34 UTC
    If $Config{archname} is not MSWin32-x64 then you cannot do anything on files greater than 4GB.
[reply]
      Are you sure about that? Win32 API for 32-bit hosts includes functions for working with 64 bit file offsets, and 32-bit perl can be compiled with 64-bit IV. I would think the config to check would be $Config{ivsize}. Well, unless you know for sure that some detail of 32-bit windows perl builds prevented 64-bit file offsets.
[reply]
[d/l]
        Are you sure about that?

        Good question.
        I readily accepted sectokia's assertion, but I don't really know.

        One can (and I routinely do) build "MSWin32-x86" perls on my 64-bit Windows 11 box.
        Would the same limitation apply to those perls ?
        Some of those 32-bit builds have ivsize of 4, others have ivsize of 8, but they all have ptrsize of only 4.

        Cheers,
        Rob
[reply]
      Well, TinyPerl 5.8 says "MSWin32-x86-multi-thread"

      and I can confirm that it can open a 7GB file and seek way past the 4GB mark and correctly read from the file using the sysopen, sysseek, and sysread functions.

      The 4GB limitation applies to DOS, because the DOS system calls limit the file pointer to 4GB. So, you can't seek past the 4GB mark with programs written for DOS mode. EDIT: Actually, I tested Perl 5.004, and it won't even open the file that is larger than 4GB. But that's ancient technology right there.

[reply]
        For the sake of research, can you confirm the $Config{ivsize} of your tinyperl?
[reply]
[d/l]

Back to Seekers of Perl Wisdom