Anonymous Monk has asked for the wisdom of the Perl Monks concerning the following question:

Greetings,

I need to figure out the size of large files (several GB). A simple -s results in an overflow as you can see in my logged session. Is my perl broken? Is perl broken (in this particular case)? 
# ls -l foo
-rw-r--r--  1 root  wheel  8366218824 Apr 24 23:38 foo
# perl -e 'print (-s "foo")."\n";'
-223715768
# 
# perl -e '($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
$atime,$mtime,$ctime,$blksize,$blocks) = stat "foo"; print "$size\n";'
-223715768
# perl -e 'use POSIX; ($dev,$ino,$mode,$nlink,$uid,$gid,$rdev,$size,
$atime,$mtime,$ctime,$blksize,$blocks) = POSIX::stat "foo"; print "$size\n";'
-223715768

Replies are listed 'Best First'.
Re: get the size of a LARGE file
by dws (Chancellor) on Apr 25, 2002 at 07:57 UTC
    I need to figure out the size of large files (several GB). A simple -s results in an overflow as you can see in my logged session. Is my perl broken?

    This may be a Perl problem, or it may be an OS large file rough-edge. Unless the underlying stat provides a 64 bit size, the best you're going to be able to do is turn $blksize and $blocks into instances of Math::BigInt, and multiply the two use this to get the approximate file size.

    (On second thought: if the file can get that large, the OS stat should provide a 64-bit interface. You probably need to update your Perl.)

[reply]
[d/l]

      The underlying stat structure has an 64 bit st_size. But the idea with $blksize and $blocks works even w/o Math::BigInt. I may even get the accurate size through modulus of the broken negative size.

      Thank you!

      --Björn

[reply]

      Just one thing I forgot: st_blksize is not the size of one block, but the preferred I/O block size for the file system.

      So you have to get the size of one block somewhere else.

      --Björn

[reply]

        Hi,
        beside the fact that st_blksize is not the blocksize but only the system preferred bloksize, there is a much more upsetting problem. Multiplying the number of blocks by the size of one block does not give you the size of the file. See the following example, where the file a just contains 4 carcters. 

        @a = stat "a";
print "Real size = $a[7]\n";
print "blocksize * blocknum = ",$a[11]*$a[12],"\n";

        [download]
        prints 
        size = 4
blocksize * blocknum =  32768

        [download]
        and everyone will agree that a file size function which returns 32768 instead of 4 is quite wrong.

        Of course, one my object that 32768 is the space that the file is taking up on your HD, but this is another story, as Kipling would have said.

        Cheers
        Leo TheHobbit
[reply]
[d/l]
[select]
Re: get the size of a LARGE file
by samtregar (Abbot) on Apr 25, 2002 at 07:54 UTC
    What version of Perl is this? Did you compile it with 64bit integers? Try building 5.6.1 with 64bit ints and see if that helps. I think it will but your system's underlying stat() might get in the way.

    -sam

[reply]

      On my system (FreeBSD 4.5, x86) st_size is off_t which is a long long unsigned (64 bit). The perl version is 5.005_03, which is shipped as part of the os.

      I will try some workarounds and if I don't succeed I will give 5.6.1 a try.

      Thank you.

      --Björn

[reply]
Re: get the size of a LARGE file
by shotgunefx (Parson) on Apr 25, 2002 at 07:59 UTC
    Not sure why it is doing it but you could multiply $blocksize * $blocks to get it. It may have something to do with using 32-bit signed integers but it's not a pow of 2, it's slighly more than 2^31. Weird.

    For fun sake, try initilizing $size with a float and see if it works after that. I belive if you initilize it with a float, it will stay a double precision. If it works then I would think it would have to be overflowing the int.

    -Lee

    "To be civilized is to deny one's nature."
[reply]
Re: get the size of a LARGE file
by asdfgroup (Beadle) on Apr 25, 2002 at 16:05 UTC
    Also Perl have to be compiled with largefiles support. in Otehr case you can work only with files up to 2 GB
    At least redhat's shipped perl 5.00503 compiled without largefiles support
[reply]
Re: get the size of a LARGE file
by asdfgroup (Beadle) on Apr 25, 2002 at 16:11 UTC
    You can check it with help of perl -V my Perl return : 
    Summary of my perl5 (revision 5.0 version 6 subversion 0) configuratio
+n:
  Platform:
    osname=linux, osvers=2.2.17-8smp, archname=i386-linux
    uname='linux porky.devel.redhat.com 2.2.17-8smp #1 smp fri nov 17 
+16:12:17 est 2000 i686 unknown '

... cuted ...

    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultipl
+icity=undef
    useperlio=undef d_sfio=undef uselargefiles=undef
    use64bitint=undef use64bitall=undef uselongdouble=undef usesocks=u
+ndef
 
... cuted ...
 
Characteristics of this binary (from libperl):
  Compile-time options:
  Built under linux
  Compiled at Mar 23 2001 12:49:50
  @INC:
    /usr/lib/perl5/5.6.0/i386-linux
    /usr/lib/perl5/5.6.0
    /usr/lib/perl5/site_perl/5.6.0/i386-linux
    /usr/lib/perl5/site_perl/5.6.0

    [download]
    so my Perl doesn't support large files and Big64
[reply]
[d/l]

Back to Seekers of Perl Wisdom