Danny has asked for the wisdom of the Perl Monks concerning the following question:

file1 has size 2147483648 (2**31)
file2 has size 2147483647 (2**31-1) 
perl -we 'my $x = `cat file1`' # seg faults

[download]
perl -we 'my @x = `cat file1`' # works fine

[download]
perl -we 'my $x = `cat file2`' # works fine

[download]
perl -we 'my $x = `cat file2`; $x .= `cat file2`' # works fine

[download]
So it seems that in scalar context if the output of the command in backticks is greater than 2**31-1 then it segfaults. It seems the backticks use some buffer that has a size limit. The results were the same on 64 bit linux and cygwin.

Update: Curiously if I do: 

perl -we 'my $x = `cat file2 file2`'

[download]
I get a 
Out of memory in perl:util:safesysmalloc (cygwin perl 5.40)

[download]
or 
Out of memory! (linux perl 5.38)

[download]
instead of a 
Segmentation fault

[download]

Replies are listed 'Best First'.
Re: $x = `cat big_file` seg faults
by tonyc (Hermit) on Sep 13, 2024 at 02:28 UTC
[reply]
Re: $x = `cat big_file` seg faults
by LanX (Saint) on Sep 09, 2024 at 22:54 UTC
    I'm not sure why you get different results, but I'm pretty sure using Perl IO via open is better suited than buffering stuff from qx

    Anyway from the docs

    • In scalar context, it comes back as a single (potentially multi-line) string, or undef if the shell (or command) could not be started. In list context, returns a list of lines (however you've defined lines with $ or $INPUT_RECORD_SEPARATOR), or an empty list if the shell (or command) could not be started./
    So your list context might lead to splitting into smaller parts ("lines")

    Cheers Rolf
    (addicted to the Perl Programming Language :)
    see Wikisyntax for the Monastery

[reply]
Re: $x = `cat big_file` seg faults
by choroba (Cardinal) on Sep 10, 2024 at 10:01 UTC
    I can confirm the behaviour. OS is Linux, Ubuntu 22.04.

    With Perl 5.34.0:

    $ time perl -wE 'my $x = `cat 2`'

real    0m4.558s
user    0m0.802s
sys    0m3.192s

$ time perl -wE 'my $x = `cat 1`'
Segmentation fault (core dumped)

real    1m28.649s
user    0m0.430s
sys    0m2.341s

    [download]

    With Perl 5.39.10: 

    $ time blead -wE 'my $x = `cat 2`'

real    0m8.851s
user    0m0.980s
sys    0m3.795s

$ time blead -wE 'my $x = `cat 1`'
Segmentation fault      (core dumped)

real    0m5.694s
user    0m0.397s
sys    0m1.378s

    [download]

    So the newer perl still fails, but much faster :-)

    map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]
[select]

      So the newer perl still fails, but much faster :-)

      So much for "Perl is slow". We even optimize the performance of bugs ;-)

      But really, i rather suspect that after the first run with 5.34, the operating system cached parts of the file in memory.

      PerlMonks XP is useless? Not anymore: XPD - Do more with your PerlMonks XP
      Also check out my sisters artwork and my weekly webcomics
[reply]
        You are right. Repeated experiments show comparable times across Perl versions.

        map{substr$_->[0],$_->[1]||0,1}[\*||{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^*ARGV,3]
[reply]
[d/l]

Back to Seekers of Perl Wisdom