in reply to Re^3: RE on lines read from in-memory scalar is very slow
in thread RE on lines read from in-memory scalar is very slow

I ran the code using use re 'debug' and there is no difference in the regex processing.

I then instrumented the code with some metamod::Devel::Peek Dumps. The in-memory strings have rapidly increasing amounts of memory allocated (the LEN field), plateauing at close to the size of the input string. This pattern is the same for both Strawberry Perl and MSYS2 Perl, which makes me wonder if the delay is related to memory management. Others are more qualified to comment on that front than me, though.

Edit: Just for completeness I also tested using Perl 5.36.0 on Ubuntu via WSL and the memory usage is the same.

Updated code is below behind inside the readmore tags.

tempfile.txt is size 35 Mb SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x177fa280250 " Querysome random text 0.271320203145251\n"\0 CUR = 41 LEN = 408 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x177fa34f4d0 " Querysome random text 0.775348369818055some ran +dom text 0.775348369818055\n"\0 CUR = 75 LEN = 201 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x177fa3a28b0 " Querysome random text 0.785001144808529\n"\0 CUR = 41 LEN = 43 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x177fa297c60 " Querysome random text 0.894431999356865\n"\0 CUR = 41 LEN = 309 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x177f88a13b0 " Querysome random text 0.412049736815259some ran +dom text 0.412049736815259some random text 0.412049736815259some rand +om text 0.412049736815259some random text 0.412049736815259some rando +m text 0.412049736815259some random text 0.412049736815259some random + text 0.412049736815259some random text 0.412049736815259\n"\0 CUR = 313 LEN = 392 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x177fa3750d0 " Querysome random text 0.809515115277865\n"\0 CUR = 41 LEN = 275 COW_REFCNT = 2 0.005142 read lines from disk and do RE (6 matches). SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x177f88b3ed0 " Querysome random text 0.271320203145251\n"\0 CUR = 41 LEN = 656 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x1778000b070 " Querysome random text 0.775348369818055some ran +dom text 0.775348369818055\n"\0 CUR = 75 LEN = 37784908 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x1778242b070 " Querysome random text 0.785001144808529\n"\0 CUR = 41 LEN = 37784634 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x1778000d070 " Querysome random text 0.894431999356865\n"\0 CUR = 41 LEN = 37784593 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x1778242b070 " Querysome random text 0.412049736815259some ran +dom text 0.412049736815259some random text 0.412049736815259some rand +om text 0.412049736815259some random text 0.412049736815259some rando +m text 0.412049736815259some random text 0.412049736815259some random + text 0.412049736815259some random text 0.412049736815259\n"\0 CUR = 313 LEN = 37784176 COW_REFCNT = 2 SV = PV(0x177f89db120) at 0x177f88bcca0 REFCNT = 1 FLAGS = (POK,IsCOW,pPOK) PV = 0x1778000f070 " Querysome random text 0.809515115277865\n"\0 CUR = 41 LEN = 37783182 COW_REFCNT = 2 0.004073 read lines from in-memory file and do RE (6 matches).
#!/usr/bin/env perl use warnings; use strict; use Time::HiRes qw( time ); use Devel::Peek; my $file = shift @ARGV; my ($fh, $time); if (!$file) { # should use Path::Tiny::tempfile $file = 'tempfile.txt'; open my $ofh, '>', $file or die "Cannot open $file for writing, $! +"; srand(1234567); for my $i (0..200000) { my $string = 'some random text ' . rand(); $string = $string x (1 + int (rand() * 10)); if (rand() < 0.163) { $string = " Query${string}"; } say {$ofh} $string; } $ofh->close or die "Cannot close $file, $!"; printf "%s is size %i Mb\n", $file, (-s $file) / (1028**2); } open $fh, "<", $file; $time = time; my $match_count1; my $i1 = 0; while(<$fh>) { /^ ?Query/ && $match_count1 ++; if (/^ Query/) { Dump $_; $i1 ++; last if $i1 > 5; } } printf "%f read lines from disk and do RE ($match_count1 matches).\n", + time - $time; seek $fh, 0, 0; my $s = ""; while(<$fh>) { $s .= $_; } $fh->close; #Dump $s; print "\n\n"; open $fh, "<", \$s; $time = time; my $match_count2; my $i2 = 0; while(<$fh>) { /^ ?Query/ && $match_count2++; if (/^ Query/) { Dump $_; $i2++; last if $i2 > 5; } } printf "%f read lines from in-memory file and do RE ($match_count2 mat +ches).\n", time - $time;

Replies are listed 'Best First'.
Re^5: RE on lines read from in-memory scalar is very slow
by swl (Prior) on Jan 23, 2024 at 08:11 UTC

    The Anonymous Monk post in 11157181 made me wonder if the the memory usage pattern changed with perl versions.

    Running against the most recent releases of Strawberry Perl for each major version, the memory usage is low for perls prior to 5.20. From 5.20 (tested with 5.20.3) it shows the large usage pattern.

    5.20 is when COW was first on by default. https://metacpan.org/release/RJBS/perl-5.20.0/view/pod/perldelta.pod#Performance-Enhancements

    FWIW, the in-memory loop runs faster than the file-read loop for Strawberry Perls 5.16 and 5.18, and then becomes slower after 5.20. The difference is just not so large on non-MSYS2 and Cygwin perls.

      swl et. al. For consistency, if you want to run my file you can get it at tmp.txt. Not sure if there is another way to share files here.