in reply to Re: RE on lines read from in-memory scalar is very slow
in thread RE on lines read from in-memory scalar is very slow

I just found another interesting observation. That regex I was using matches 125,277 times (16.3% of the lines) in the example file I was using. If I change the regex to /^ ?QueryXXX/; so that it matches nothing I get: 
0.114286 read lines from disk and do RE; n=769114.
0.104568 read lines from in-memory file and do RE; n=769114.

[download]
This is probably more for the perl developers or cygwin distribution people, but still pretty quirky.

Replies are listed 'Best First'.
Re^3: RE on lines read from in-memory scalar is very slow
by swl (Prior) on Jan 23, 2024 at 06:53 UTC

    Modifying my code so 16.3% of the lines will match the regex (previously it was 1%) gives me these timings for Strawberry Perl 5.38:

    tempfile.txt is size 29 Mb
0.121952 read lines from disk and do RE (32571 matches).
0.491674 read lines from in-memory file and do RE (32571 matches).

    [download]

    So consistent with your results for Strawberry Perl.

    And for MSYS2 perl 5.38.2:

    tempfile.txt is size 29 Mb
0.064073 read lines from disk and do RE (32571 matches).
9.538524 read lines from in-memory file and do RE (32571 matches).

    [download]

    So something would appear to be awry with the regex matching under MSYS2 and Cygwin.

[reply]
[d/l]
[select]

      I ran the code using use re 'debug' and there is no difference in the regex processing.

      I then instrumented the code with some metamod::Devel::Peek Dumps. The in-memory strings have rapidly increasing amounts of memory allocated (the LEN field), plateauing at close to the size of the input string. This pattern is the same for both Strawberry Perl and MSYS2 Perl, which makes me wonder if the delay is related to memory management. Others are more qualified to comment on that front than me, though.

      Edit: Just for completeness I also tested using Perl 5.36.0 on Ubuntu via WSL and the memory usage is the same.

      Updated code is below behind inside the readmore tags.

      tempfile.txt is size 35 Mb
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x177fa280250 " Querysome random text 0.271320203145251\n"\0
  CUR = 41
  LEN = 408
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x177fa34f4d0 " Querysome random text 0.775348369818055some ran
+dom text 0.775348369818055\n"\0
  CUR = 75
  LEN = 201
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x177fa3a28b0 " Querysome random text 0.785001144808529\n"\0
  CUR = 41
  LEN = 43
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x177fa297c60 " Querysome random text 0.894431999356865\n"\0
  CUR = 41
  LEN = 309
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x177f88a13b0 " Querysome random text 0.412049736815259some ran
+dom text 0.412049736815259some random text 0.412049736815259some rand
+om text 0.412049736815259some random text 0.412049736815259some rando
+m text 0.412049736815259some random text 0.412049736815259some random
+ text 0.412049736815259some random text 0.412049736815259\n"\0
  CUR = 313
  LEN = 392
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x177fa3750d0 " Querysome random text 0.809515115277865\n"\0
  CUR = 41
  LEN = 275
  COW_REFCNT = 2
0.005142 read lines from disk and do RE (6 matches).


SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x177f88b3ed0 " Querysome random text 0.271320203145251\n"\0
  CUR = 41
  LEN = 656
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x1778000b070 " Querysome random text 0.775348369818055some ran
+dom text 0.775348369818055\n"\0
  CUR = 75
  LEN = 37784908
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x1778242b070 " Querysome random text 0.785001144808529\n"\0
  CUR = 41
  LEN = 37784634
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x1778000d070 " Querysome random text 0.894431999356865\n"\0
  CUR = 41
  LEN = 37784593
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x1778242b070 " Querysome random text 0.412049736815259some ran
+dom text 0.412049736815259some random text 0.412049736815259some rand
+om text 0.412049736815259some random text 0.412049736815259some rando
+m text 0.412049736815259some random text 0.412049736815259some random
+ text 0.412049736815259some random text 0.412049736815259\n"\0
  CUR = 313
  LEN = 37784176
  COW_REFCNT = 2
SV = PV(0x177f89db120) at 0x177f88bcca0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK)
  PV = 0x1778000f070 " Querysome random text 0.809515115277865\n"\0
  CUR = 41
  LEN = 37783182
  COW_REFCNT = 2
0.004073 read lines from in-memory file and do RE (6 matches).

      [download]
      Read more... (2 kB)
[reply]
[d/l]
[select]

        The Anonymous Monk post in 11157181 made me wonder if the the memory usage pattern changed with perl versions.

        Running against the most recent releases of Strawberry Perl for each major version, the memory usage is low for perls prior to 5.20. From 5.20 (tested with 5.20.3) it shows the large usage pattern.

        5.20 is when COW was first on by default. https://metacpan.org/release/RJBS/perl-5.20.0/view/pod/perldelta.pod#Performance-Enhancements

        FWIW, the in-memory loop runs faster than the file-read loop for Strawberry Perls 5.16 and 5.18, and then becomes slower after 5.20. The difference is just not so large on non-MSYS2 and Cygwin perls.

[reply]

      FWIW, Win10 & Strawberry:

      v5.32.1
tempfile.txt is size 29 Mb
0.115239 read lines from disk and do RE (32571 matches).
0.676642 read lines from in-memory file and do RE (32571 matches).

v5.38.0
tempfile.txt is size 29 Mb
0.156708 read lines from disk and do RE (32571 matches).
0.628221 read lines from in-memory file and do RE (32571 matches).

v5.26.3
tempfile.txt is size 29 Mb
0.122374 read lines from disk and do RE (32571 matches).
0.671405 read lines from in-memory file and do RE (32571 matches).

v5.16.3
tempfile.txt is size 28 Mb
0.119628 read lines from disk and do RE (32760 matches).
0.057724 read lines from in-memory file and do RE (32760 matches).

      [download]
[reply]
[d/l]

        Couple other additions to assortment of facts:

        • if loops are changed to

          while(my $s = <$fh>) {
  $s =~ /^ ?Query/ && $match_count1++;
}

          [download]

          then deterioration gets significantly worse (compare to parent node):

          v5.32.1
tempfile.txt is size 29 Mb
0.121957 read lines from disk and do RE (32571 matches).
9.511288 read lines from in-memory file and do RE (32571 matches).

          [download]

        • if loops are changed to

          while(my $s = <$fh>) {
  index( $s, 'Query' ) != -1 && $match_count1++;;
}

          [download]

          (the same without the "$s") then using in-memory string is (of course) faster:

          v5.32.1
tempfile.txt is size 29 Mb
0.092390 read lines from disk and do RE (32571 matches).
0.051797 read lines from in-memory file and do RE (32571 matches).

          [download]
[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom