in reply to RE on lines read from in-memory scalar is very slow
G'day Danny,
I can confirm and reproduce the types of results you've presented. I have Cygwin running on Win10 with the latest updates as of a few days ago.
$ uname -a CYGWIN_NT-10.0-19045 titan 3.4.10-1.x86_64 2023-11-29 12:12 UTC x86_64 + Cygwin
I have a variety of Perl versions installed using Perlbrew:
$ perlbrew list * perl-5.39.3 perl-5.38.0 perl-5.36.0 perl-5.34.0 perl-5.33.5 perl-5.32.0 perl-5.30.0
I used perl-5.39.3, perl-5.36.0 and perl-5.30.0 for the tests below.
I didn't know too much about your test data. I created these:
$ perl -E 'my $test_rec = "X" x 45; open my $fh, ">", "test_data"; $fh +->say($test_rec) for 1..769114;' $ ls -l test_data -rw-r--r-- 1 ken None 35379244 Jan 23 17:09 test_data $ wc test_data 769114 769114 35379244 test_data $ head -1 test_data XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
$ perl -E 'my $test_rec = "Query" . "X" x 40; open my $fh, ">", "test_ +data_Q"; $fh->say($test_rec) for 1..769114;' $ ls -l test_data_Q -rw-r--r-- 1 ken None 35379244 Jan 23 17:23 test_data_Q $ wc test_data_Q 769114 769114 35379244 test_data_Q $ head -1 test_data_Q QueryXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
I made a verbatim copy of your posted code: "op_code.pl". Here's the test results:
v5.39.3
$ perl -v | head -2 | tail -1 This is perl 5, version 39, subversion 3 (v5.39.3) built for cygwin-th +read-multi $ ./op_code.pl test_data 0.111368 read lines from disk and do RE. 0.096441 read lines from in-memory file and do RE. $ ./op_code.pl test_data_Q 0.356852 read lines from disk and do RE. 398.759730 read lines from in-memory file and do RE.
v5.36.0
$ perl -v | head -2 | tail -1 This is perl 5, version 36, subversion 0 (v5.36.0) built for cygwin-th +read-multi $ ./op_code.pl test_data 0.118364 read lines from disk and do RE. 0.094379 read lines from in-memory file and do RE. $ ./op_code.pl test_data_Q 0.362871 read lines from disk and do RE. 399.115542 read lines from in-memory file and do RE.
v5.30.0
$ perl -v | head -2 | tail -1 This is perl 5, version 30, subversion 0 (v5.30.0) built for cygwin-th +read-multi $ ./op_code.pl test_data 0.112352 read lines from disk and do RE. 0.096379 read lines from in-memory file and do RE. $ ./op_code.pl test_data_Q 0.327187 read lines from disk and do RE. 398.730810 read lines from in-memory file and do RE.
I have ~32GB memory available; at no time during the running of these tests was more than 6.5GB used.
I see throughout this thread that both you and others have not encountered this anomaly on other operating systems: I can't readily discern a reason for it.
— Ken
|
---|
Replies are listed 'Best First'. | |
---|---|
Re^2: RE on lines read from in-memory scalar is very slow
by Danny (Chaplain) on Jan 23, 2024 at 09:59 UTC |