Re^2: use regular expressions across multiple lines from a very large input file

Replies are listed 'Best First'.
Re^3: use regular expressions across multiple lines from a very large input file by BrowserUk (Patriarch) on Dec 06, 2010 at 19:42 UTC
In order to speed up the search, I dare to suggest to choose a large value of n, Don't assume that the bigger the read, the faster it will run, it just doesn't work out that way. On my systems, 64kb reads work out marginally best (YMMV): C:\test>junk -B=4 < 1gb.dat Found 6559 matches in 10.778 seconds using 4 kb reads C:\test>junk -B=64 < 1gb.dat Found 6559 matches in 10.567 seconds using 64 kb reads C:\test>junk -B=256 < 1gb.dat Found 6559 matches in 10.574 seconds using 256 kb reads C:\test>junk -B=1024 < 1gb.dat Found 6559 matches in 10.938 seconds using 1024 kb reads C:\test>junk -B=4096 < 1gb.dat Found 6559 matches in 10.995 seconds using 4096 kb reads C:\test>junk -B=65536 < 1gb.dat Found 6559 matches in 12.533 seconds using 65536 kb reads [download] Code: `#! perl -slw use strict; use Time::HiRes qw[ time ]; our $B //= 64; $/ = \( $B *1024 ); binmode STDIN, ':raw:perlio'; my $start = time; my $count = 0; while( <STDIN> ) { ++$count while m[123]g; } printf "Found %d matches in %.3f seconds using %d kb reads\n", $count, time()-$start, $B;` [download] Examine what is said, not who speaks -- Silence betokens consent -- Love the truth but pardon error. "Science is about questioning the status quo. Questioning authority". In the absence of evidence, opinion is indistinguishable from prejudice.	[reply] [d/l] [select]
Re^4: use regular expressions across multiple lines from a very large input file by CountZero (Bishop) on Dec 06, 2010 at 23:33 UTC
Yes, these things are always quite tricky to predict. So many things may influence it: buffer-size, cache-effects, other processes stealing the disk-controller from your script or pushing your script out to disk, ... Trial & Error is probably the only way to go. CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply]
Re^3: use regular expressions across multiple lines from a very large input file by LanX (Saint) on Dec 06, 2010 at 09:48 UTC
> In order to speed up the search, I dare to suggest to choose a large value or n, say a value slightly less than the amount that causes the "Out of Memory" error. I think you mean half that size. Cheers Rolf	[reply]
Re^4: use regular expressions across multiple lines from a very large input file by CountZero (Bishop) on Dec 06, 2010 at 18:56 UTC
Why half the size? CountZero A program should be light and agile, its subroutines connected like a string of pearls. The spirit and intent of the program should be retained throughout. There should be neither too little or too much, neither needless loops nor useless variables, neither lack of structure nor overwhelming rigidity." - The Tao of Programming, 4.1 - Geoffrey James	[reply]
Re^5: use regular expressions across multiple lines from a very large input file by LanX (Saint) on Dec 06, 2010 at 21:12 UTC
One always will need to have two joined blocks of size n in memory, whenever there are no matches anymore which start in the front block one will have to delete it, treat the rear block as new front and read a new rear block from disk. `loaded Blocks \|------[++++++\|++++++]------\|------\|---\| file A B C D E F <-----> match` [download] Actually I'm not sure if joining two strings can be done without needing twice as much memory! Anyway I don't think going to extremes is a good idea... Cheers Rolf	[reply] [d/l]
Re^6: use regular expressions across multiple lines from a very large input file by CountZero (Bishop) on Dec 06, 2010 at 23:27 UTC
Re^7: use regular expressions across multiple lines from a very large input file by LanX (Saint) on Dec 06, 2010 at 23:55 UTC