chiburashka has asked for the wisdom of the Perl Monks concerning the following question:

What is the fastest way to search for a text that is in a variable (e.g. $text) in a file (e.g. data.txt) ?

one way i thought about is opening the file and going line by line and checking if $text is there( by using =~). but it takes some time when the file get's big (like 3 or 4 Megabites).
So how can i make it faster ?

Replies are listed 'Best First'.
Re: Search a text in a file
by tinita (Parson) on Aug 03, 2004 at 13:08 UTC
    checking if $text is there( by using =~)
    if $text is supposed to be a regex, you won't have another choice. but if it's only text why don't you use the index()-function or the 'eq' opreator for that?

    anything else would depend on your text file, i'd say. do you have sorted lines or something like that? then you ca do a binary search which will be much quicker.

    update: and if you have grep available on your system i'd go for that

[reply]
Re: Search a text in a file
by sweetblood (Prior) on Aug 03, 2004 at 13:11 UTC
    I don't know how fast you want it but on my system with a 14 meg file I can do a trivial pattern match over the entire file in 1.2 seconds.
    $ time perl -e 'while (<>){$c++ if ($_=~/compaq/i)};print $c' *t
454
real        1.2
user        1.0
sys         0.1

    [download]
    Can you be more specific?

    Sweetblood

[reply]
[d/l]
    A reply falls below the community's threshold of quality. You may see it by logging in.
Re: Search a text in a file
by davidj (Priest) on Aug 03, 2004 at 19:00 UTC
    Chiburashka,

    This is OT and I may get whacked for it, but so be it.

    I have seen a marked improvement in your last two posts. The titles are descriptive, they are well formed, and they contain information that is relevant to us helping you solve your problem. As you can clearly see by the replies you have received, posts like this result in other monks taking your questions more seriously and replying in kind.

    You've been chastised a lot (and deservedly so) for the poor quality of your posts. I just wanted to take a moment and commend you for your efforts to make your posts better.

    davidj

[reply]
Re: Search a text in a file
by Aristotle (Chancellor) on Aug 03, 2004 at 14:25 UTC

    If all the data you have is a flat text file, that's how you gotta do it.

    If you are searching the same file(s) over and over and can afford some overhead in storage, you should build a so-called inverted index. That's how search engines work. There's even a module on CPAN to make this easier for you: Search::InvertedIndex.

    Makeshifts last the longest.

[reply]

Back to Seekers of Perl Wisdom