Re: Matching simple patterns - is there a faster way?

Realistically if you're downloading thousands of HTML pages and matching them against a regexp, it's unlikely to be the regexp that's slowing you down.

The first place to look at optimizing is the HTTP layer itself. When you request the pages are you sending an Accept-Encoding header? In many cases, this will allow the pages to be transmitted with compression. (Many of the Perl HTTP libs will automatically handle the decompression for you.)

If you're using LWP::UserAgent, perhaps try switching to Furl::HTTP which is somewhat faster. It doesn't have quite as many features as LWP::UserAgent, but it might have enough for you.

Could you try requesting pages in parallel rather than one at a time? If so, take a look at AnyEvent::HTTP.

perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'

Comment on Re: Matching simple patterns - is there a faster way? Download Code

Replies are listed 'Best First'.
Re^2: Matching simple patterns - is there a faster way? by mrguy123 (Hermit) on Aug 09, 2012 at 08:09 UTC
You're indeed correct that the HTML download is the bottleneck, and I am doing a lot of work to optimize that (including parallel processing) However, if I can make the regexes a bit faster it will also help, and to be honest I am also curious about the answer to this question	[reply]