Re^2: Regex combining /(foo|bar)/ slower than using foreach (/foo/,/bar/) ???

Thanx for enlightment.

Using index isn't an option here the program I delivered is an tailored down to verify the original problem.

The original problem is that I have THOUSANDS of Regexes to match and replace (not just fixed strings like /foo/ ) in a string that is a few hundred k'Bytes long.

Any further suggestions - Jolly

Comment on Re^2: Regex combining /(foo\|bar)/ slower than using foreach (/foo/,/bar/) ???

Replies are listed 'Best First'.
Re^3: Regex combining /(foo\|bar)/ slower than using foreach (/foo/,/bar/) ??? by kvale (Monsignor) on Feb 18, 2005 at 16:44 UTC
If your regexes are more complex than simple fixed strings, then it would seem that optimization-destroying alternation would have less of an effect than on speed than the examples above. The regex engine optimises the search for fixed strings down to a fast Boyer Moore algorithm (really fast; I once spent a week trying to tune it with different versions of BM I know - it beat all comers!) but more general patterns need to use the full engine, But if you are so heavily rewriting a string, perhaps it is time to rethink your algorithm. One problem might be that the engine needs to search through the whole string for every search and replace pattern and that search tests the pattern at every position in the string. If you know that the string has a lot of structure and that a given pattern could only match at certain places, attempting a match at every position is a lot of wasted effort. You mentioned XML; if you are replacing particular attributes or elements, it may be a whole lot faster to parse the file using XML::Parser with the lightwieght SAX interface, replacing things as you go. That way the string is only traversed once and replacement becomes, e.g., a simple hash lookup. Update: added a phrase. -Mark	[reply]

Replies are listed 'Best First'.

Re^3: Regex combining /(foo|bar)/ slower than using foreach (/foo/,/bar/) ???
by kvale (Monsignor) on Feb 18, 2005 at 16:44 UTC

But if you are so heavily rewriting a string, perhaps it is time to rethink your algorithm. One problem might be that the engine needs to search through the whole string for every search and replace pattern and that search tests the pattern at every position in the string. If you know that the string has a lot of structure and that a given pattern could only match at certain places, attempting a match at every position is a lot of wasted effort.

You mentioned XML; if you are replacing particular attributes or elements, it may be a whole lot faster to parse the file using XML::Parser with the lightwieght SAX interface, replacing things as you go. That way the string is only traversed once and replacement becomes, e.g., a simple hash lookup.

Update: added a phrase.

-Mark

[reply]