fazedandconfused has asked for the wisdom of the Perl Monks concerning the following question:

I'm struggling with a regex requirement on our Xymon monitoring system which uses Perl format regular expression.


I have the following data,

red Queue TEST1:DI.CATALOGUE.ITEM.IMPORT.BO has depth 10 (critical: 10, warn: 1)
red Queue TEST1:DI.PRODUCT.HIERARCHY.IMPORT.BO has depth 20 (critical: 10, warn: 1)
yellow Queue TEST1:DI.CATSETUP.PRODUCT.IMPORT.BO has depth 9 (warn: 1, critical: 10)
green Queue TEST1:ATP.CANCEL.PARCEL.CONFIRM.EXPORT.BO has depth 0 (warn: 1, critical: 10)
green Queue TEST1:ATP.CANCEL.PARCEL.CONFIRM.EXPORT.LQ has depth 0 (warn: 30, critical: -1)
green Queue TEST1:ATP.DELIVERY.DEFINITION.EXPORT.BO has depth 0 (warn: 1, critical: 10)
green Queue TEST1:ATP.DELIVERY.DEFINITION.EXPORT.LQ has depth 0 (warn: 10, critical: -1)
green Queue TEST1:ATP.FOREIGN.PARCEL.EXPORT.BO has depth 0 (warn: 1, critical: 10)
green Queue TEST1:ATP.FOREIGN.PARCEL.EXPORT.LQ has depth 0 (warn: 5, critical: -1)
green Queue TEST1:ATP.PARCEL.DESPATCH.EXPORT.BO has depth 0 (warn: 1, critical: 10)

In most situations I need to match all lines containing a string ending ".BO" which I do like this,
([A-Z.]*BO)<br>
however for this particular requirement I need to match the same but exclude the 2 lines below

DI.PRODUCT.HIERARCHY.IMPORT.BO
DI.CATSETUP.PRODUCT.IMPORT.BO

This is where my knowledge of regular expression lets me down as I can't seem to find a way to match all ".BO" except these 2.

It would appear I missed a couple of things.

1. I will probably only ever need to exclude this 2 lines
2. It needs to be a single statement (a peculiarity of Xymon monitoring)
3. The exclusions ideally need to be the whole string but could at a push be restricted to the words HEIRARCHY and CATSETUP

Replies are listed 'Best First'.
Re: Regex to match with exclusions
by kcott (Archbishop) on Oct 01, 2015 at 15:34 UTC

    G'day fazedandconfused,

    The regex you show (i.e. ([A-Z.]*BO)<br>) matches nothing!

    To match all lines containing a string ending ".BO", you could use a regex like this:

    qr{ : ( .*? \. BO \b ) }x

    This captures:

    DI.CATALOGUE.ITEM.IMPORT.BO DI.PRODUCT.HIERARCHY.IMPORT.BO DI.CATSETUP.PRODUCT.IMPORT.BO ATP.CANCEL.PARCEL.CONFIRM.EXPORT.BO ATP.DELIVERY.DEFINITION.EXPORT.BO ATP.FOREIGN.PARCEL.EXPORT.BO ATP.PARCEL.DESPATCH.EXPORT.BO

    For your data, consider capturing your "most situations" strings (as shown above) and then, separately, excluding your "particular requirement" lines. This should provide more flexibility by allowing you to conditionally include a (possibly variable) "particular requirement" regex.

    Here's my test:

    #!/usr/bin/env perl -l use strict; use warnings; my $most_situations_re = qr{ : ( .*? \. BO \b ) }x; my $particular_requirement_exclude_re = qr{ PRODUCT }x; while (<DATA>) { next unless /$most_situations_re/; my $capture = $1; next if $capture =~ /$particular_requirement_exclude_re/; print $capture; } __DATA__ red Queue TEST1:DI.CATALOGUE.ITEM.IMPORT.BO has depth 10 (critical: 10 +, warn: 1) red Queue TEST1:DI.PRODUCT.HIERARCHY.IMPORT.BO has depth 20 (critical: + 10, warn: 1) yellow Queue TEST1:DI.CATSETUP.PRODUCT.IMPORT.BO has depth 9 (warn: 1, + critical: 10) green Queue TEST1:ATP.CANCEL.PARCEL.CONFIRM.EXPORT.BO has depth 0 (war +n: 1, critical: 10) green Queue TEST1:ATP.CANCEL.PARCEL.CONFIRM.EXPORT.LQ has depth 0 (war +n: 30, critical: -1) green Queue TEST1:ATP.DELIVERY.DEFINITION.EXPORT.BO has depth 0 (warn: + 1, critical: 10) green Queue TEST1:ATP.DELIVERY.DEFINITION.EXPORT.LQ has depth 0 (warn: + 10, critical: -1) green Queue TEST1:ATP.FOREIGN.PARCEL.EXPORT.BO has depth 0 (warn: 1, c +ritical: 10) green Queue TEST1:ATP.FOREIGN.PARCEL.EXPORT.LQ has depth 0 (warn: 5, c +ritical: -1) green Queue TEST1:ATP.PARCEL.DESPATCH.EXPORT.BO has depth 0 (warn: 1, +critical: 10)

    Output:

    DI.CATALOGUE.ITEM.IMPORT.BO ATP.CANCEL.PARCEL.CONFIRM.EXPORT.BO ATP.DELIVERY.DEFINITION.EXPORT.BO ATP.FOREIGN.PARCEL.EXPORT.BO ATP.PARCEL.DESPATCH.EXPORT.BO
    "This is where my knowledge of regular expression lets me down ..."

    I don't know your level of regex knowledge. One or more of these should help:

    — Ken

Re: Regex to match with exclusions
by kroach (Pilgrim) on Oct 01, 2015 at 15:03 UTC

    You can use look-behind assertions for that:

    / ([A-Z.]* (?<!DI\.PRODUCT\.HIERARCHY\.IMPORT\.) (?<!DI\.CATSETUP\.PRODUCT\.IMPORT\.) BO) /x
    This will match only if the part behind 'BO' is not one of your special cases.

Re: Regex to match with exclusions
by GotToBTru (Prior) on Oct 01, 2015 at 15:09 UTC

    Is there a reason why you can't use more than one regex to accomplish this? I'm guessing at the exclusion criteria; but you should be able to adapt the approach if there is more to it.

    @lines = grep { /\.BO / && ! /CATSETUP|PRODUCT/ } <DATA>;

    Somebody more skilled than I will no doubt come up with a single regex solution.

    Dum Spiro Spero
Re: Regex to match with exclusions
by stevieb (Canon) on Oct 01, 2015 at 14:43 UTC

    Hi fazedandconfused,

    That's trivial for us to help you with, but what exactly is the criteria for excluding lines? ie. is it ONLY those two lines you'll ever want to skip? What word or characters in particular make it so you want to skip it?

Re: Regex to match with exclusions
by Anonymous Monk on Oct 01, 2015 at 16:44 UTC
    #!/usr/bin/perl # http://perlmonks.org/?node_id=1143557 use strict; use warnings; while(<DATA>) { print if /^(?!.*(?:DI.PRODUCT.HIERARCHY.IMPORT.BO|DI.CATSETUP.PRODUC +T.IMPORT.BO)).*\.BO /; } __DATA__ red Queue TEST1:DI.CATALOGUE.ITEM.IMPORT.BO has depth 10 (critical: 10 +, warn: 1) red Queue TEST1:DI.PRODUCT.HIERARCHY.IMPORT.BO has depth 20 (critical: + 10, warn: 1) yellow Queue TEST1:DI.CATSETUP.PRODUCT.IMPORT.BO has depth 9 (warn: 1, + critical: 10) green Queue TEST1:ATP.CANCEL.PARCEL.CONFIRM.EXPORT.BO has depth 0 (war +n: 1, critical: 10) green Queue TEST1:ATP.CANCEL.PARCEL.CONFIRM.EXPORT.LQ has depth 0 (war +n: 30, critical: -1) green Queue TEST1:ATP.DELIVERY.DEFINITION.EXPORT.BO has depth 0 (warn: + 1, critical: 10) green Queue TEST1:ATP.DELIVERY.DEFINITION.EXPORT.LQ has depth 0 (warn: + 10, critical: -1) green Queue TEST1:ATP.FOREIGN.PARCEL.EXPORT.BO has depth 0 (warn: 1, c +ritical: 10) green Queue TEST1:ATP.FOREIGN.PARCEL.EXPORT.LQ has depth 0 (warn: 5, c +ritical: -1) green Queue TEST1:ATP.PARCEL.DESPATCH.EXPORT.BO has depth 0 (warn: 1, +critical: 10)
Re: Regex to match with exclusions
by fazedandconfused (Novice) on Oct 01, 2015 at 15:17 UTC

    It would appear I missed a couple of things.

    1. I will probably only ever need to exclude this 2 lines
    2. It needs to be a single statement (a peculiarity of Xymon monitoring)
    3. The exclusions ideally need to be the whole string but could at a push be restricted to the words HEIRARCHY and CATSETUP

Re: Regex to match with exclusions
by locked_user sundialsvc4 (Abbot) on Oct 01, 2015 at 15:39 UTC

    Use any of the surroundings that you know exist.   Maybe:
    /\.BO has/

    or:
    /\.BO\b/   (using the “word boundary” match).