matth has asked for the wisdom of the Perl Monks concerning the following question:

There is a problem with the regexp:

if ($line =~ /\s{3}\<gene\sid\s\=\s\"(\d{1,6})\"\slabel\s\=\s\"([. +|\.]{1,40})\"\>/){

For working with with line.

<gene id = "3" label = "gene_of_interest">
I don't see the problem with the regexp
/^\s{4}\<gene_seq\sid\s\=\s\"(\d{0,6})\"\sstatus\s{0,2}\=\s{0,2} \"(.{0,50})\"\s{0,2}CDS_number\s{0,2}\=\s{0,2}\"(\d{1,3})\"\s{0,2} number_of_CDSs\s{0,2}\=\s{0,2}\"(\d{0,5})\"\s{0,2}sequence_source \s{0,2}\=\s{0,2}\"(.{0,300})\"\s{0,2}startpos\s{0,2}\=\s{0,2}\" (\d{0,9})\"\s{0,2}endpos\s{0,2}\=\s{0,2}\"(\d{0,9})\"\s{0,2} startopen\s{0,2}\=\s{0,2}\"(\d{0,1})\"\sendopen\s{0,2}\=\s{0,2}\"( \d{0,1})\"\s{0,2}complement\s{0,2}\=\s{0,2}\"(.{0,1})\"\>/
for working with the input line:
<gene_seq id = "3" status = "Sanger source DNA code" CDS_number = "1" number_of_CDSs = "" sequence_source = "/data/databases/flatfiles/seque +nces/species/genome/embl/ch1_Sp.embl" startpos = "125676" endpos = "126224" startopen = "1" endopen = "1" complement = "F">

I have tried using the /x to enable white spaces without success. I think that maybe I need to return to my regexp chapters.

Replies are listed 'Best First'.
(jeffa) Re: Regexp problems
by jeffa (Bishop) on Dec 16, 2002 at 15:32 UTC
    I do not have a solution for you ... just some possible work arounds.

    If you are storing gene sequences in an XML-like markup, why not just use XML? Now you have the power of XML::Parser, XML::Twig, etc. at your disposal. You can also use DTD to enforce rules.

    Those regexes look like they will be a nightmare to maintain ... i would look for another solution. Besides, do attributes really have to be in a certain order with exactly one space inbetween? Sounds very fragile.

    UPDATE:
    Here is some code for you to play with, just install XML::Simple first. Notice how easy it is for me to get the individual attributes from the XML tag:

    use strict; use warnings; use XML::Simple; my $data = do {local $/;<DATA>}; my $xml = XMLin($data); print $xml->{label}, "\n"; print $xml->{id}, "\n"; __DATA__ <?xml version="1.0" ?> <gene id = "3" label = "gene_of_interest" />

    jeffa

    L-LL-L--L-LL-L--L-LL-L--
    -R--R-RR-R--R-RR-R--R-RR
    B--B--B--B--B--B--B--B--
    H---H---H---H---H---H---
    (the triplet paradiddle with high-hat)
    
Re: Regexp problems
by diotalevi (Canon) on Dec 16, 2002 at 15:49 UTC

    That's tremendously ugly. I'm not even going to try to make the code work. In fact, all I'm doing here is making it somewhat readable so your code can be worked with. This code is still very fragile and you should be using a proper parser. I figure that whether you use this or not it's a good example of how to rewrite your regex for readability and what to do with the /x parameter.

    The specific things I did were throw away all those \s{0,2} constructs in favor of \s+. The .{0,50} construct was completely breaking things so I replaced those with [^"]+ since that better expresses your intent. And of course I used /x so I could throw white space into the expression and format it for readability.

    m{<gene_seq \s+ id \s+ = \s+ "(\d+)" \s+ status \s+ = \s+ "([^"]+)" \s+ CDS_number \s+ = \s+ "(\d+)" \s+ number_of_CDSs \s+ = \s+ "(\d+)" \s+ sequence_source \s+ = \s+ "([^"]+)" \s+ startpos \s+ = \s+ "(\d+)" \s+ endpos \s+ = \s+ "(\d+)" \s+ startopen \s+ = \s+ "(\d+)" \s+ endopen \s+ = \s+ "(\d+)" \s+ complement \s+ = \s+ "([^"]+)" \s+ >}x

    Fun Fun Fun in the Fluffy Chair

Re: Regexp problems
by BrowserUk (Patriarch) on Dec 16, 2002 at 17:00 UTC

    I think that the specific problem you are having lies not with your regex, but with your data.

    You appear to be missing an '=' character

    ... sequence_source "/data/ ...

                   ^ HERE!

    Your regex would probably work if that was corrected.

    That said, you might like to try this regex.

    my $eq_qq = qr/ \s+ = \s+ "([^"]*?)" \s* /x; #!" my $long_regex = qr/^\s+ <gene_seq\s+ id $eq_qq status $eq_qq CDS_number $eq_qq number_of_CDSs $eq_qq sequence_source $eq_qq startpos $eq_qq endpos $eq_qq startopen $eq_qq endopen $eq_qq complement $eq_qq > /x; #!" if ($line =~ $long_regex) { print "$1\n$2\n$3\n$4\n$5\n$6\n$7\n$8\n$9\n$10"; }

    Once the = char is added to your data, this captures all 10 fields

    c:\test>220232 3 Sanger source DNA code 1 /data/databases/flatfiles/sequences/species/genome/embl/ch1_Sp.embl 125676 126224 1 1 F

    Now this version of the regex is not as strict in verification of the format of the data as your original, but it is way easier to maintain. If you really need to verify the length and type of the captured fields, you can do this once you have captured them. That way, if the data is corrupted, you will know which field is bad which would simplify the problem of correction.

    Also, you can make life easier for yourself by using the regex to capture the bits your interested in directly into named vars or an array. Using the two part regex above that if statement could become either:

    my @fields = $line =~ $long_regex; if (@fields == 10) { # process $fields[0] .. $fields[9] here }

    or

    my ($id, $status, $CDS_number, $number_of_CDSs, $sequence_source, $startpos, $endpos, $startopen, $endopen, $complement) = $line =~ $long_regex; # Updated: Added ~ if ( $id =~ /\d{0,6}/ ) { print "ID:", $id, "appears to be valid\n"; # Use $id... } ...

    There are many ways to improve this further, but it may get you started.

    Update: It might also be worth pointing out that unless you are in control of the source of this data, or the sample you gave is not a complete line, this is not XML. It is XML-like, but unless there is a closing tag or you ommited the / before the final >, then non of the XML parsers are likely to help you.


    Examine what is said, not who speaks.

Re: Regexp problems
by rdfield (Priest) on Dec 16, 2002 at 15:32 UTC
    Have you tried breaking down the regex piece by piece to see where it starts to go wrong?

    rdfield

Re: Regexp problems
by Monky Python (Scribe) on Dec 16, 2002 at 15:45 UTC
    Hi,
    you are checking for three whitespaces. Is this ok?
    It seems to me you only have two.
    MP
      I missed that third white space in the copy and paste. Well spotted. But that is not the problem.
Re: Regexp problems
by graff (Chancellor) on Dec 17, 2002 at 01:47 UTC
    I would be wary of depending on a regex that expects a rigid order of tag attributes. BrowserUk's comments about the quality of the data are well taken, and I would just modify his approach so that you can simply take any tag and parse it into its various attributes, using a hash to keep track of them, so you don't need to worry about what order they happen to have been written in:
    if ( /<gene_seq ([^>]+)>/ ) { my $attribString = $1; # The challenge is to split the line properly into key/val # pairs; first, let's get rid of boundary stuff that might # get in the way: $attribString =~ s{^\s+}{}; # initial whitespace $attribString =~ s{[/\s]+$}{}; # final whitespace and/or "/" # now split into key,val pairs, assuming that " = " falls # between attrib name and attrib value, while the boundary # between a value and the next attrib name is whitespace # preceded by double-quote and followed by alphanumeric: my %attribHash = split( /\s+=\s+|(?<=\w\")\s+(?=\w)/, $attribStrin +g ); # That's it; now work with the keys and values of that hash...
    Note that this preserves the double quotes around attrib values. If there's a flaw in the data, like the one that BrowserUk pointed out, this will produce a very confused result -- which you would detect if you happened to find double-quote characters in any of the hash keys. So you could follow that split with:
    die "Bad xml tag:\n$_\n" if ( grep( /\"/, keys %attribHash );