iburrell has asked for the wisdom of the Perl Monks concerning the following question:

First the rant, then the question.

Some code I am maintaining has a really bad regular expression for canonicalizing line endings. It is supposed to process a potentially multi-line input field and canonicalize changed the line ending to CRLF. 

$main::stateIn{'address'} =~ s/(?<=[^\015])[\012\015](?=[^\012])/\015\
+012/g;

[download]
Can anyone spot the problem? It removes the character before and after the line ending. This is a big problem when the second line is an address and "1201 Some St" turns into "201 Some St". The package goes to the wrong address and the customer doesn't get his order.

The question: what is the best regular expression for transforming line endings characters. It should handle Windows (\015\012), Unix (\012), and Mac (\015). It should turn them into \n, and the one that Perl handles best.

This is easy when the line endings of the input is known, like with the standard DOS to Unix: s/\r\n/\n/g. It is much harder to get right when the format is unknown like with web forms. One problem area is multiple blank lines. The first attempt I tried with look-ahead and look-behind fails with blank lines: 

s/(?<=[^\015])[\012\015](?=[^\012])/\015\012/g;

[download]

Replies are listed 'Best First'.
•Re: Line ending transformation
by merlyn (Sage) on Feb 28, 2003 at 20:44 UTC
    Can anyone spot the problem? It removes the character before and after the line ending.
    No it doesn't. Those are assertions, not included in the part of the string being replaced. Your problem must lie elsewhere.

    -- Randal L. Schwartz, Perl hacker
    Be sure to read my standard disclaimer if this is a reply.

[reply]
      I posted the corrected code by mistake. The original version didn't have assertions. 
      $main::stateIn{'address'} =~ s/[^\015][\012\015][^\012]/\015\012/g;

      [download]
      The code with assertions works properly with single lines but fails with multiple blank lines.
[reply]
[d/l]
[reply]
[d/l]
Re: Line ending transformation
by OM_Zen (Scribe) on Feb 28, 2003 at 21:02 UTC
    Hi ,

    The pattern match within the look_ahead and look_behind positive assertions are not included int he pattern itself and hence are not removed .

    This is an example of using the $' and $`

    You can capture the pattern before the match with $` and after the match with $' and remove it . The above code is just an example of that

    I am not sure if you want to do tsomething like this or what , but I am sure the look_behind and look_ahead assertions are not as you have used 
    
     if ($_=~ /(\012\015)/){

       local $prev = $`;
       local $nxt = $';

       s/$prev//g;
       s/$next//g;
 
    }

    [download]


[reply]
[d/l]
Re: Line ending transformation
by OM_Zen (Scribe) on Feb 28, 2003 at 21:11 UTC
    Hi ,

    The note by merlyn is simple and best . I posted my previous message not seeing merlyn's reply and yours too . You can ignore it , just that the extended pattern match with assertions was not included in the pattern and that is what I had to mention

[reply]
      Such a comment best goes into an update to your original node. Simply put <strong>Update</strong> and your update at the end of your orignial node. You know that you can change/update most of your nodes (apart from root nodes in some sections), don't you?

      -- Hofmator

[reply]
[d/l]

Back to Seekers of Perl Wisdom