splitting text into lines -- code -> regex

theorbtwo has asked for the wisdom of the Perl Monks concerning the following question:

Replies are listed 'Best First'.
Re: splitting text into lines -- code -> regex by BrowserUk (Patriarch) on Jul 12, 2004 at 08:32 UTC
`#! perl -slw use strict; while( my $line = <DATA> ) { chomp $line; my @l27s; push @l27s, pack 'a27', substr( $line, 0, 1+rindex( $line, ' ', 27 ), '' ) while length $line > 27; push @l27s, pack 'a27', $line; print '123456789012345678901234567', $/, '------'; print "$_<"for @l27s; print '--------', $/; } __DATA__ 1234567890 VON 14.06.2004 BUCHUNGSKONTO 1234567890 012345678-A1234567 INCL.EUR 3,31 MWST JULI MONATL. GEB HR T-DSL FLAT 0 +1.07.04-3` [download] Gives `P:\test>373533 123456789012345678901234567 ------ 1234567890 VON 14.06.2004 < BUCHUNGSKONTO 1234567890 < -------- 123456789012345678901234567 ------ 012345678-A1234567 INCL.EUR< 3,31 MWST JULI MONATL. GEB < HR T-DSL FLAT 01.07.04-3 < --------` [download] Examine what is said, not who speaks. "Efficiency is intelligent laziness." -David Dunham "Think for yourself!" - Abigail "Memory, processor, disk in that order on the hardware side. Algorithm, algoritm, algorithm on the code side." - tachyon	[reply] [d/l] [select]
Re: splitting text into lines -- code -> regex by theorbtwo (Prior) on Jul 12, 2004 at 07:58 UTC
Examples: `012345678901234567890123456789012345678901234567890 1234567890 VON 14.06.2004 BUCHUNGSKONTO 1234567890` [download] Should become: `123456789012345678901234567 1234567890 VOM 14.06.2004__ BUCHUNGSKONTO 1234567890___` [download] (underscores and count on first line added for clarity) `0123456789012345678901234567890123456789012345678901234567890123456789 +012345678901234567890123456 R12345678-A1234567 INCL.EUR 3,31 MWST JULI MONATL. GEB HR T-DSL FLAT +01.07.04-31.07.04 12,34` [download] Should become: `123456789012345678901234567 R12345678-A1234567 INCL.EUR 3,31 MWST JULI MONATL. GEB_ HR T-DSL FLAT______________ 01.07.03-31.07.04 12,34` [download] (underscores and count on first line added for clarity) (Note that in this case, my algorithm doesn't split it in quite the same place that the original did -- and not that yes, DTAG does spell Gebuehr as two words, for some strange reason, for those that know German.) It appears that my bank is doing (the equivlent of) `$comments = join (' ', '', map {s/ *$//; $_} @comments;`; I'm attempting to undo that, and get back @comments. Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).	[reply] [d/l] [select]
Re^2: splitting text into lines -- code -> regex by davido (Cardinal) on Jul 12, 2004 at 08:13 UTC
`use strict; use warnings; my $text = " R12345678-A1234567 INCL.EUR 3,31 MWST JULI MONATL. GEB +HR T-DSL FLAT 01.07.04-31.07.04 12,34"; while ( $text =~ / (.{0,27}(?!\S))/g ) { my $out = $1 . ( '_' x ( 27 - length( $1 ) ) ); print "$out\n"; }` [download] Underscores added to output for clarity. Update: Just for the record, `(?!\S)` is a negative lookahead assertion. It ensures that lines are always split on the last possible space character up to the 27 character line limit. In other words, each line must contain a leading space, followed by up to 27 characters, and the next character can't be a non-space. That way the next line will always synch up with the requisite 'space' delimiter. Dave	[reply] [d/l] [select]
Re^2: splitting text into lines -- code -> regex by bart (Canon) on Jul 12, 2004 at 08:40 UTC
This regex seems to match both your examples. `/(?:\s\|^)(.{1,27})(?=\s\|$)/g` [download] You must do some padding, with sprintf or pack for example, to append extra padding spaces. `@padded = map { sprintf "-%27s", $_ } /(?:\s\|^)(.{1,27})(?=\s\|$)/g; @padded = map { pack "A27", $_ } /(?:\s\|^)(.{1,27})(?=\s\|$)/g;` [download]	[reply] [d/l] [select]
Re^2: splitting text into lines -- code -> regex by pelagic (Priest) on Jul 12, 2004 at 09:19 UTC
To me this looks like a wrapping problem. And there's a CPAN Module out there that might help: Text::Wrap. pelagic	[reply]
Re^2: splitting text into lines -- code -> regex by graff (Chancellor) on Jul 13, 2004 at 03:08 UTC
I would agree with pelagic: it smells like a wrapping problem. Looking at your second example, what if it had been: `0123456789012345678901234567890123456789012345678901234567890123456789 +01234567890123 R12345678-A1234567 INCL.EUR 3,31 MWST JULI MONATL. GEB HR T-DSL FLAT +01.07.04 12,34` [download] Should it then come out like this? `123456789012345678901234567 R12345678-A1234567 INCL.EUR 3,31 MWST JULI MONATL. GEB_ HR T-DSL FLAT 01.07.04_____ 12,34______________________` [download] The point is that, if your resulting @comments elements must all be 27 characters long, with space-padding at the end where necessary, and you must not break up any existing (\S+) token across elements, just add tokens to a current element string until the next token would put it over 27 characters, pad to 27 if necessary, and make the next token the start of the next element. I believe Text::Wrap supports this sort of logic, but it doesn't seem that hard to roll it from scratch. (Or maybe the bank really is doing something more complicated than that, or maybe they just screwed up big-time and made it impossible for you to reliably do what you want.)	[reply] [d/l] [select]
Re^3: splitting text into lines -- code -> regex by theorbtwo (Prior) on Jul 13, 2004 at 05:29 UTC
It does sound an awful lot like a wrapping problem. However, with reference to your parenthetical: The bank already makes it impossible to figure out exactly what the original was, by de-wrapping. The reason is that the 27-character lines can be mostly spaces. For example, the dewrapping I gave for the T-DSL example above was how my algo de-wrapped it. The original looks like (pipes added to show end-of-field): `R12345678-A1234567 INCL.EUR\| 3,31 MWST JULI \| MONATL. GEB HR T-DSL FLAT \| 01.07.04-31.07.04 23,99\|` [download] Unfornatly, there is no way of telling that without domain knowladge. (So how do I know that? I load the more-info page on that transaction, but I can't figure out how to do that programatacilly reliably, as the bank keeps changing session IDs. Worse, if I get it wrong, I don't get an error reliably -- I simply get information on the wrong transaction!) As to using Text::Wrap... well, I was going to point out the bit in the docs where it says "it will destory any whitespace in the original text", but it turns out that only applies to Text::Wrap::fill, not Text::Wrap::wrap. OTOH, I don't see how it could be better then the current solution, and the current solution is already working, tested, and readable. Warning: Unless otherwise stated, code is untested. Do not use without understanding. Code is posted in the hopes it is useful, but without warranty. All copyrights are relinquished into the public domain unless otherwise stated. I am not an angel. I am capable of error, and err on a fairly regular basis. If I made a mistake, please let me know (such as by replying to this node).	[reply] [d/l]
Re^4: splitting text into lines -- code -> regex by ysth (Canon) on Jul 13, 2004 at 06:21 UTC
Re: splitting text into lines -- code -> regex by hawtin (Prior) on Jul 12, 2004 at 07:39 UTC
You need to be a little more specific. From my reading it seems that the 27 characters never have any spaces in. Is that right? If so then what you want is either split() or the magic variable `$/` If spaces can be validly placed within the codes then I would suggest that you split on spaces and then reassemble your lines until you have enough information: `$/ = ' '; $line = ''; while($chunk = <>) { if((length($line) + length($chunk)) > 27) { # Do the line processing $line = ''; } $line .= $chunk; } # Process the final line` [download] (Code untested of course)	[reply] [d/l]
Re^2: splitting text into lines -- code -> regex by davido (Cardinal) on Jul 12, 2004 at 07:55 UTC
That might present a problem, as some spaces are actual delimiters, and others are going to be literal text. If you treat both the same way (split on plain spaces and reassemble all the "words"), you may end up splitting words that happened to span a 27-char limit. Dave	[reply]
Re^3: splitting text into lines -- code -> regex by hawtin (Prior) on Jul 12, 2004 at 21:43 UTC
That might present a problem, as some spaces are actual delimiters, and others are going to be literal text. If you treat both the same way (split on plain spaces and reassemble all the "words"), you may end up splitting words that happened to span a 27-char limit. The way the problem was stated it seemed to me that you can guarantee that each 'line' ends in a space. If that is correct then you know each 'line' is the concatenation of a number of 'chunks', that is no 'chunk' can belong to two different lines. Is that not correct? That is not, of course, to say that every space is the delimiter of a 'line' (otherwise it would be simple). My code did take into account the fact that spaces have two different meanings. While I did forget to allow for adding one for the extra space a working version is quite close to my original code: use strict; use warnings; $/ = ' '; my $line = ''; print "\|012345678901234567890123456\|\n"; while(my $chunk = <DATA>) { if((length($line) + length($chunk)) > 28) { # Remove the delimiter space and pad out chop($line); $line .= ' 'x(27-length($line)); # Do the line processing print "\|$line\|\n"; $line = ''; } $line .= $chunk; } # Process the final line chop($line); $line .= ' 'x(27-length($line)); print "\|$line\|\n"; __END__ 012345678-A1234567 INCL.EUR 3,31 MWST JULI MONATL. GEB HR T-DSL FLAT 0 +1.07.04-3 [download] Gives `\|012345678901234567890123456\| \|012345678-A1234567 INCL.EUR\| \|3,31 MWST JULI MONATL. GEB \| \|HR T-DSL FLAT 01.07.04-3 \|` [download] If however my assumption about spaces at the end of each line is wrong (for example if there could be words that are longer than 27 characters without a space) then a simple if statement will take care of that, something like: `while(my $chunk = <DATA>) { if(length($chunk) > 27) { # Process holdover line process_line($line) if($line); $line = ''; while($chunk =~ s/^(.{27})//) { process_line($1); } $line = $chunk; } elsif((length($line) + length($chunk)) > 28) {` [download] It is true that this is a more simplistic approach than using a "negative lookahead assertion", but there again I don't know how one of them works :-)	[reply] [d/l] [select]