Re^4: Suggestions to make this code more Perlish

Hi Ken,

The best place to read up about Perl 6 regexes is the specification itself.

You mused:

While I suspect this has something to do with '\0' terminated strings in C, I don't fully understand what's happening.

No, it's not anything to do with C string terminators.

The problem with your previous version was that you were matching an optional comma at the end of each field and then replacing it with a definite "\037" every time. So, for the last field in each record (which, of course, isn't followed by a comma), your were nevertheless appending an unwanted "\037".

The global substitution would then loop one last time, matching a final zero-character field (because of the (?<a>[^,]*) alternative, which can match nothing). The substitution on that empty field then causes a second unnecessary "\037" to be appended.

You could fix that by rewriting your original version something like this:

    open my $csv_fh, '<', 'input.csv';
    open my $tff_fh, '>', 'output.tff';

    my $field = qr{ " (?<field> [^"]* ) " | (?<field> [^,"]* ) }x;

    while (my $line = <$csv_fh>) {
        $line =~ s{   $field     (?<comma> ,?)        }
                  { $+{field} . ($+{comma} && chr 31) }gxe;

        $line =~ s{\n}{chr 30}xe;

        print {$tff_fh} $line;
    }
[download]

This version still matches the optional comma each time, but now only appends a "\037" if there actually was a comma. Which means there are no extras to remove, once the line is complete.

Note that I also removed the chomp and replaced it with an explicit substitution of the trailing newline. I felt that this highlights the transformation more clearly than did your clever (but subtle and "at-a-distance") use of $\.

Damian

Comment on Re^4: Suggestions to make this code more Perlish Select or Download Code

Replies are listed 'Best First'.
Re^5: Suggestions to make this code more Perlish by kcott (Archbishop) on Mar 31, 2014 at 06:55 UTC
Thanks for the documentation link. That certainly has a lot more text than I found where I was previously looking: and no "TODO"s in sight. I have a bit of reading ahead of me. Yes, I was aware of why I was getting two `\037` characters at the end (in the first solution). What I haven't figured out yet is why I was getting zero `\037` characters at the end (when I changed '`,?`' to '`(?:,\|\000)`' — in the second solution). Thanks also for the additional feedback: much appreciated. -- Ken	[reply] [d/l] [select]
Re^6: Suggestions to make this code more Perlish by TheDamian (Vicar) on Mar 31, 2014 at 07:40 UTC
What I haven't figured out yet is why I was getting zero* `\037` characters at the end (when I changed `',?'` to `'(?:,\|\000)'` — in the second solution).* My apologies for misinterpreting your implied question. The reason your second solution is producing zero trailing `"\037"` characters is because `(?:,\|\000)` can never match nothing. It either matches a trailing comma, or a trailing null character. So on the very last field (which has neither a trailing comma nor a trailing null-byte), your field pattern wasn't matching at all, so you were not rewriting the last field at all, hence no extra `"\037"` was added after it. And, because that final field failed to match, the global matching sequence was terminated at that point, so the regex didn't do that one extra "match an empty field at the end" iteration, which was previously adding the second `"\037"`. Technically, the use of `'(?:,\|\000)'` introduced a bug, as it would then treat any embedded null as a field separator. Granted, it is quite unlikely to find an embedded null in a CSV file, but not impossible. If you wanted to keep using this approach, you could avoid that nasty edge-case by replacing the `(?:,\|\000)` subpattern with a simple comma: `my $re = qr{ (?: "(?<a>[^"])" \| (?<a>[^,]) )``,` `}x;` Damian	[reply] [d/l] [select]
Re^7: Suggestions to make this code more Perlish by kcott (Archbishop) on Mar 31, 2014 at 07:56 UTC
++ Thankyou very much. Once explained, it now seems obvious. One of those "need a fresh set of eyes" situations. My focus had been on why '`$+{a}`' was being replaced instead of '`$+{a}\037`'. Of course, no replacement is taking place at all. Doh! -- Ken	[reply] [d/l] [select]


Problems? Is your data what you think it is?
	PerlMonks