Re: This looks like whitespace in my CSV but doesn't seem to be
by Anonymous Monk on Sep 30, 2012 at 08:09 UTC
|
#!/usr/bin/perl --
use strict; use warnings;
use Text::CSV;
use Data::Dump;
open my $io, '<', \q{
"My Company","Gavin","Henry ","ghenry@ghenry.co.uk"," 1.00"
};
my $csv = Text::CSV->new( { binary => 1, eol => $/ } );
while ( my $row = $csv->getline($io) ) {
my @fields = @$row;
next unless $fields[4];
dd \@fields;
$fields[4] =~ s/\s+//g;
print $fields[4], "\n";
dd \@fields;
}
__END__
["My Company", "Gavin", "Henry ", "ghenry\@ghenry.co.uk", " 1.00"]
1.00
["My Company", "Gavin", "Henry ", "ghenry\@ghenry.co.uk", "1.00"]
| [reply] [d/l] |
|
[
"My Company",
"Gavin",
"Henry\xC2\xA0",
"ghenry\@ghenry.co.uk",
"\xC2\xA1.00",
]
which isn't whitespace after all. Thanks for the Data::Dump tip. I've always used Data::Dumper. Swapping for that produces:
$VAR1 = [
'My Company',
'Gavin',
'Henry ',
'ghenry@ghenry.co.uk',
' 1.00'
];
$VAR1 = [
'My Company',
'Gavin',
'Henry ',
'ghenry@ghenry.co.uk',
' 1.00'
];
i.e.
next unless $fields[4];
print Dumper \@fields;
$fields[4] =~ s/\s+//g;
print Dumper \@fields;
Walking the road to enlightenment... I found a penguin and a camel on the way.....
Fancy a yourname@perl.me.uk? Just ask!!!
| [reply] [d/l] [select] |
|
\xC2\xA0 is a Unicode non-breaking space encoded into UTF-8. If you decode your UTF-8 string into a native Perl Unicode string (see Encode), and then add the /u modifier to your regular expression, that ought to enable \s to remove non-ASCII whitespace.
perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
| [reply] [d/l] [select] |
|
|
|
|
I don't work with multi-byte characters or UTF-8 files. But Perl has an amazing ability to deal with these things.
From the open() docs, it could be that all you need to do is open the file in the right encoding mode and that could enable even Perl 5.10.X to work. I don't know for sure, but this idea is worth a try.
open(my $fh, "<:encoding(UTF-8)", "filename")
|| die "can't open UTF-8 encoded filename: $!";
| [reply] [d/l] |
|
|
| [reply] |
Re: This looks like whitespace in my CSV but doesn't seem to be
by tobyink (Canon) on Sep 30, 2012 at 08:18 UTC
|
"This looks like whitespace but is never removed. What am I missing?"
Text::CSV strips whitespace from unquoted fields by default, but your fields are quoted. Check out String::Strip.
(Meh, ignore me. I think I got the wrong end of the stick.)
perl -E'sub Monkey::do{say$_,for@_,do{($monkey=[caller(0)]->[3])=~s{::}{ }and$monkey}}"Monkey say"->Monkey::do'
| [reply] |
Re: This looks like whitespace in my CSV but doesn't seem to be
by Marshall (Canon) on Sep 30, 2012 at 09:29 UTC
|
This one way to delete leading and trailing spaces from CSV input.
#!/usr/bin/perl -w
use strict;
use Text::CSV;
use Data::Dumper; # a core module
use Data::Dump qw(pp);
open my $io, '<',
\q{"My Company","Gavin","Henry ","ghenry@ghenry.co.uk"," 1.00"};
my $csv = Text::CSV->new(); # nothing "fancy" needed here
# defaults are fine
while ( my $row = $csv->getline($io) )
{
my @fields =map {
s/^\s*//; #delete leading spaces
s/\s*$//; #delete trailing spaces
$_; #this is the "return value" from map
#map returns the value of the last
#statement - without this $_; you
#get the zero or non-zero scalar value
#of the last s/// statement
} @$row;
print "Using pp from Data::Dump...\n";
print pp \@fields;
print "\n";
print "Using Dumper from Data::Dumper...\n";
print Dumper \@fields;
print "\n";
print "5th field is $fields[4]\n";
}
__END__
Prints:
Using pp from Data::Dump...
["My Company", "Gavin", "Henry", "ghenry\@ghenry.co.uk", "1.00"]
Using Dumper from Data::Dumper...
$VAR1 = [
'My Company',
'Gavin',
'Henry',
'ghenry@ghenry.co.uk',
'1.00'
];
5th field is 1.00
| [reply] [d/l] |
|
my $csv= text::CSV_XS->new ({ binary => 1, allow_whitespace => 1, auto
+_diag => 1 });
But that would not help in this case for two reasons
- It does not strip whitespace inside quotation, but only surrounding sep_char:
allow_whitespace
When this option is set to true, whitespace (TAB's and SPAC
+E's)
surrounding the separation character is removed when parsin
+g. If
either TAB or SPACE is one of the three major characters
"sep_char", "quote_char", or "escape_char" it will not be
considered whitespace.
- Thee whitespace stripped is only space or TAB's, and not non-breaking Unicode space stuff
Spreadsheet::Read however offers to strip leading and trailing whitespace from every field. I could extend that on request to allow it to strip Unicode whitespace too.
strip
If set, "ReadData ()" will remove trailing- and/or leading-
whitespace from every field.
strip leading strailing
----- ------- ---------
0 n/a n/a
1 strip n/a
2 n/a strip
3 strip strip
Enjoy, Have FUN! H.Merijn
| [reply] [d/l] [select] |
|
| [reply] |
|
| [reply] |
Re: This looks like whitespace in my CSV but doesn't seem to be
by rpnoble419 (Pilgrim) on Sep 30, 2012 at 21:15 UTC
|
Just a point to help you in the future. Does your editor allow you to view your files in Hex? You could easily see that "Whitespace" is not a space and is really a character. This will save you hours of hair pulling. | [reply] |
|
| [reply] |