Seems like a somewhat strange requirement. Presumably the application is dumb as a box of hammers and is quoting any internal *char type. Essentially you need to remember if a given field was quoted or not which is typically information that CSV parsers discard. You really need a specialised split that returns array refs for the fields where the first element is the field data and the second the quote status. Here is the basic idea:

#!/usr/bin/perl -w
use strict;

while( my $data = <DATA> ) {
    chomp($data);
    next unless $data;
    my $fields = get_fields( $data );
    $fields->[rand(3)]->[0] = 'newval';
    my $str = remap_fields( $fields );
    print "$str\n";
}

sub get_fields {
    [ map{ s/^"|"$//g ? [ $_, 1 ] : [ $_, 0 ] } split /(?<=\S),(?=\S)/
+, $_[0] ];
}
sub remap_fields {
    join ',', map{ $_->[1] ? qq!"$_->[0]"! : $_->[0] } @{$_[0]};
}
__DATA__
"hi",3,20.6,"green","32"
16,"alpha",0.00
[download]

The get_fields() function may or may not work for you. The logic it uses not to split on embedded commas is to only split on \S,\S, This is making the invalid assumption that embeded commas probably look, like, this ie have spaces. You may have no embedded commas or the assumption may be true. If it is not you will need a better parser.

It is not really that weird. Some of the application that is type-aware, do generate CSV file like this, especially database applications, adn Text::CSV is ready to handle this, both in and out.

Like this?

sub dequote {
   local $_ = @_ ? $_[0] : $_;
   return $_ unless (s/^"(.*)"$/$1/s);
   s/\\(.)/$1/gs;
   return $_;
}

my @data;
while (<DATA>) {
   chomp;
   my @fields;
   $_ = $_ . ',';
   push(@fields, $1) while (/\G("(?:[^"\\]|\\.)*"|[^,]*)(?:,)/g);
   @fields = map { dequote } @fields;
   push(@data, \@fields);
}

require Data::Dumper;
print(Data::Dumper::Dumper(\@data));

__DATA__
"hi",3,20.6,"green","32"
16,"alpha",0.00
[download]

With Text::CSV, if you want to embed double-quote, represent it by a pair of consecutive double-quotes.

The default behavior for Text::CSV_XS works as you want things to. Example:

use strict;

use Text::CSV_XS;
use IO::File;
my $csv = Text::CSV_XS->new();
my @row;

while (<DATA>)
{
    chomp;
    @row = split(/\|/, $_);
    $csv->print(\*STDOUT, \@row);
    print "\n";
}

__DATA__
Field1|Field Number Two|Field Number Three|Field4
Another One|foo|fubar|foo and fubar
[download]

Field1,"Field Number Two","Field Number Three",Field4
"Another One",foo,fubar,"foo and fubar"
[download]

I've found Text::CSV_XS to be the most accurate at generating true CSV, versus just delimited by commas, which some people seem to think is CSV.

The default behavior for Text::CSV_XS works as you want things to.

Actually what you have shown and what Text::CSV_XS does has no relationship whatsoever to what the OP wanted.

You are correct tachyon. It took me three cups of coffee to see that. My apologies. The issue here is rather custom since none of the CSV packages I've used will tell you if something was quoted coming in -- they hide that. It seems like to do this right you need to know what was quoted coming in and mirror that on the way out. If the rules are based on content (e.g., something that can be regexp matched) then maybe you don't need to know how it was originally.

As far as I can tell, Text::CSV can handle all of this except writing out with quotes the way I need. It either writes everything with quotes or nothing.

You might want to examine your data to make sure that isn't what is happening.

If your fields are not quoted according to their parseability, then it is most likely that they are quoted by field type (as understood by the original app) so that they will be consistent *by field*. In other words if fields 1, 4, and 5 are thought by the original app to contain strings, then fields 1, 4, and 5 will be quoted. Here's a script which will honor the original apps' field definitions - it reads CSV fields (including embedded commas) and writes out the fields quoted as in the original. It uses AnyData, another CSV-capable module. The example uses a DATA section, but the module also supports reading from files.

#!/usr/bin/perl -w
use strict;
use AnyData;
my %is_quoted_field = ( c1=>1, c4=>1, c5=>1 );
my @cols            = qw ( c1 c2 c3 c4 c5 c6 );
my $table = adTie('CSV',[<DATA>],'r',{cols=>[@cols]});
while (my $row = each %$table) {
    my @fields;
    for (@cols) {
        my $field = ($is_quoted_field{$_})
                  ? q{"} . $row->{$_} . q{"}
                  : $row->{$_};
        push @fields,$field;
    }
    print join(',',@fields)."\n";
}
__DATA__
"bye",3,27.6,"green","32",stuff
"x,y",5,28.2,"yellow","33",more stuff
[download]

If the strings can have commas, I don't know if Text::CSV_XS can help you.

As I mentioned, I do have file layouts detailing the datatypes of each field I'm going to encounter. I think the approach I'll take will be to read the file layouts into a data structure, and re-quote the text fields when they're in an array. That way I can still use the well-tested Text::CSV or Text::CSV_XS for parsing and rewriting the CSVs. It'll take more work, but that's probably the most bulletproof and flexible approach.

Again, thanks!

-E