Re^2: Speeds vs functionality

Replies are listed 'Best First'.
Re^3: Speeds vs functionality by Tux (Canon) on Jul 30, 2014 at 12:21 UTC
The cache, as implemented currently, was implemented to achief a boost of (iirc) about 25%. It is needed to reduce the access to the object (the `$self` hash), as those lookups are very very expensive. Unicode whitespace isn't important for this parser, as it is no special "character", unless it being the separator, the quotation or the escape character. Unicode whitespace will just end up being binary. XS is not PP :) Those characters could be int indeed, but that would probably mean that the whole parser (written in 1998 and modified/extended over time) has to be rewritten. It /might/ be worth the effort in the end, but I do not have the time to start that experiment. Never tried fsm (unless the current state-machine already is an FSM). I simplified the parser as I got it when I took over maint. Over time a lot of bugs were fixed and new (required and requested) features where added. update: added remark about FSM Enjoy, Have FUN! H.Merijn	[reply] [d/l]
Re^4: Speeds vs functionality by salva (Canon) on Jul 31, 2014 at 09:11 UTC
Is there any reason stopping you for keeping the parser state as a persistent C struct? Correct me if I am wrong: currently, the state is keep exclusively on the Perl side and the cache is a (ugly) hack to be able to regenerate the C struct faster. Why don't just store the state on the C side and keep it as a pointer inside a IV? That's what most XS libs do and I am sure it would improve the parser speed a lot, and at the same time simplify the code! Are the module users allowed to modify the object hash directly?	[reply]
Re^5: Speeds vs functionality by Tux (Canon) on Jul 31, 2014 at 09:38 UTC
I am keeping it as a cache, because the user is allowed to alter the behavior of the parser between parses. And that is useful. I also agree that the cache-dealing code has by now grown out to a horrid state, and I was already playing with the idea of replacing it with a single PV that holds a memcpy of the csv struct (which might need some extending then). In earlier days, the end user was allowed to alter the hash. I now only allow (reliable) changes through the method calls. That implies that I in theory can move all code to XS. I however am not yet sure if that would simplify the code, though it probably will :) Enjoy, Have FUN! H.Merijn	[reply]
Re^4: Speeds vs functionality by Jim (Curate) on Jul 31, 2014 at 00:09 UTC
I believe Modern Perl should have a ~~core~~ module that can easily parse these simple Unicode CSV records. It should handle them in any character encoding scheme of Unicode: UTF-8, UTF-16, or UTF-32. And it should handle the Unicode byte order mark seamlessly. Why not? 🎥Film🎥🎬🎥Year🎥🎬🎥Awards🎥🎬🎥Nominations🎥🎬🎥Director🎥 🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥 🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥 🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥 🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥 🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥 🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥 🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen 🎥🎥 Ethan Coen🎥 🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥 sep_char 🎬 U+1F3AC CLAPPER BOARD (UTF-8: F0 9F 8E AC) quote_char 🎥 U+1F3A5 MOVIE CAMERA (UTF-8: F0 9F 8E A5) escape_char 🎥 U+1F3A5 MOVIE CAMERA (UTF-8: F0 9F 8E A5) "Film","Year","Awards","Nominations","Director" "12 Years a Slave",2013,3,9,"🎥 Steve McQueen" "Argo",2012,3,7,"🎥 Ben Affleck" "The Artist",2012,5,10,"🎥 Michel Hazanavicius" "The King's Speech",2010,4,12,"🎥 Tom Hooper" "The Hurt Locker",2009,6,9,"🎥 Kathryn Bigelow" "Slumdog Millionaire",2008,8,10,"🎥 Danny Boyle" "No Country for Old Men",2007,4,8,"🎥 Joel Coen 🎥 Ethan Coen" "The Departed",2006,4,5,"🎥 Martin Scorsese" I recognize that the current XS ~~core~~ module for parsing CSV records, Text::CSV_XS (marvelously maintained by Tux), may not be the right module to use as the basis for a new, fully Unicode-capable module. But because Perl's native Unicode capabilities exceed those of most other programming languages, Perl should have a proper FSM-based Unicode CSV parser, even if it's pure Perl and not XS. I long ago accepted that Unicode conformance and comparative slowness go hand in hand 👫. So what? Look what you're trading a few seconds here and there for: the technological foundation of World Peace ☮ and Universal Love 💕. UPDATE: Removed references to core module. I don't care about that. I just want a Unicode-capable Perl CSV module.	[reply]
Re^5: Speeds vs functionality by farang (Chaplain) on Jul 31, 2014 at 23:50 UTC
Text::CSV_PP is able to parse that text, at least in UTF-8. use v5.12; use warnings; use utf8::all; use Text::CSV_PP; my $csv = Text::CSV_PP->new ( { binary => 1 , quote_char => '🎥' , escape_char => '🎥' , sep_char => '🎬' } ) or die "Cannot use CSV_PP: " .Text::CSV_PP->error_diag (); my @rows; my $fh = *DATA; while ( my $row = $csv->getline( $fh ) ) { push @rows, $row; } $csv->eof or $csv->error_diag(); for ( @rows ) { printf("%-25s%s\n", $_->[0], $_->[4]); } __DATA__ 🎥Film🎥🎬🎥Year🎥🎬🎥Awards🎥🎬🎥Nominations🎥🎬🎥Director🎥 🎥12 Years a Slave🎥🎬2013🎬3🎬9🎬🎥🎥🎥 Steve McQueen🎥 🎥Argo🎥🎬2012🎬3🎬7🎬🎥🎥🎥 Ben Affleck🎥 🎥The Artist🎥🎬2012🎬5🎬10🎬🎥🎥🎥 Michel Hazanavicius🎥 🎥The King's Speech🎥🎬2010🎬4🎬12🎬🎥🎥🎥 Tom Hooper🎥 🎥The Hurt Locker🎥🎬2009🎬6🎬9🎬🎥🎥🎥 Kathryn Bigelow🎥 🎥Slumdog Millionaire🎥🎬2008🎬8🎬10🎬🎥🎥🎥 Danny Boyle🎥 🎥No Country for Old Men🎥🎬2007🎬4🎬8🎬🎥🎥🎥 Joel Coen 🎥🎥 Ethan Coen🎥 🎥The Departed🎥🎬2006🎬4🎬5🎬🎥🎥🎥 Martin Scorsese🎥 Output: Film Director 12 Years a Slave 🎥 Steve McQueen Argo 🎥 Ben Affleck The Artist 🎥 Michel Hazanavicius The King's Speech 🎥 Tom Hooper The Hurt Locker 🎥 Kathryn Bigelow Slumdog Millionaire 🎥 Danny Boyle No Country for Old Men 🎥 Joel Coen 🎥 Ethan Coen The Departed 🎥 Martin Scorsese	[reply]
Re^6: Speeds vs functionality by Jim (Curate) on Aug 01, 2014 at 01:38 UTC
Re^6: Speeds vs functionality by Anonymous Monk on Aug 01, 2014 at 02:32 UTC
Re^6: Speeds vs functionality by Jim (Curate) on Aug 01, 2014 at 18:23 UTC
Re^5: Speeds vs functionality (utf8 csv) by tye (Sage) on Jul 31, 2014 at 03:09 UTC
There are already at least a few CSV parsing modules on CPAN that don't just wrap Text::CSV_XS. A pure-Perl CSV parser is likely going to "just work" when given a file handle with the right encoding declared and separator/quote/escape strings properly decoded. Parse::CSV and Text::xSV are the first two I would try. My expectation is that both will handle utf-8 just fine. And if either doesn't, I suspect that fixing that problem won't be difficult. - tye	[reply]
Re^6: Speeds vs functionality by Jim (Curate) on Jul 31, 2014 at 04:17 UTC
Re^7: Speeds vs functionality (utf8 csv) by tye (Sage) on Jul 31, 2014 at 07:33 UTC
Some notes below your chosen depth have not been shown here
Re^7: Speeds vs functionality by Tux (Canon) on Jul 31, 2014 at 06:37 UTC
Re^5: Speeds vs functionality by Anonymous Monk on Jul 31, 2014 at 03:06 UTC
I believe Modern Perl should have a core module ... Why not? ;) Because its not something perl5-porters want to maintain ? Text::CSV::Encoded Remove CGI.pm from Perl Core? http://blogs.perl.org/users/brian_d_foy/2011/03/what-should-be-core-in-perl-516.html http://blogs.perl.org/users/jeff_thalhammer1/2013/03/fat-versus-thin-core.html Why Perl does not support database access through core modules?	[reply]