in reply to problem with csv parsing

This removes the error.

#!/usr/bin/env perl
use strict;
use warnings;

use Text::CSV;

my $fileTable = shift;
my $codepage  = 'utf8';

my $CSV_H = Text::CSV->new (
    {   sep_char         => ";",
        binary           => 1,
        blank_is_undef   => 1,
        empty_is_undef   => 1,
        allow_whitespace => 0,
        quote_char       => undef
    }
);

if (open my $TBL_H, "<:encoding($codepage)", $fileTable) {
    while (my $row = $CSV_H->getline ($TBL_H)) { }
    $CSV_H->eof or $CSV_H->error_diag ();
    close $TBL_H;
}

[download]

If your input is, as you say, not completely valid then you will have to consider the possibility that further problems may arise. Pre-processing the input into a valid form might be a good idea.

PS. Are you sure your input is really utf8?

Replies are listed 'Best First'.
Re^2: problem with csv parsing
by igoryonya (Pilgrim) on May 20, 2020 at 12:42 UTC
    Thank you, it worked partially. I.e., it didn't split all the delimiters to fields, but it's ok. I think, I will feed the parser line by line, using csv handler config, as in my example, and, if an error occurs, feed with the config, suggested by you, recreate the csv from parsing, than reparse again with my original config and so on.
    It was saved ad utf8 from one bookkeeping program. I am sure, that it's utf8, because I've had Russian chars there. They wouldn't display correctly, in other codepage.
[reply]
Re^2: problem with csv parsing
by igoryonya (Pilgrim) on May 21, 2020 at 03:02 UTC
    I added callback after the handler initialisation: 
    $CSV_H->callbacks(error=>\&onerror);

    [download]
    and added a function: 
    sub onerror{
    print '=' x 100, "\n";
    printf "ERROR_INPUT: %s\n", $CSV_H->error_input();
    printf "EOF: %s\n", ($CSV_H->eof)?('EOF'):('ERROR');
}

    [download]
    , but it doesn't seem to work. None of the print statements from the sub have put anything out to the screen. There was no output.
[reply]
[d/l]
[select]

      That is the hard way to get neat errors. Just add auto_diag => 2

      Another approach might be to use csv-check and get the options right before you go for the real thing:

      $ csv-check -v1 test.ssv
Checked test.ssv with csv-check 2.05
using Text::CSV_XS 1.42 with perl 5.30.0 and Unicode 12.1.0
test.ssv record 1 at line 1/104 - 2034 - EIF - Loose unescaped quote
    |147;lakjfh lkjsfh ehjd;134-324-730 31;291;24.04.2020;15 000,00;severo-vostocnoe otdelenie \x{02116} 8645 pao "sberbank rossii";4243972345;347636334;23452347344633423542;severo-vostocnoe otdelenie N8645 pao sberbank g. magadan;896868986;98375423895239529987;;96764128476876487264\n|
    |                                                                                                             ▲                                                                                                                                                                          |
# CSV_XS ERROR: 2034 - EIF - Loose unescaped quote @ rec 0 pos 104 field 2

$ csv-check -v1 test.ssv --allow-loose-quotes
Checked test.ssv with csv-check 2.05
using Text::CSV_XS 1.42 with perl 5.30.0 and Unicode 12.1.0
OK: rows: 16, columns: 2
    sep = <,>, quo = <">, bin = <1>, eol = <"\n">

      Once you get the options so that you are able to parse your (invalid) data, show the used attributes with -L:

      $ csv-check -v1 test.ssv --allow-loose-quotes -L
  allow_loose_escapes   : 0
  allow_loose_quotes    : 1
  allow_unquoted_escape : 0
  allow_whitespace      : 0
  always_quote          : 0
  auto_diag             : 1
  binary                : 1
  blank_is_undef        : 0
  callbacks             : (undef)
  decode_utf8           : 1
  diag_verbose          : 0
  empty_is_undef        : 0
  eol                   :
  escape_char           : "
  escape_null           : 1
  formula               : diag
  keep_meta_info        : 1
  quote                 : (undef)
  quote_binary          : 1
  quote_char            : "
  quote_empty           : 0
  quote_space           : 1
  sep                   : (undef)
  sep_char              :
  strict                : 0
  types                 : (undef)
  undef_str             : (undef)
  verbatim              : 0

      [download]

      Or just show the options that changed the defaults (note that csv-check sets some sane attributes that are not default):

      $ csv-check -v1 test.ssv --allow-loose-quotes -X
  allow_loose_quotes    : 1
  auto_diag             : 1
  binary                : 1
  formula               : diag
  keep_meta_info        : 1

      [download]

      So, your code would now be:

      use Text::CSV_XS; # Text::CSV_XS is much faster that Text::CSV

my $fileTable = shift;

my $CSV_H = Text::CSV_XS->new ({
    sep_char           => ";",
    binary             => 1,
    blank_is_undef     => 1,
    empty_is_undef     => 1,
    allow_whitespace   => 0,
    allow_loose_quotes => 1, # Should work. If not, maybe a bug in Tex
+t::CSV
    auto_diag          => 2, # Added
    });

      [download]

      Enjoy, Have FUN! H.Merijn
[reply]
[d/l]
[select]

      If your error callback isn't printing anything, that means that it isn't being called which means that there are no errors. Surely that's a good thing?

[reply]
        As I've wrote in my original message, there are errors, just the error call back wasn't printing anything, but I've figure it out.
        I think, it was because I needed to configure auto_diag=>1.
        Now it works.
[reply]

In Section Seekers of Perl Wisdom