in reply to Re^2: Replace a Hex Value in Column
in thread Replace a Hex Value in Column

But I am dealing with piped traffic so STDIN will be the input and lines should be written one by one

STDIN and STDOUT are filehandles, like $ifh and $ofh in my code. Here, I'm using select to get STDOUT and ARGV to get Perl's magic command-line argument filehandle, which reads from STDIN when no files are specified on the command line. I've assumed you don't want some of the binary data from here, but note that this still doesn't strip all of it out.

use warnings;
use strict;
use Text::CSV; # also install Text::CSV_XS for speed

my $csv = Text::CSV->new({ binary=>1, auto_diag=>2, escape_char=>'\\',
    eol=>$/, quote_char=>'"', sep_char=>";", always_quote=>1 });
while ( my $row = $csv->getline(*ARGV) ) {
    $row->[5] = pack '(H2)*', split /:/, $row->[5];
    $row->[5] =~ s/\A\x19\x68\x09\x24\x00+|\x00+\z//g;
    $csv->print(select, $row);
}
$csv->eof or $csv->error_diag;

[download]

Replies are listed 'Best First'.
Re^4: Replace a Hex Value in Column
by choroba (Cardinal) on Feb 03, 2018 at 10:19 UTC
    tr might be faster than split: 
    #!/usr/bin/perl
use warnings;
use strict;
use Syntax::Construct qw( /r );

my $s = '19:68:09:24:00:5b:00:7e:53:53:4c:2e:47:6f:6f:67:6c:65:00:00:0
+0:00:00:00';

sub ch {
    my $s = $_[0];
    pack 'H*', $s =~ tr/://dr
}

sub hx {
    my $s = $_[0];
    pack '(H2)*', split /:/, $s
}

use Test::More tests => 1;
is ch($s), hx($s), 'same';

use Benchmark qw{ cmpthese };
cmpthese(-2, {
    ch => sub { ch($s) },
    hx => sub { hx($s) },
});

    [download]
    ($q=q:Sq=~/;[c](.)(.)/;chr(-||-|5+lengthSq)`"S|oS2"`map{chr |+ord
}map{substrSq`S_+|`|}3E|-|`7**2-3:)=~y+S|`+$1,++print+eval$q,q,a,

    [download]
[reply]
[d/l]
[select]
Re^4: Replace a Hex Value in Column
by ktham33n (Novice) on Feb 03, 2018 at 10:54 UTC
    Thanks Guys! tr introduced difference almost 500%, I managed to clean up the binaries, below is the final code: 
    #!/usr/bin/env perl
use warnings;
use strict;
use Text::CSV_XS; # also install Text::CSV_XS for speed

my $csv = Text::CSV_XS->new({ binary=>1, auto_diag=>2, escape_char=>'\
+\',
    eol=>$/, quote_char=>'"', sep_char=>";", always_quote=>1 });
while ( my $row = $csv->getline(*ARGV) ) {
    $row->[5] = pack 'H*',$row->[5]=~ tr/://dr;
    $row->[5] =~ s/\A\x19\x68\x09\x24\x00+|\x00+\z//g;
    $row->[5] =~ s/[^\w:]|(?<!:):(?!:)//g;
    $csv->print(select, $row);
}
$csv->eof or $csv->error_diag;

    [download]
[reply]
[d/l]
Re^4: Replace a Hex Value in Column
by ktham33n (Novice) on Feb 03, 2018 at 09:39 UTC
    Yup that fixed it, those binaries are much less also! excuse my comment before its just when I saw you opening 2 files I just turned away immediately, I just a have strange error on the exist but output is quiet fine : 
    #!/usr/bin/env perl
use warnings;
use strict;
use Text::CSV_XS; # also install Text::CSV_XS for speed

my $csv = Text::CSV_XS->new({ binary=>1, auto_diag=>2, escape_char=>'\
+\',
    eol=>$/, quote_char=>'"', sep_char=>";", always_quote=>1 });
while ( my $row = $csv->getline(*ARGV) ) {
    $row->[5] = pack '(H2)*', split /:/, $row->[5];
    $row->[5] =~ s/\A\x19\x68\x09\x24\x00+|\x00+\z//g;
    $csv->print(select, $row);
}
$csv->eof or $csv->error_diag;

    [download]

    OUTPUT 

    "Feb  2, 2018 18:06:49.993152000 ";"172.16.13.158";"1169";"151.101.112
+.102";"443";"[\0vSSL.Slack"
"Feb  2, 2018 18:06:49.993349000 ";"172.16.16.75";"26504";"31.13.86.8"
+;"443";"[\0wSSL.Facebook"
"Feb  2, 2018 18:06:49.993467000 ";"172.16.13.158";"28802";"151.101.11
+2.102";"443";"[\0vSSL.Slack"
"Feb  2, 2018 18:06:49.996467000 ";"192.167.170.118";"48620";"216.58.2
+10.206";"443";"&#65533;\0|QUIC.YouTube"
"Feb  2, 2018 18:06:49.997690000 ";"172.16.15.155";"33318";"172.217.18
+.46";"443";"[\0&#65533;SSL.PlayStore"
"Feb  2, 2018 18:06:49.998318000 ";"172.16.23.85";"26617";"31.13.86.51
+";"443";"[\0SL.WhatsApp"
"Feb  2, 2018 18:06:49.998889000 ";"172.16.16.75";"26497";"172.217.18.
+42";"443";"[\0&#65533;SSL.GoogleServi"
"Feb  2, 2018 18:06:49.999517000 ";"172.16.14.116";"27917";"216.58.209
+.36";"443";"[\0~SSL.Google"
"Feb  2, 2018 18:06:50.001838000 ";"172.16.14.163";"26396";"172.217.18
+.42";"443";"[\0&#65533;SSL.GoogleServi"
"Feb  2, 2018 18:06:50.002442000 ";"192.167.175.103";"1169";"107.154.5
+0.239";"21120";"Unknown"
Use of uninitialized value in split at ./dpi.pl line 9, <> line 11.
"";;;;;""

    [download]
    ~
[reply]
[d/l]
[select]
      I just a have strange error on the exist but output is quiet fine 
      Use of uninitialized value in split at ./dpi.pl line 9, <> line 11.
"";;;;;""

      [download]

      That's not an error, just a warning, about the fact that $row->[5] is undef. From the output I suspect you've just got an empty line at the end of your input (line 11, as the message says).

[reply]
[d/l]
[select]

In Section Seekers of Perl Wisdom