I have a csv file that has several columns. I created the csv using perl but need to add code to it to remove duplicate lines and when creating it have it only print once on either source field or destination field me being allow to select one or the other.

So basically if option 1 will be source field only print line if the line hasnt printed. I will never know the ip addresses in this field bc the data will print on the fly as it does right now creating the csv. Currently to create the csv i am not using any modules just comma delimiting my text.

Below is the sample code

my @textfiles = <*.txt *.log>;
my $input_file;
my $input_fh;
my $src;
my $dst;

my $output_file = "simon.csv";
    
    open(my $output_fh, '>', $output_file)
        or die "Failed to open $output_file - $!";    
        print {$output_fh} "uploadfiles,submitter,description,SIP,DIP,
+Date_occurred_detected,Time_occurred_detected,Report_Severity,Inciden
+t_Type_Details\n";
    close $output_fh;

foreach my $textfile (@textfiles) {
    if ($textfile =~ /(\d+.\d+.\d+.\d+)/) {
    
    my $ipaddy = $textfile =~ /(\d+.\d+.\d+.\d+)/;
    
    print "Processing $textfile\n";
        
        open(my $input_fh, '<', $textfile)
            or die "Failed to open $textfile: $!";
            
        open($output_fh, '>>', $output_file)
            or die "Failed to open $output_file - $!";
        
        while (my $line = <$input_fh>) {
        
        if ($line =~ /\d{4}-\d+-\d+\s\d{2}:\d{2}:\d{2}\s\d+\s\d+.\d+.\
+d+.\d+/) {
            $src = $line =~ /\d{4}-\d+-\d+\s\d{2}:\d{2}:\d{2}\s\d+\s(\
+d+.\d+.\d+.\d+)/;
            print {$output_fh} "$1.zip,$newcontent,Malicious activity 
+found when mining proxylog data,$1,";
            $dst = $line =~ /SG-HTTP-Service (\d+.\d+.\d+.\d+)/g;
            print {$output_fh} "$1,$now_Month\/$now_Day\/$now_Year,$no
+w_Hour:$now_Min $am_pm2,3,24\n";
            }
            
        }
    }
}

[download]

In reply to correct angle on tackling a problem by diamondsandperls

Title:
Use:  <p> text here (a paragraph) </p>
and:  <code> code here </code>
to format your post, it's "PerlMonks-approved HTML":


  • Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!
  • Titles consisting of a single word are discouraged, and in most cases are disallowed outright.
  • Read Where should I post X? if you're not absolutely sure you're posting in the right place.
  • Please read these before you post! —
  • Posts may use any of the Perl Monks Approved HTML tags:
    a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr
  • You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)
            For:     Use:
    & &amp;
    < &lt;
    > &gt;
    [ &#91;
    ] &#93;
  • Link using PerlMonks shortcuts! What shortcuts can I use for linking?
  • See Writeup Formatting Tips and other pages linked from there for more info.