comment on

Dear Monks, I'm working on a task to load millions of hash records which are generated based on the data in the text files on the system.

To accomplish this I wrote a script which reads every text file from the system , gets data from "Subject" line until the eof() and generates a MD5 hash.

After loading thru some data I found an issue, where some particular text files (having XUZ in the "Subject") have an attachment, as given below.

I'd like to know the best way to remove this attachment headers and the attachment data (from the "Content-Type:" ) from the temporarily read text files.

Below is the format of the actual text file


Message-Id: <200707020704.l6274QG9029301@smtp2.corp.abb.com>
Subject: System Alert from XUZ of sts WARNING
Mime-Version: 1.0

dhcp 0
ip 172.19.22.255
netmask 255.255.255.255
gateway 192.168.1.1
HOSTNAME=ABB
dns.enable on
ab_to_abb[0]
ab_to_local[0]
plex_to_abb[1]
plex_to_local[1]
ab_to_abb[1]
Content-Type: application/x-gzip
Content-Disposition: attachment; filename="sys_logs.gz"
Content-Transfer-Encoding: base64

H4sIAHujiEYCA+1dW2/bOhJ+D5DwIN9aAukqqirrcUukJM2pznIpUjScxZYLApZlhJtZMm
+rS9Ls27znSElx4ouiWXasRMVhW1J5MfhcDgcfhlJO3979I8cR1fkL5QkqR2nJPUnLrnIQ
+vJ7
FhBCiaJYqm5p8EOWzZ2dpsppnIWOnbpjYqel6qqlGJZm8urk1vMDN4amko/JbSArH289lV
+zb
CUmus5SMo7tQkqTd3R34f3B0eEGCyLEDkrjxLdSKwqI+/5KJzyumfng1VnO9tkZSodQ1Yn
+C
cUImUew+H7dGork+UQP7pFPy7fMl+eesLxZUl5I0mk7dsXVnxyHI8C8rx7TIu/lO03d/JX
+lR
0dDK6qDVdmitAj299T3JDoJjP7xJPoMurYPzo8ujg/1jaGIO2fdkahEoSAIsydS+u/N1//
+jSIk6QJSmMzRSsM4RvHJvUvnFDEuHgvfcmH3AEq+KYTBy0u90dsttsum44rlo9M1t90Gr
+1StOU

CREATED_ON=Mon Jul  2 00:04:28 2007
[download]

After removal of the content-type headers and the attachment, the text file data should look as below


Message-Id: <200707020704.l6274QG9029301@smtp2.corp.abb.com>
Subject: System Alert from XUZ of sts WARNING
Mime-Version: 1.0

dhcp 0
ip 172.19.22.255
netmask 255.255.255.255
gateway 192.168.1.1
HOSTNAME=ABB
dns.enable on
ab_to_abb[0]
ab_to_local[0]
plex_to_abb[1]
plex_to_local[1]
ab_to_abb[1]

CREATED_ON=Mon Jul  2 00:04:28 2007
[download]

Below is the script which is used only to read the text files and generate hash. Please let me know how to remove the unwated attachment in the most efficient manner.

#!/usr/bin/perl
use strict;
use warnings;

my @dates = qw(20070202 20070703 20070704);
my $datapath = "/p/data/";
foreach my $date (@dates){
    if(-e "$datapath/$date"){
        opendir(THISDIR, "$datapath/$date") || die("Cannot read dir $d
+atapath/$date");
        my @files_list = grep(/^ABB.*/, readdir(THISDIR));
        closedir THISDIR;
        print OUT "$datapath/$date COUNT:". scalar(@files_list) ."\n";

        foreach my $file(@files_list) {
        
            if (-e "$datapath/$date/$file/HDR_FILE"){
                
                open(DATA, "$datapath/$date/$file/HDR_FILE") ;
                my $content="";
                while(<DATA>) {
                    if( (/Subject/ .. ( eof() ) ) {
                        $content .= $_;
                    }

                }
                
                close DATA;
                

               my $hex = md5_hex($content);
                
               if(!$hex || !$content) {
                print  "Could not generate MD5 : $file \n";
                next;
               }
               
               } else {
                print "No HDR_FILE Found, skipp \n";
               }
    }
}
[download]

Thanks for your time.

In reply to Remove attachment data from the read file by chanakya

Posts are HTML formatted. Put <p> </p> tags around your paragraphs. Put <code> </code> tags around your code and data!

Titles consisting of a single word are discouraged, and in most cases are disallowed outright.

Read Where should I post X? if you're not absolutely sure you're posting in the right place.

Please read these before you post! —

Posts may use any of the Perl Monks Approved HTML tags:

a, abbr, b, big, blockquote, br, caption, center, col, colgroup, dd, del, details, div, dl, dt, em, font, h1, h2, h3, h4, h5, h6, hr, i, ins, li, ol, p, pre, readmore, small, span, spoiler, strike, strong, sub, summary, sup, table, tbody, td, tfoot, th, thead, tr, tt, u, ul, wbr

You may need to use entities for some characters, as follows. (Exception: Within code tags, you can put the characters literally.)

	For:		Use:
	&		`&`
	<		`<`
	>		`>`
	[		`[`
	]		`]`

Link using PerlMonks shortcuts! What shortcuts can I use for linking?

See Writeup Formatting Tips and other pages linked from there for more info.