comment on

Dear Monks, I'm working on a task to load millions of hash records which are generated based on the data in the text files on the system.

To accomplish this I wrote a script which reads every text file from the system , gets data from "Subject" line until the eof() and generates a MD5 hash.

After loading thru some data I found an issue, where some particular text files (having XUZ in the "Subject") have an attachment, as given below.

I'd like to know the best way to remove this attachment headers and the attachment data (from the "Content-Type:" ) from the temporarily read text files.

Below is the format of the actual text file


Message-Id: <200707020704.l6274QG9029301@smtp2.corp.abb.com>
Subject: System Alert from XUZ of sts WARNING
Mime-Version: 1.0

dhcp 0
ip 172.19.22.255
netmask 255.255.255.255
gateway 192.168.1.1
HOSTNAME=ABB
dns.enable on
ab_to_abb[0]
ab_to_local[0]
plex_to_abb[1]
plex_to_local[1]
ab_to_abb[1]
Content-Type: application/x-gzip
Content-Disposition: attachment; filename="sys_logs.gz"
Content-Transfer-Encoding: base64

H4sIAHujiEYCA+1dW2/bOhJ+D5DwIN9aAukqqirrcUukJM2pznIpUjScxZYLApZlhJtZMm
+rS9Ls27znSElx4ouiWXasRMVhW1J5MfhcDgcfhlJO3979I8cR1fkL5QkqR2nJPUnLrnIQ
+vJ7
FhBCiaJYqm5p8EOWzZ2dpsppnIWOnbpjYqel6qqlGJZm8urk1vMDN4amko/JbSArH289lV
+zb
CUmus5SMo7tQkqTd3R34f3B0eEGCyLEDkrjxLdSKwqI+/5KJzyumfng1VnO9tkZSodQ1Yn
+C
cUImUew+H7dGork+UQP7pFPy7fMl+eesLxZUl5I0mk7dsXVnxyHI8C8rx7TIu/lO03d/JX
+lR
0dDK6qDVdmitAj299T3JDoJjP7xJPoMurYPzo8ujg/1jaGIO2fdkahEoSAIsydS+u/N1//
+jSIk6QJSmMzRSsM4RvHJvUvnFDEuHgvfcmH3AEq+KYTBy0u90dsttsum44rlo9M1t90Gr
+1StOU

CREATED_ON=Mon Jul  2 00:04:28 2007
[download]

After removal of the content-type headers and the attachment, the text file data should look as below


Message-Id: <200707020704.l6274QG9029301@smtp2.corp.abb.com>
Subject: System Alert from XUZ of sts WARNING
Mime-Version: 1.0

dhcp 0
ip 172.19.22.255
netmask 255.255.255.255
gateway 192.168.1.1
HOSTNAME=ABB
dns.enable on
ab_to_abb[0]
ab_to_local[0]
plex_to_abb[1]
plex_to_local[1]
ab_to_abb[1]

CREATED_ON=Mon Jul  2 00:04:28 2007
[download]

Below is the script which is used only to read the text files and generate hash. Please let me know how to remove the unwated attachment in the most efficient manner.

#!/usr/bin/perl
use strict;
use warnings;

my @dates = qw(20070202 20070703 20070704);
my $datapath = "/p/data/";
foreach my $date (@dates){
    if(-e "$datapath/$date"){
        opendir(THISDIR, "$datapath/$date") || die("Cannot read dir $d
+atapath/$date");
        my @files_list = grep(/^ABB.*/, readdir(THISDIR));
        closedir THISDIR;
        print OUT "$datapath/$date COUNT:". scalar(@files_list) ."\n";

        foreach my $file(@files_list) {
        
            if (-e "$datapath/$date/$file/HDR_FILE"){
                
                open(DATA, "$datapath/$date/$file/HDR_FILE") ;
                my $content="";
                while(<DATA>) {
                    if( (/Subject/ .. ( eof() ) ) {
                        $content .= $_;
                    }

                }
                
                close DATA;
                

               my $hex = md5_hex($content);
                
               if(!$hex || !$content) {
                print  "Could not generate MD5 : $file \n";
                next;
               }
               
               } else {
                print "No HDR_FILE Found, skipp \n";
               }
    }
}
[download]

Thanks for your time.

In reply to Remove attachment data from the read file by chanakya

Are you posting in the right place? Check out Where do I post X? to know for sure.
Posts may use any of the Perl Monks Approved HTML tags. Currently these include the following:
<code> <a> <b> <big> <blockquote> <br /> <dd> <dl> <dt> <em> <font> <h1> <h2> <h3> <h4> <h5> <h6> <hr /> <i> <li> <nbsp> <ol> <p> <small> <strike> <strong> <sub> <sup> <table> <td> <th> <tr> <tt> <u> <ul>
Snippets of code should be wrapped in <code> tags not <pre> tags. In fact, <pre> tags should generally be avoided. If they must be used, extreme care should be taken to ensure that their contents do not have long lines (<70 chars), in order to prevent horizontal scrolling (and possible janitor intervention).
Want more info? How to link or How to display code and escape characters are good places to start.


Pathologically Eclectic Rubbish Lister
	PerlMonks