Remove attachment data from the read file

chanakya has asked for the wisdom of the Perl Monks concerning the following question:

Dear Monks, I'm working on a task to load millions of hash records which are generated based on the data in the text files on the system.

To accomplish this I wrote a script which reads every text file from the system , gets data from "Subject" line until the eof() and generates a MD5 hash.

After loading thru some data I found an issue, where some particular text files (having XUZ in the "Subject") have an attachment, as given below.

I'd like to know the best way to remove this attachment headers and the attachment data (from the "Content-Type:" ) from the temporarily read text files.

Below is the format of the actual text file


Message-Id: <200707020704.l6274QG9029301@smtp2.corp.abb.com>
Subject: System Alert from XUZ of sts WARNING
Mime-Version: 1.0

dhcp 0
ip 172.19.22.255
netmask 255.255.255.255
gateway 192.168.1.1
HOSTNAME=ABB
dns.enable on
ab_to_abb[0]
ab_to_local[0]
plex_to_abb[1]
plex_to_local[1]
ab_to_abb[1]
Content-Type: application/x-gzip
Content-Disposition: attachment; filename="sys_logs.gz"
Content-Transfer-Encoding: base64

H4sIAHujiEYCA+1dW2/bOhJ+D5DwIN9aAukqqirrcUukJM2pznIpUjScxZYLApZlhJtZMm
+rS9Ls27znSElx4ouiWXasRMVhW1J5MfhcDgcfhlJO3979I8cR1fkL5QkqR2nJPUnLrnIQ
+vJ7
FhBCiaJYqm5p8EOWzZ2dpsppnIWOnbpjYqel6qqlGJZm8urk1vMDN4amko/JbSArH289lV
+zb
CUmus5SMo7tQkqTd3R34f3B0eEGCyLEDkrjxLdSKwqI+/5KJzyumfng1VnO9tkZSodQ1Yn
+C
cUImUew+H7dGork+UQP7pFPy7fMl+eesLxZUl5I0mk7dsXVnxyHI8C8rx7TIu/lO03d/JX
+lR
0dDK6qDVdmitAj299T3JDoJjP7xJPoMurYPzo8ujg/1jaGIO2fdkahEoSAIsydS+u/N1//
+jSIk6QJSmMzRSsM4RvHJvUvnFDEuHgvfcmH3AEq+KYTBy0u90dsttsum44rlo9M1t90Gr
+1StOU

CREATED_ON=Mon Jul  2 00:04:28 2007
[download]

After removal of the content-type headers and the attachment, the text file data should look as below


Message-Id: <200707020704.l6274QG9029301@smtp2.corp.abb.com>
Subject: System Alert from XUZ of sts WARNING
Mime-Version: 1.0

dhcp 0
ip 172.19.22.255
netmask 255.255.255.255
gateway 192.168.1.1
HOSTNAME=ABB
dns.enable on
ab_to_abb[0]
ab_to_local[0]
plex_to_abb[1]
plex_to_local[1]
ab_to_abb[1]

CREATED_ON=Mon Jul  2 00:04:28 2007
[download]

Below is the script which is used only to read the text files and generate hash. Please let me know how to remove the unwated attachment in the most efficient manner.

#!/usr/bin/perl
use strict;
use warnings;

my @dates = qw(20070202 20070703 20070704);
my $datapath = "/p/data/";
foreach my $date (@dates){
    if(-e "$datapath/$date"){
        opendir(THISDIR, "$datapath/$date") || die("Cannot read dir $d
+atapath/$date");
        my @files_list = grep(/^ABB.*/, readdir(THISDIR));
        closedir THISDIR;
        print OUT "$datapath/$date COUNT:". scalar(@files_list) ."\n";

        foreach my $file(@files_list) {
        
            if (-e "$datapath/$date/$file/HDR_FILE"){
                
                open(DATA, "$datapath/$date/$file/HDR_FILE") ;
                my $content="";
                while(<DATA>) {
                    if( (/Subject/ .. ( eof() ) ) {
                        $content .= $_;
                    }

                }
                
                close DATA;
                

               my $hex = md5_hex($content);
                
               if(!$hex || !$content) {
                print  "Could not generate MD5 : $file \n";
                next;
               }
               
               } else {
                print "No HDR_FILE Found, skipp \n";
               }
    }
}
[download]

Thanks for your time.

Back to Seekers of Perl Wisdom