Beefy Boxes and Bandwidth Generously Provided by pair Networks
Just another Perl shrine
 
PerlMonks  

Double Dot Stuffing

by skazat (Chaplain)
on Oct 04, 2020 at 18:17 UTC ( #11122537=perlquestion: print w/replies, xml ) Need Help??

skazat has asked for the wisdom of the Perl Monks concerning the following question:

Hello Monks,

My time in the forest alone writing poetry about nature has been long, but a curious question has crossed my path, and I find myself again in this bustling city.

A user of mine is sending emails that are received with broken URLs. Dots are removed. I suspect a broken sendmail implementation, that's not doing its job dot stuffing, when it finds a dot as the first character of a line, for example:

#!/usr/bin/perl use MIME::Entity; my $str = 'filler text to get the first char of line 2 to be a dot____ +__ http://google.com'; print MIME::Entity->build( Data => $str, Type => 'text/plain', Encoding => 'quoted-printable', )->as_string;
Returns,
Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) filler text to get the first char of line 2 to be a dot______ http://g +oogle= .com=

To me, this seems correct.

One option for me as the app owner is to do this dot stuffing myself, which I am very much against, as I feel it's a hack to a broken system upstream. And I'll probably do it wrong - or at least do it badly. I'll have users now with broken links - but only because there's now too many dots!

Another option is to encode this myself so, "." turns into, "=2E". But, I don't see any way to do this through the tool chain. Looking at the docs for MIME::QuotedPrint, there is no option to encode dots, nor are there any similar options in MIME::Entity, nor is the issue brought up in Net::SMTP. This makes me think that all the authors of all these modules in the tool chain understand this problem better than I, and there was a concerted effort to not do this, "newline starts with a dot encoding".

The only thing similar to this that I have found is that lines that only have a single dot are encoded, stopping the SMTP server from truncating a message, like so:

perl -MMIME::Entity -e 'print MIME::Entity->build(Encoding => "quoted- +printable", Data => "some text\n.\nmore text")->as_string;'

prints,

Content-Type: text/plain Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) some text =2E more text=

So, am I missing something obvious? are there solutions that anyone else has come to, to appease broken SMTP clients without needing to post filtering output from MIME::Entity?

-skazat

Replies are listed 'Best First'.
Re: Double Dot Stuffing
by BillKSmith (Monsignor) on Oct 04, 2020 at 20:47 UTC
    I am surprised that your test case works at all. The EXAMPLES section of MIME::Entity specifies that the data field is a reference to an array of strings (lines). Your have a string. I suspect that your problem has nothing to do with 'dot stuffing'. The 'Output' section of the same document specifies that 'stringify' (as_string) returns the entity exactly as print would print it. (I take this to mean that any 'stuffing' is removed.) In your case, a long line is split into two. The URL is split over the two lines. It appears that the module uses the '=' (at the end of the first line) to indicate this split. Are you sure that you are extracting the URL correctly in this case?
    Bill

      Interesting observation, but it appears that passing a string returns the same output as passing an array ref. Passing a string as the value of the paramater is even used in the source of the MIME::Entity module itself for its own internal methods (see, add_sig() ).

      The docs actually state the value for the Data param should be,

      Single-part entities only. Optional. An alternative to Path (q.v.): the actual data, either as a scalar or an array reference (whose elements are joined together to make the actual scalar).

      So both are right. I'm very pleased with how MIME::Entity is encoding the message - one line that I feed it splitting into two lines when encoded is what I expect. Decoding things back works just as well. (I'm not reporting a bug in MIME::Entity.)

       
      #!/usr/bin/perl 
      
      use MIME::Entity; 
      
      my $str = 'filler text to get the first char of line 2 to be a dot______ http://google.com'; 
      
      MIME::Entity->build(
      	Data     => $str, 
      	Type     => 'text/plain',
      	Encoding => 'quoted-printable',
      )->bodyhandle->print; 
      
      Prints back,
      filler text to get the first char of line 2 to be a dot______ http://google.com
      

      The broken URL isn't happening in my app, but upstream.

      The problem is in the SMTP client not double-dotting a line that starts with a dot. Lines that start with a dot could happen because of my example (wrapping a URL onto multiple lines during encode). I'm asking what it is I should do (if anything) to deal with a broken SMTP client? I'm leaning on, "nothing", but is this instead a universal problem, with a known solution? Googling this problem specifically for Perl examples leads to no results, although other languages/libraries say to encode the dot. If I were to encode the dot, what's the least hackiest way to do so? I can make a one-line regex to handle this, but then I fear I'll then have two problems.

      -skazat
        It appears to me that a long line in the Entity object is just that - a long line. Only the display is word- wrapped. There is no issue with a line starting with a dot. If some 'upstream' software (or person) did the analogous word-wrap before the build, the object would indeed have two lines one of which might start with a period. In that case, your URL would contain a newline character right before the dot. If this does not describe the problem, please post code that builds two similar objects, one that is correct and one that is not.
        Bill

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11122537]
Front-paged by Corion
help
Chatterbox?
and the web crawler heard nothing...

How do I use this? | Other CB clients
Other Users?
Others rifling through the Monastery: (5)
As of 2022-01-19 14:42 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?
    In 2022, my preferred method to securely store passwords is:












    Results (55 votes). Check out past polls.

    Notices?