Mailing List Archive

Bug in email.generator.BytesGenerator() [was: Why does SMTP.send_message() do from mangling?]
On 2021-09-27, Grant Edwards <grant.b.edwards@gmail.com> wrote:
> Why does SMTP.send_message(msg) do from mangling even though msg's
> policy has mangle_from_ set to False? The msg policy is
> email.policy.SMTP which has mangle_from_ disabled.
>
> One might expect that SMTP.send_message(msg) would use either msg's
> policy or email.policy.SMTP to send the message, but it does neither.

I've been looking at the smtplib.py sources, and the problem appears
to be in this section of send_message():


912 def send_message(self, msg, from_addr=None, to_addrs=None,
913 mail_options=(), rcpt_options=()):
914 """Converts message to a bytestring and passes it to sendmail.
...
963 # Make a local copy so we can delete the bcc headers.
964 msg_copy = copy.copy(msg)
...
977 with io.BytesIO() as bytesmsg:
978 if international:
979 g = email.generator.BytesGenerator(
980 bytesmsg, policy=msg.policy.clone(utf8=True))
981 mail_options = (*mail_options, 'SMTPUTF8', 'BODY=8BITMIME')
982 else:
983 g = email.generator.BytesGenerator(bytesmsg)
984 g.flatten(msg_copy, linesep='\r\n')
985 flatmsg = bytesmsg.getvalue()

If 'international' is Frue, then the BytesGenerator uses msg.policy
with utf8 added, and I don't get the bogus from mangling: the
generator only does from mangling if the message policy has it
enabled.

If 'international' is False, then the generator always does from
mangling regardless of the message's policy.

According to
https://docs.python.org/3/library/email.generator.html#email.generator.BytesGenerator
the default from mangling behavior is _supposed_ to obey the message
policy if (as seen at 983) no policy or mangle_from_ value was
provided to the call to BytesGenerator(). In my tests, it doesn't
actually seem to work that way. AFAICT, the default behavior when no
policy or mangle_from_ value is passed to BytesGenerator() is to
enable from mangling regardless of the message's policy. I belive that
is a bug.

This can be worked around by changing

983 g = email.generator.BytesGenerator(bytesmsg)
to

983 g = email.generator.BytesGenerator(bytesmsg, policy=msg.policy)


Or BytesGenerator() could be fixed...

--
Grant
--
https://mail.python.org/mailman/listinfo/python-list
Re: Bug in email.generator.BytesGenerator() [was: Why does SMTP.send_message() do from mangling?] [ In reply to ]
On 2021-09-27, Grant Edwards <grant.b.edwards@gmail.com> wrote:

> According to
> https://docs.python.org/3/library/email.generator.html#email.generator.BytesGenerator
> the default from mangling behavior is _supposed_ to obey the message
> policy if no policy or mangle_from_ value was
> provided to the call to BytesGenerator().

Nope, I think both the author of smtplib.py and I were misled by the
documentation for email.generator.BytesGenerator. After rereading it,
it does say in one place that if no mangle_from_, value is passed to
BytesGenerator(), it is supposed to default to the mangle_from_
setting in the policy **passed to BytesGenerator()**:

If optional mangle_from_ is True, put a > character in front of
any line in the body that starts with the exact string "From ",
that is From followed by a space at the beginning of a
line. mangle_from_ defaults to the value of the mangle_from_
setting of the policy.

Where "the policy" refers to the one passed to BytesGenerator().

However, later on it also says

If policy is None (the default), use the policy associated with
the Message or EmailMessage object passed to flatten to control
the message generation.

That's misleading and only partially true. If you don't pass a policy
to BytesGenerator(), only _some_ of the settings from the message's
policy will be used. Some policy settings (e.g. mangle_from_) are
obeyed when passed to BytesGenerator(), but ignored in the message's
policy even if there was no policy passed to BytesGenerator(). I
think that last sentence above needs to be changed, and smtplib.py
needs to be fixed as shown in my previous post.

--
Grant
--
https://mail.python.org/mailman/listinfo/python-list