Mailing List Archive

UTF8 support wreaking havoc with temporary deferrals
Hi,

We are using Exim that has utf8 internationlisation support compiled in. We do not advertise SMTPUTF8 support, as we have smtputf8_advertise_hosts set to an empty list.

However, Exim appears to still accept utf8 messages even if they aren't advertised at EHLO time. I assume this is due to misbehaving senders as these messages appear to all be spam.

When this happens, and our server forwards these messages to a remote smtp recipient, we are often receiving a "utf8 support required but not offered for forwarding" error in the Exim error log. However, instead of this being treated as a permanent 5xx error, it's treated as a 4xx error. This causes the remote host to enter into a retry-period for all other messages, and repeats indefinitely as the offending messages remain in the queue.

I can't find a way to blackhole these messages when they arrive, as the $message_smtputf8 variable doesn't seem to be set on these messages. And I can't find a way to make these errors generate an immediate 5xx fail, and I can't find a way to stop them from being accepted.

I'm not necessarily suggesting this is a bug, but it's strange to me that these delivery failures are treated as temporary. If the remote server doesn't support utf8, why would it an hour from now?

Not sure which of these might best solve our problem here:

- Make the "utf8 support required but not offered for forwarding" a 5xx fatal error

- Make downconversion the default in ALL cases

- Outright rejection of utf8 messages if smtputf8_advertise_hosts does not match the current host

- A new transport option forcing downconvert on smtp transports

If anyone can shed some light on how we can work around this for now, would appreciate it.

Thanks,

Dean Brooks
dean@iglou.com

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: UTF8 support wreaking havoc with temporary deferrals [ In reply to ]
On 2018-07-03, Dean Brooks via Exim-users <exim-users@exim.org> wrote:
> Hi,
>
> We are using Exim that has utf8 internationlisation support compiled in. We do not advertise SMTPUTF8 support, as we have smtputf8_advertise_hosts set to an empty list.
>
> However, Exim appears to still accept utf8 messages even if they
> aren't advertised at EHLO time. I assume this is due to misbehaving
> senders as these messages appear to all be spam.

EXIM will still pass UTF8 content when is ASCII mode. it always has.

what are you seeing that suggests that SMTPUTF8 features are being
exploited while disabled.

--
?

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: UTF8 support wreaking havoc with temporary deferrals [ In reply to ]
On 3 July 2018 17:32:23 CEST, Dean Brooks via Exim-users <exim-users@exim.org> wrote:
>However, Exim appears to still accept utf8 messages even if they aren't
>advertised at EHLO time. I assume this is due to misbehaving senders
>as these messages appear to all be spam.

You should be able to inspect the SMTP mail command in the mail ACL. Presumably it has an smtputf8 modifier. Deny based on that.

Probably you're right that Exim should do that for you; please raise a bug.

Afraid I cannot think about the rest of the effects you describe as I'm away from my development facilities.
--
Jeremy

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: UTF8 support wreaking havoc with temporary deferrals [ In reply to ]
Hi,

Thanks for the reply. The real problem here is the 4xx-style deferral response to a remote server that doesn't support UTF8. I'm surprised nobody else is running into this problem. A single message will halt all reliable deliveries to a remote domain in the default configuration.

I'll issue a bug report on this, thanks.

--
Dean Brooks
dean@iglou.com

On Wed, Jul 04, 2018 at 09:03:09AM +0200, Jeremy Harris via Exim-users wrote:
>
>
> On 3 July 2018 17:32:23 CEST, Dean Brooks via Exim-users <exim-users@exim.org> wrote:
> >However, Exim appears to still accept utf8 messages even if they aren't
> >advertised at EHLO time. I assume this is due to misbehaving senders
> >as these messages appear to all be spam.
>
> You should be able to inspect the SMTP mail command in the mail ACL. Presumably it has an smtputf8 modifier. Deny based on that.
>
> Probably you're right that Exim should do that for you; please raise a bug.
>
> Afraid I cannot think about the rest of the effects you describe as I'm away from my development facilities.
> --
> Jeremy
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: UTF8 support wreaking havoc with temporary deferrals [ In reply to ]
On 2018-07-05, Dean Brooks via Exim-users <exim-users@exim.org> wrote:
> Hi,
>
> Thanks for the reply. The real problem here is the 4xx-style deferral response to a remote server that doesn't support UTF8. I'm surprised nobody else is running into this problem. A single message will halt all reliable deliveries to a remote domain in the default configuration.
>
> I'll issue a bug report on this, thanks.

Are you doing queue runs in random order?

> --
> Dean Brooks
> dean@iglou.com
>
> On Wed, Jul 04, 2018 at 09:03:09AM +0200, Jeremy Harris via Exim-users wrote:
>>
>>
>> On 3 July 2018 17:32:23 CEST, Dean Brooks via Exim-users <exim-users@exim.org> wrote:
>> >However, Exim appears to still accept utf8 messages even if they aren't
>> >advertised at EHLO time. I assume this is due to misbehaving senders
>> >as these messages appear to all be spam.
>>
>> You should be able to inspect the SMTP mail command in the mail ACL. Presumably it has an smtputf8 modifier. Deny based on that.
>>
>> Probably you're right that Exim should do that for you; please raise a bug.
>>
>> Afraid I cannot think about the rest of the effects you describe as I'm away from my development facilities.
>> --
>> Jeremy
>>
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>> --
>> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
>> ## Exim details at http://www.exim.org/
>> ## Please use the Wiki with this list - http://wiki.exim.org/
>>
>


--
?

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: UTF8 support wreaking havoc with temporary deferrals [ In reply to ]
On 07/04/2018 08:03 AM, Jeremy Harris via Exim-users wrote:
> On 3 July 2018 17:32:23 CEST, Dean Brooks via Exim-users <exim-users@exim.org> wrote:
>> However, Exim appears to still accept utf8 messages even if they aren't
>> advertised at EHLO time. I assume this is due to misbehaving senders
>> as these messages appear to all be spam.
>
> You should be able to inspect the SMTP mail command in the mail ACL. Presumably it has an smtputf8 modifier. Deny based on that.
>
> Probably you're right that Exim should do that for you; please raise a bug.

Implemented: 8c34c611e0
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/