Mailing List Archive

cut subjects that are too long + delete emojii from subject lines...
How can I reliably cut subjects that are too long, without breaking
for example quoted-printable encodings and such?

Ergo, if the subject line exceeds X characters (where X characters are
a limit I as system administrator decides) it should just truncate the
decoded subject line, reencode it, and then send the mail along.

The reason I want to do it, is because some IMAP clients do crash when
it tries to export/backup email with too long subject lines as it
tries to write the subject as a filename, and the operating system
returns a error code the IMAP client don't understand = crash.

Also I want in the same way, filter away all and any emojii (for
example the truck in Ebay's "Order is now being delivered" emails)
from subject lines because emojii also causes these IMAP clients im
talking about to crash (because obviously emojii can't be used as
filenames), and emoji in email subjects are so useless so im
speakless...

Any ideas?

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On Wed, 28 Nov 2018, Sebastian Nielsen via Exim-users wrote:

> How can I reliably cut subjects that are too long, without breaking
> for example quoted-printable encodings and such?
>
> Ergo, if the subject line exceeds X characters (where X characters are
> a limit I as system administrator decides) it should just truncate the
> decoded subject line, reencode it, and then send the mail along.
>
> The reason I want to do it, is because some IMAP clients do crash when
> it tries to export/backup email with too long subject lines as it
> tries to write the subject as a filename, and the operating system
> returns a error code the IMAP client don't understand = crash.
>
> Also I want in the same way, filter away all and any emojii (for
> example the truck in Ebay's "Order is now being delivered" emails)
> from subject lines because emojii also causes these IMAP clients im
> talking about to crash (because obviously emojii can't be used as
> filenames), and emoji in email subjects are so useless so im
> speakless...
>
> Any ideas?

Which format is exim using to save messages: unix mailbox, MBX, maildir,
mailstore ?

I'm surprised thst you are using the subject as the filename; you
certainly don't have to do that; the appendfile driver for the
local_delivery transport allows you to set the filename exim uses; you
have presumably explicitly set this to include the subject line, so
can change it to something else ?

Which IMAP *server* is causing these problems ?

--
Andrew C. Aitchison Cambridge, UK
andrew@aitchison.me.uk


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 28/11/2018 06:59, Sebastian Nielsen via Exim-users wrote:
> How can I reliably cut subjects that are too long, without breaking
> for example quoted-printable encodings and such?

Short answer: you can't, it's too hard.

You'd have to write something that understood all possible encodings.
And you'd be breaking dkim signing.


> Also I want in the same way, filter away all and any emojii (for
> example the truck in Ebay's "Order is now being delivered" emails)
> from subject lines because emojii also causes these IMAP clients im
> talking about to crash

Similar problem, though more tractable as these are almost certainly
UTF-8, so you only have one suite of encoding rules to deal with.
Indeed, there might be enough support in the RE library that ${sg }
sits on to spot UTF-8 characters as single items.

Or, if they're always in the same place, use surrounding info to
know when they're there and just edit those bytes positions out.
--
Cheers,
Jeremy


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
Hi, Sebastian -

What you suggest doesn't sound to be a good idea to me, unless you're
planning to do it *only* at the point of final delivery into the
recipient's mailbox. (You saying you'll make the change then "send the
message along" suggests you're not thinking of doing this though, but
applying it to all messages as they pass through.)

I say this because of the message has been digitally signed using DKIM
altering the Subject line will invalidate the signature and possibly cause
problems at any downstream mail servers the message is transmitted to:
possibly causing the message to be marked as spam or rejected.

If the problem lies in certain IMAP clients your users use wouldn't it be
better to report the bug in them and get them fixed? It sounds like they're
broken!

Cheers,
Mike B-)

On Wed, 28 Nov 2018 at 07:04, Sebastian Nielsen via Exim-users <
exim-users@exim.org> wrote:

> How can I reliably cut subjects that are too long, without breaking
> for example quoted-printable encodings and such?
>
> Ergo, if the subject line exceeds X characters (where X characters are
> a limit I as system administrator decides) it should just truncate the
> decoded subject line, reencode it, and then send the mail along.
>
> The reason I want to do it, is because some IMAP clients do crash when
> it tries to export/backup email with too long subject lines as it
> tries to write the subject as a filename, and the operating system
> returns a error code the IMAP client don't understand = crash.
>
> Also I want in the same way, filter away all and any emojii (for
> example the truck in Ebay's "Order is now being delivered" emails)
> from subject lines because emojii also causes these IMAP clients im
> talking about to crash (because obviously emojii can't be used as
> filenames), and emoji in email subjects are so useless so im
> speakless...
>
> Any ideas?
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/
>


--
Systems Administrator & Change Manager
IT Services, University of York, Heslington, York YO10 5DD, UK
Tel: +44-(0)1904-323811

Web: www.york.ac.uk/it-services
Disclaimer: www.york.ac.uk/docs/disclaimer/email.htm
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
Hi.

On November 28, 2018 9:59:22 AM GMT+03:00, Sebastian Nielsen via Exim-users <exim-users@exim.org> wrote:
>How can I reliably cut subjects that are too long, without breaking
>for example quoted-printable encodings and such?

Decoded subject is in $h_Subject: , to reencode it you may use ${rfc2047}. We use the following for altering spam subject:

headers_remove = ${if bool{${extract {spam-tag} {$address_data}}} {Subject}}
headers_add = ${if bool{${extract {spam-tag} {$address_data}}} {Subject: ${rfc2047:*****SPAM***** $h_Subject:}}}

It certainly breaks dkim, but some people still wants this..

Also, resulting encoding won't be the same as original, but it still will be correct.

>Ergo, if the subject line exceeds X characters (where X characters are
>a limit I as system administrator decides) it should just truncate the
>decoded subject line, reencode it, and then send the mail along.
>
>The reason I want to do it, is because some IMAP clients do crash when
>it tries to export/backup email with too long subject lines as it
>tries to write the subject as a filename, and the operating system
>returns a error code the IMAP client don't understand = crash.
>
>Also I want in the same way, filter away all and any emojii (for
>example the truck in Ebay's "Order is now being delivered" emails)
>from subject lines because emojii also causes these IMAP clients im
>talking about to crash (because obviously emojii can't be used as
>filenames), and emoji in email subjects are so useless so im
>speakless...
>
>Any ideas?

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
The problem lies in the CLIENT, not SERVER.
The server works fine, and the client works fine, receiving mail normally.

The problem is when the user, in the IMAP client, selects to "backup"
or "export" the mails to harddrive.
This will cause *.eml or *.msg files to be written to a user-specified
folder, for example a USB thumb drive on computer, local storage in
the Android phone, or whatever, depending on how that IMAP client
works.

The problem is that these IMAP clients, then directly write the
subject as a file name, and of course some characters are not allowed
as file name, and some subjects are too long, causing the operating
system to reject the filename.
This in turn is unhandled in the IMAP client, causing it to suddenly
crash during copying.

Thats why I have to fix it on server end, so the user's IMAP client is
never fed a email that would fail backup later, if the user wants to
backup his own email himself.
Theres no problem at server end, no problem in exim, no problem in
dovecot. The problem is the IMAP client at user end, failing to handle
specific email subjects.

Because its a hell to track down if the user has like 500 emails in
the inbox and client crash somewhere like after 231th email being
copied, and then user have to find the culprit email in their inbox
and then delete it, for the user to succeed copying without crashing
the client.
Den ons 28 nov. 2018 kl 10:44 skrev Andrew C Aitchison <andrew@aitchison.me.uk>:
>
> On Wed, 28 Nov 2018, Sebastian Nielsen via Exim-users wrote:
>
> > How can I reliably cut subjects that are too long, without breaking
> > for example quoted-printable encodings and such?
> >
> > Ergo, if the subject line exceeds X characters (where X characters are
> > a limit I as system administrator decides) it should just truncate the
> > decoded subject line, reencode it, and then send the mail along.
> >
> > The reason I want to do it, is because some IMAP clients do crash when
> > it tries to export/backup email with too long subject lines as it
> > tries to write the subject as a filename, and the operating system
> > returns a error code the IMAP client don't understand = crash.
> >
> > Also I want in the same way, filter away all and any emojii (for
> > example the truck in Ebay's "Order is now being delivered" emails)
> > from subject lines because emojii also causes these IMAP clients im
> > talking about to crash (because obviously emojii can't be used as
> > filenames), and emoji in email subjects are so useless so im
> > speakless...
> >
> > Any ideas?
>
> Which format is exim using to save messages: unix mailbox, MBX, maildir,
> mailstore ?
>
> I'm surprised thst you are using the subject as the filename; you
> certainly don't have to do that; the appendfile driver for the
> local_delivery transport allows you to set the filename exim uses; you
> have presumably explicitly set this to include the subject line, so
> can change it to something else ?
>
> Which IMAP *server* is causing these problems ?
>
> --
> Andrew C. Aitchison Cambridge, UK
> andrew@aitchison.me.uk
>

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
DKIM signatures are not a problem, as incoming mail are validated
Before applying any rules, and outgoing mail is signed last, after all
rules have been applied.
The server never passes already-signed email to a third-party server,
it Always act as a source-signer or destination-verifier, ergo either
its the final destination of email and DKIM is being verified Before
rules are applied, or its the first MTA after user's SMTP client, and
thus it will apply rules and then sign the email.
Den ons 28 nov. 2018 kl 11:09 skrev Mike Brudenell via Exim-users
<exim-users@exim.org>:
>
> Hi, Sebastian -
>
> What you suggest doesn't sound to be a good idea to me, unless you're
> planning to do it *only* at the point of final delivery into the
> recipient's mailbox. (You saying you'll make the change then "send the
> message along" suggests you're not thinking of doing this though, but
> applying it to all messages as they pass through.)
>
> I say this because of the message has been digitally signed using DKIM
> altering the Subject line will invalidate the signature and possibly cause
> problems at any downstream mail servers the message is transmitted to:
> possibly causing the message to be marked as spam or rejected.
>
> If the problem lies in certain IMAP clients your users use wouldn't it be
> better to report the bug in them and get them fixed? It sounds like they're
> broken!
>
> Cheers,
> Mike B-)
>
> On Wed, 28 Nov 2018 at 07:04, Sebastian Nielsen via Exim-users <
> exim-users@exim.org> wrote:
>
> > How can I reliably cut subjects that are too long, without breaking
> > for example quoted-printable encodings and such?
> >
> > Ergo, if the subject line exceeds X characters (where X characters are
> > a limit I as system administrator decides) it should just truncate the
> > decoded subject line, reencode it, and then send the mail along.
> >
> > The reason I want to do it, is because some IMAP clients do crash when
> > it tries to export/backup email with too long subject lines as it
> > tries to write the subject as a filename, and the operating system
> > returns a error code the IMAP client don't understand = crash.
> >
> > Also I want in the same way, filter away all and any emojii (for
> > example the truck in Ebay's "Order is now being delivered" emails)
> > from subject lines because emojii also causes these IMAP clients im
> > talking about to crash (because obviously emojii can't be used as
> > filenames), and emoji in email subjects are so useless so im
> > speakless...
> >
> > Any ideas?
> >
> > --
> > ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> > ## Exim details at http://www.exim.org/
> > ## Please use the Wiki with this list - http://wiki.exim.org/
> >
>
>
> --
> Systems Administrator & Change Manager
> IT Services, University of York, Heslington, York YO10 5DD, UK
> Tel: +44-(0)1904-323811
>
> Web: www.york.ac.uk/it-services
> Disclaimer: www.york.ac.uk/docs/disclaimer/email.htm
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 28/11/2018 14:08, Sebastian Nielsen via Exim-users wrote:
> The problem is when the user, in the IMAP client, selects to "backup"
> or "export" the mails to harddrive.

You could take the extreme-violence route:

Copy the Subject: to a new header. Then just delete
any top-bit-set bytes in the Subject header, and don't
care about any encoding.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 28/11/2018 14:11, Sebastian Nielsen via Exim-users wrote:
> DKIM signatures are not a problem, as incoming mail are validated
> Before applying any rules

... except for MUAs that do their own DKIM-checking
--
Cheers,
Jeremy



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 2018-11-28 15:31, Jeremy Harris wrote:

> > DKIM signatures are not a problem, as incoming mail are validated
> > Before applying any rules
>
> ... except for MUAs that do their own DKIM-checking

And spamassassin if it is plugged somewhere betwixt exim and the MUA.

--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On Wed, Nov 28, 2018 at 3:23 PM Sebastian Nielsen via Exim-users <
exim-users@exim.org> wrote:

>
> The problem is that these IMAP clients, then directly write the
> subject as a file name, and of course some characters are not allowed
> as file name, and some subjects are too long, causing the operating
> system to reject the filename.
> This in turn is unhandled in the IMAP client, causing it to suddenly
> crash during copying.
>

Could you please let us know which IMAP clients do this?

Also, have you reported the bug to the IMAP client vendor(s)/developer(s)?
--
Jan
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
Thank you for the Point into the right direction.
Stumbled upon a new problem now:

I use:

remove_header = subject
add_header = Subject: ${rfc2047:${length_64:$h_subject:}}

in acl_data along with a accept rule.

Now to the problem.
It seems that it does cut the ENCODED subject into 64 characters.
According to documentation, $h_subject: should return the decoded
subject.
So the resulting subject becomes like this:
=?UTF-8?B?dGVzdGFyIMOlw6TDtiDDhcOEw5YgcsOka3Ntw7ZyZ8OlcyBSw4RLU0

Where obviously the encoded string was cut into 64 characters.

Also tried with:
add_header = Subject: ${rfc2047:${length_64:${rfc2047d:$h_subject:}}}

but didn't work either.

Somehow it seems that the ${rfc2047:} part is executed Before
${length_64:} part. And I tried swapping them but didn't work either.
(subject got even weirder).
Den ons 28 nov. 2018 kl 11:46 skrev Dmitriy Matrosov via Exim-users
<exim-users@exim.org>:
>
> Hi.
>
> On November 28, 2018 9:59:22 AM GMT+03:00, Sebastian Nielsen via Exim-users <exim-users@exim.org> wrote:
> >How can I reliably cut subjects that are too long, without breaking
> >for example quoted-printable encodings and such?
>
> Decoded subject is in $h_Subject: , to reencode it you may use ${rfc2047}. We use the following for altering spam subject:
>
> headers_remove = ${if bool{${extract {spam-tag} {$address_data}}} {Subject}}
> headers_add = ${if bool{${extract {spam-tag} {$address_data}}} {Subject: ${rfc2047:*****SPAM***** $h_Subject:}}}
>
> It certainly breaks dkim, but some people still wants this..
>
> Also, resulting encoding won't be the same as original, but it still will be correct.
>
> >Ergo, if the subject line exceeds X characters (where X characters are
> >a limit I as system administrator decides) it should just truncate the
> >decoded subject line, reencode it, and then send the mail along.
> >
> >The reason I want to do it, is because some IMAP clients do crash when
> >it tries to export/backup email with too long subject lines as it
> >tries to write the subject as a filename, and the operating system
> >returns a error code the IMAP client don't understand = crash.
> >
> >Also I want in the same way, filter away all and any emojii (for
> >example the truck in Ebay's "Order is now being delivered" emails)
> >from subject lines because emojii also causes these IMAP clients im
> >talking about to crash (because obviously emojii can't be used as
> >filenames), and emoji in email subjects are so useless so im
> >speakless...
> >
> >Any ideas?
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 28 Nov 2018, at 1:59, Sebastian Nielsen via Exim-users wrote:

> How can I reliably cut subjects that are too long, without breaking
> for example quoted-printable encodings and such?
>
> Ergo, if the subject line exceeds X characters (where X characters are
> a limit I as system administrator decides) it should just truncate the
> decoded subject line, reencode it, and then send the mail along.

You should be able to use Exim's embedded Perl interpreter to do this,
after writing a suitable Perl script to call. Using the MIME::Tools
modules should make it pretty painless. See
https://www.exim.org/exim-html-current/doc/html/spec_html/ch-systemwide_message_filtering.html#SECTaddremheasys
for where to hook it in.

But, as others have said: you shouldn't.

> The reason I want to do it, is because some IMAP clients do crash when
> it tries to export/backup email with too long subject lines as it
> tries to write the subject as a filename, and the operating system
> returns a error code the IMAP client don't understand = crash.

That's a very broken piece of software. No email client should blindly
use a Subject header as a file name without sanitizing it and no
software that saves files should crash just because its attempt to
create a file fails. I have a hard time believing that there are
actually multiple such "IMAP clients" because these basic issues are far
simpler to get right than implementing IMAP.

> Also I want in the same way, filter away all and any emojii (for
> example the truck in Ebay's "Order is now being delivered" emails)
> from subject lines because emojii also causes these IMAP clients im
> talking about to crash (because obviously emojii can't be used as
> filenames),

Not obvious at all. Not even true on all common systems. Emoji are just
standard Unicode characters and any filesystem that can handle UTF-8 or
UTF-16 in names can use emoji. What filesystem are you using that can't
use emoji in filenames?

> and emoji in email subjects are so useless so im
> speakless...
>
> Any ideas?

Get better client software?

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 2018-11-28 18:18, Bill Cole wrote:

> Using the MIME::Tools modules should make it pretty painless.

[...]

> But, as others have said: you shouldn't.

An extra warning to anyone thinking about this: if you let Perl regenerate
the _entire message_ (including the body), you'll be sorry.

http://lists.mutt.org/pipermail/mutt-users/Week-of-Mon-20181119/000560.html

(and up and down the thread)

--
Please don't Cc: me privately on mailing lists and Usenet,
if you also post the followup to the list or newsgroup.
To reply privately _only_ on Usenet and on broken lists
which rewrite From, fetch the TXT record for no-use.mooo.com.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On Wed, 28 Nov 2018, Bill Cole via Exim-users wrote:
> On 28 Nov 2018, at 1:59, Sebastian Nielsen via Exim-users wrote:
>> The reason I want to do it, is because some IMAP clients do crash when
>> it tries to export/backup email with too long subject lines as it
>> tries to write the subject as a filename, and the operating system
>> returns a error code the IMAP client don't understand = crash.
>
> That's a very broken piece of software. No email client should blindly use a
> Subject header as a file name without sanitizing it and no software that
> saves files should crash just because its attempt to create a file fails. I
> have a hard time believing that there are actually multiple such "IMAP
> clients" because these basic issues are far simpler to get right than
> implementing IMAP.

I'm not even sure that these are regular email clients.
A quick search took me to
https://stackoverflow.com/questions/19291796/save-mail-with-subject-as-filename
(OK, not IMAP) and other places where people are rolling there own
scripts which cannot handle unconstrained filenames.
This one fails on the ":" in "Re:" :-(

>> Any ideas?
>
> Get better client software?

While using the Subject: line as the filename is superficially
attractive, I'm not sure it makes sense as it isn't unique
(are they trying to keep threads together ?) so you need to
save multiple messages in one file ...

--
Andrew C. Aitchison Cambridge, UK
andrew@aitchison.me.uk

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 2018-11-28, Jeremy Harris via Exim-users <exim-users@exim.org> wrote:
> On 28/11/2018 14:08, Sebastian Nielsen via Exim-users wrote:
>> The problem is when the user, in the IMAP client, selects to "backup"
>> or "export" the mails to harddrive.
>
> You could take the extreme-violence route:
>
> Copy the Subject: to a new header. Then just delete
> any top-bit-set bytes in the Subject header, and don't
> care about any encoding.

also remove slashes backslashes dots, colons, and any other ascii characters
that windows can't handle in filenames... I'm not convinced that the
cure is better than the disease.

> --
> Cheers,
> Jeremy
>


--
When I tried casting out nines I made a hash of it.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On 2018-11-28, Sebastian Nielsen via Exim-users <exim-users@exim.org> wrote:
> Thank you for the Point into the right direction.
> Stumbled upon a new problem now:
>
> I use:
>
> remove_header = subject
> add_header = Subject: ${rfc2047:${length_64:$h_subject:}}
>
> in acl_data along with a accept rule.
>
> Now to the problem.
> It seems that it does cut the ENCODED subject into 64 characters.
> According to documentation, $h_subject: should return the decoded
> subject.
> So the resulting subject becomes like this:
>=?UTF-8?B?dGVzdGFyIMOlw6TDtiDDhcOEw5YgcsOka3Ntw7ZyZ8OlcyBSw4RLU0

I can't say that it's not working correctly.

Probably you've got the originator feeding you subject lines that are not encoded according to RFC2047 there's a switch somewhere that makes exim incompatible with RFC2047 to support these broken email sources.
It's something about length.

--
When I tried casting out nines I made a hash of it.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
On November 29, 2018 2:34:35 PM GMT+03:00, Jasen Betts via Exim-users <exim-users@exim.org> wrote:
>On 2018-11-28, Sebastian Nielsen via Exim-users <exim-users@exim.org>
>wrote:
>> Thank you for the Point into the right direction.
>> Stumbled upon a new problem now:
>>
>> I use:
>>
>> remove_header = subject
>> add_header = Subject: ${rfc2047:${length_64:$h_subject:}}
>>
>> in acl_data along with a accept rule.
>>
>> Now to the problem.
>> It seems that it does cut the ENCODED subject into 64 characters.
>> According to documentation, $h_subject: should return the decoded
>> subject.
>> So the resulting subject becomes like this:
>>=?UTF-8?B?dGVzdGFyIMOlw6TDtiDDhcOEw5YgcsOka3Ntw7ZyZ8OlcyBSw4RLU0
>
>I can't say that it's not working correctly.
>
>Probably you've got the originator feeding you subject lines that are
>not encoded according to RFC2047 there's a switch somewhere that makes
>exim incompatible with RFC2047 to support these broken email sources.
>It's something about length.

'check_rfc2047_length = false' at main level.. That may be the case, i completely forget about it.. As far as i remember, thunderbird uses wrong encoded words length.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
Thank you. That solved the problem. Its Actually gmail supplying
rfc2047 with incorrect length, so I assume its a good idea to keep it
at false then.
Maybe it should even be default false?
Den tors 29 nov. 2018 kl 12:54 skrev Dmitriy Matrosov via Exim-users
<exim-users@exim.org>:
>
>
>
> On November 29, 2018 2:34:35 PM GMT+03:00, Jasen Betts via Exim-users <exim-users@exim.org> wrote:
> >On 2018-11-28, Sebastian Nielsen via Exim-users <exim-users@exim.org>
> >wrote:
> >> Thank you for the Point into the right direction.
> >> Stumbled upon a new problem now:
> >>
> >> I use:
> >>
> >> remove_header = subject
> >> add_header = Subject: ${rfc2047:${length_64:$h_subject:}}
> >>
> >> in acl_data along with a accept rule.
> >>
> >> Now to the problem.
> >> It seems that it does cut the ENCODED subject into 64 characters.
> >> According to documentation, $h_subject: should return the decoded
> >> subject.
> >> So the resulting subject becomes like this:
> >>=?UTF-8?B?dGVzdGFyIMOlw6TDtiDDhcOEw5YgcsOka3Ntw7ZyZ8OlcyBSw4RLU0
> >
> >I can't say that it's not working correctly.
> >
> >Probably you've got the originator feeding you subject lines that are
> >not encoded according to RFC2047 there's a switch somewhere that makes
> >exim incompatible with RFC2047 to support these broken email sources.
> >It's something about length.
>
> 'check_rfc2047_length = false' at main level.. That may be the case, i completely forget about it.. As far as i remember, thunderbird uses wrong encoded words length.
>
> --
> ## List details at https://lists.exim.org/mailman/listinfo/exim-users
> ## Exim details at http://www.exim.org/
> ## Please use the Wiki with this list - http://wiki.exim.org/

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: cut subjects that are too long + delete emojii from subject lines... [ In reply to ]
Thanks to everyone who helped out with this.
After a lot of tinkering, the final rules came out like this inside an
accept rule:

#Fix for wonky IMAP clients
remove_header = date
remove_header = subject
add_header = Date: $tod_full
add_header = Subject:
${rfc2047:${length_100:${sg{${sg{${sg{${sg{${sg{${sg{${sg{${sg{${sg{${sg{${sg{${sg{$h_subject:}{\\xE5}{å}}}{\\xC4}{Ä}}}{\\xD6}{Ö}}}{\\xC5}{Å}}}{\\xF6}{ö}}}{\\xE4}{ä}}}{\N[^a-zA-Z0-9åäöÅÄÖ
!"\@#\$%&\/\{(\[)\]=\}?+\\\-_:.;,*><|^~]\N}{}}}{\N\xC3[^\xA5\xA4\xB6\x85\x84\x96]{1}\N}{}}}{\\xC3\$}{}}}{
}{ }}}{ }{ }}}{ }{ }}}}

First off, it fixes the date header. It does this by removing the date
header and then adding it with correct server time which is synced by
NTP. This because some IMAP clients do set their date to 1970-01-01
00:00:00 (unix time 0) for some weird reason, and this causes either
the mail to appear at bottom of the recipients screen, or being
outright rejected as spam.
Additionally, if a email arrives with a Date: header that is a bit
off, it will, in certain wonky IMAP clients cause the mail to be
sorted weirdly where a unread mail might pop in just a few mails below
the newest, and when this behaviour cannot be changed in the client,
then you simply have to change it in server :-)

Then the subject rule:

If we start inwards (from $h_subject:), the first 6 ${sg{'s do replace
certain Swedish characters ISO-8859-1 equvalient with their UTF-8
equvalients. The reason for doing this, is because certain newsletter
mailers, do encode their subjects in ISO-8859-1, but do put UTF-8 as
encoding format, causing these characters to become garbled. The rule
fixes their incorrect encoding. I have never yet stumbled upon a
sender who transmits as UTF-8 but specify ISO-8859-1 as encoding.
And no, I couldn't use tr for some reason because then successive
characters (like åäö) become incorrectly decoded.

Then, there comes a rule that strictly limit characters in subject to:
a-zA-Z0-9åäöÅÄÖ !"@#$%&/{([)]=}?+\-_:.;,*><|^~
I found out by testing, that Actually, the broken IMAP clients do
replace forbidden characters in this particular set to _ (underscore)
in filenames.
Its just that they have forgot certain other characters that cause
files to not to be written.

After that, it comes a rule that Begins on \xC3 but with negated
following \xA5 and so on. This is because of a limitation of PCRE and
UTF-8 characters, which will allow all UTF-8 characters beginning on
\xC3, but still filter away the second part of the character. This
additional rule filters away the leftover \xC3 from those UTF-8
characters that isn't åäöÅÄÖ.

Finally, there comes a few replacement rules that recursively tidies
up the subject line by replacing all occurences of multiple spaces
into one single space. Since ${sg only "Scans" one time (to prevent
endless loops) , I redo it a few times to ensure that all multiple
spaces are compressed into one space.
The reason to do this, is because a subject that have a forbidden
character sorrounded by 2 spaces, like "something ñ something", after
filtering it will be 2 spaces in a row. If theres multiple such
occurences, it can come a lot of spaces in a row.
This tidies up the subject line greatly and makes it nice.

After that, the subject line is cut into 100 characters to ensure file
names does not become too long and being rejected on certain
filesystems, and then its encoded with rfc 2047.

These rules are run on:
1: Just Before a mail is being delivered into a user's mailbox
(dovecot) by final delivery. At this time, DKIM is already verified.
OR
2: Just Before a mail is being DKIM-signed due to a local relay from
an authorized user, and being sent off to a remote SMTP server.


Im so happy, it works flawlessly with all sorts of wonky and weird
IMAP clients, mailer software and even buggy printers who check IMAP
to print out a attached PDF file.
Thats how you do it to ensure interoperability with clients that don't
strictly adhere to standards :-)

Feel free to tweak the rule to add/remove characters belonging to your
local language if you want to adopt the rule in your exim4 server.

Best regards, Sebastian Nielsen

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/