Mailing List Archive

Disable deduplication
Scenario: I want to archive customer e-mail. Right now I am sending
copies through a router via "manualroute" and "unseen" which works
perfectly fine.

The only problem occurs when mails have multiple recipients like Bcc.
The recipient of the mails must be rewritten to the same address (i.e.
foobar@archive.tld) for the mailarchive to work – I use headers_add for
adding "X-Envelope-To:" in order for the mail archive to distinguish
them later.

The problem is that Exim recognizes these mails as the same and either
sends one mail with multiple recipients or (when I add "max_rcpt = 1")
sends multiple mails - but these mails are all the same, they only have
the "X-Envelope-To:" of the latest redirection.

Debugging the process shows:

"[duplicate, would not be delivered]"

As far as I understand this is expected behaviour and only within pipes
mails are treated differently.

tl;dr: how can I force Exim to actually create single and unique mails
instead of just ignoring duplicates?

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
On 16/07/2020 16:19, Kai Bojens via Exim-users wrote:
> tl;dr: how can I force Exim to actually create single and unique mails
> instead of just ignoring duplicates?

Unclear exactly what you are wanting duplicated that is
not currently.

There was one mail message, with multiple original recipients.
I assume you are archiving one copy. Did you expect different?
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
On Thu, Jul 16, 2020 at 05:19:52PM +0200, Kai Bojens via Exim-users wrote:
> Scenario: I want to archive customer e-mail. Right now I am sending
> copies through a router via "manualroute" and "unseen" which works
> perfectly fine.
>
> The only problem occurs when mails have multiple recipients like Bcc.
> The recipient of the mails must be rewritten to the same address (i.e.
> foobar@archive.tld) for the mailarchive to work – I use headers_add for
> adding "X-Envelope-To:" in order for the mail archive to distinguish
> them later.

You are doing something strange: get original mail with several addresses
and make several copies with a single-address X-Envelope-To: header.
Why so? Put all recipients into X-Envelope-To: and write copy into archive.
Such approach saves file space.

> The problem is that Exim recognizes these mails as the same and either
> sends one mail with multiple recipients or (when I add "max_rcpt = 1")
> sends multiple mails - but these mails are all the same, they only have
> the "X-Envelope-To:" of the latest redirection.
>
> Debugging the process shows:
>
> "[duplicate, would not be delivered]"
>
> As far as I understand this is expected behaviour and only within pipes
> mails are treated differently.
>
> tl;dr: how can I force Exim to actually create single and unique mails
> instead of just ignoring duplicates?

Unique mails MUST have unique Message-Id's. Some mail storage systems
(Cyrus, for example) remove duplicates by message-id automatically.

Nevetheless, you can pipe mail to you custom script written to create
several files, with the only difference in X-Envelope-To: header, but
for me it looks ridiculous.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
Am 2020-07-16 18:10, schrieb Jeremy Harris via Exim-users:
> Unclear exactly what you are wanting duplicated that is
> not currently.

There is a mail from A@A.tld to B@B.tld and (Bcc:) C@B.tld. I then
want to redirect a copy of these mails (after all this should be seen
as two mails for two recipients) to b.tld-archive@foo.bar (yes,
both are redirected to the same address). In order for the Archive
to differentiate I add the the X-Envelope-To to these mails.

The problem is that Exim actualles does send two mails (with "max_rcpt =
1")
but these are identical - they have the exact same "X-Envelope-To" as
Exim
suppresses the duplicate.

> There was one mail message, with multiple original recipients.
> I assume you are archiving one copy. Did you expect different?

Yes. I know that this sounds crazy - due to regulations we actually have
to
save a copy of every mail. The mail archive software on the other hand
expects all the mail for customer X to arrive at b.tld-archive@foo.bar.

So, I am looking for a way to redirect a mail with several Bcc (or To,
CC)
Recipients as single mails to one recipient.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
Am 2020-07-16 18:14, schrieb Evgeniy Berdnikov via Exim-users:

> You are doing something strange: get original mail with several
> addresses
> and make several copies with a single-address X-Envelope-To: header.
> Why so? Put all recipients into X-Envelope-To: and write copy into
> archive.
> Such approach saves file space.

I have to. Regulations require me to save individual copys.

> Nevetheless, you can pipe mail to you custom script written to create
> several files, with the only difference in X-Envelope-To: header, but
> for me it looks ridiculous.

It _is_ ridiculous. I totally agree with you.

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
Am 16.07.20 um 19:43 schrieb Kai Bojens via Exim-users:
> There is a mail from A@A.tld to B@B.tld and (Bcc:) C@B.tld. I then
> want to redirect a copy of these mails (after all this should be seen
> as two mails for two recipients) to b.tld-archive@foo.bar (yes,
> both are redirected to the same address). In order for the Archive
> to differentiate I add the the X-Envelope-To to these mails.

Ähm.. you don't get BCC headers on the receiving side, that's the idea
behind it.

btw. why don't you a "a@a.tld" to "a-archiv@a.tld"  forwarder? Should be
already in the default.config, ready to use (more or less).

Best regards,
Marius

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
On 16/07/2020 18:43, Kai Bojens via Exim-users wrote:
> Am 2020-07-16 18:10, schrieb Jeremy Harris via Exim-users:
>> Unclear exactly what you are wanting duplicated that is
>> not currently.
>
> There is a mail from A@A.tld to B@B.tld and (Bcc:) C@B.tld. I then
> want to redirect a copy of these mails (after all this should be seen
> as two mails for two recipients)

Nope; your assumption is wrong. It is one message with two recipients,
as defined by the envelope. The headers are not relevant here.

> due to regulations we actually have to
> save a copy of every mail

You need to look very closely at what they are defining as "a mail".
In particular, at what stage in its lifecycle for this case of a
multi-recipient message.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
On Thu, Jul 16, 2020 at 07:43:12PM +0200, Kai Bojens via Exim-users wrote:
> So, I am looking for a way to redirect a mail with several Bcc (or To, CC)
> Recipients as single mails to one recipient.

I think you have no way but get original message out of Exim and generate
several copies with different headers by some external script:

[transports]

archive:
driver = pipe
command = "/path/to/forwarder"
envelope_to_add

The "forwarder" script should read message from stdin, extract contents
of Envelope-To: header for list of recipients, for each recipient
create copy with some marker attached (as "X-Envelope-To: address"),
and finally forward this copy to your archive host, as a single mail.

Maybe alternative solutions exist, say, with use of $recipients
variable instead of envelope_to_add.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
Evgeniy Berdnikov via Exim-users <exim-users@exim.org> (Do 16 Jul 2020 18:14:21 CEST):
> Unique mails MUST have unique Message-Id's. Some mail storage systems
> (Cyrus, for example) remove duplicates by message-id automatically.

And I'm looking for the cyrus option, that would disable exactly this.
Often I get responses to list messages as (B)CC. The (B)CC often arrives
sooner than the response via the list. But only the response via the
list would give me the right list headers for replying and such.

But, yes, it's another topic than the OP want's to solve.

--
Heiko
Re: Disable deduplication [ In reply to ]
On Fri, Jul 17, 2020 at 12:21:54AM +0200, Heiko Schlittermann via Exim-users wrote:
> Evgeniy Berdnikov via Exim-users <exim-users@exim.org> (Do 16 Jul 2020 18:14:21 CEST):
> > Unique mails MUST have unique Message-Id's. Some mail storage systems
> > (Cyrus, for example) remove duplicates by message-id automatically.
>
> And I'm looking for the cyrus option, that would disable exactly this.

AFAIK, this is a hardcoded behaviour of Cyrus, without any regulation.
But partially broken in some versions, :) say, cyrus 2.5.10 sometimes
don't remove duplicates, this is definitely a bug.

If you want to keep duplicates in IMAP box, I recommend Dovecot.

> Often I get responses to list messages as (B)CC. The (B)CC often arrives
> sooner than the response via the list. But only the response via the
> list would give me the right list headers for replying and such.

You may put procmail filter behind cyrus deliver to remove duplicates
that are Cc/Bcc copies to know list addresses, in order to wait for a
copy from list manager. It may be done with custom router for cyrus
delivery: run procmail and call cyrdeliver on the end of recipe.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Disable deduplication [ In reply to ]
On 7/17/20 3:29 AM, Evgeniy Berdnikov via Exim-users wrote:
> On Fri, Jul 17, 2020 at 12:21:54AM +0200, Heiko Schlittermann via Exim-users wrote:
>> Evgeniy Berdnikov via Exim-users <exim-users@exim.org> (Do 16 Jul 2020 18:14:21 CEST):
>>> Unique mails MUST have unique Message-Id's. Some mail storage systems
>>> (Cyrus, for example) remove duplicates by message-id automatically.
>>
>> And I'm looking for the cyrus option, that would disable exactly this.
>
> AFAIK, this is a hardcoded behaviour of Cyrus, without any regulation.
> But partially broken in some versions, :) say, cyrus 2.5.10 sometimes
> don't remove duplicates, this is definitely a bug.
>
> If you want to keep duplicates in IMAP box, I recommend Dovecot.

Not sure about the newer versions of Cyrus but 2.4.x had this option:


duplicatesuppression: 1

If enabled, lmtpd will suppress delivery of a message to a mailbox if a
message with the same message-id (or resent-message-id) is
recorded as having already been delivered to the mailbox. Records the
mailbox and message-id/resent-message-id of all successful deliveries.


>
>> Often I get responses to list messages as (B)CC. The (B)CC often arrives
>> sooner than the response via the list. But only the response via the
>> list would give me the right list headers for replying and such.
>
> You may put procmail filter behind cyrus deliver to remove duplicates
> that are Cc/Bcc copies to know list addresses, in order to wait for a
> copy from list manager. It may be done with custom router for cyrus
> delivery: run procmail and call cyrdeliver on the end of recipe.
>