Mailing List Archive

Spurious DKIM failures
Hello.

I have an informational message for developers. In some rare cases Exim
considers correct DKIM signature as invalid. My estimate of fault rate
is less than 1 event for 10,000 imcoming mails.

In my environment it can be traced by headers, because each incoming
mail passes the chain

(1) Exim MTA -> (2) Amavis -> (3) Exim MTA,

where receiving relay (1) and anti-spam filter (2) save result of
DKIM verification in headers. On the failure, headers are like:

Authentication-Results: passat.rdtex.ru (amavisd-new);
dkim=pass (1024-bit key) header.d=netology.ru header.b=OXxIl1Hh;
dkim=pass (1024-bit key) header.d=mta.mindbox.ru header.b=I5B1tR/y
Received: from passat.rdtex.ru ([127.0.0.1])
by localhost (passat.rdtex.ru [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id UjeSDm4hSaiL for <xxxxxxx.xxxxxxx@rdtex.ru>;
Mon, 5 Jul 2021 19:06:34 +0300 (MSK)
X-Authentication-Results: passat.rdtex.ru Exim-4.94.2;
iprev=pass (mta.mindbox.ru) smtp.remote-ip=185.99.9.135;
dkim=fail (body hash mismatch; body probably modified in transit)
header.d=netology.ru header.s=mindbox header.a=rsa-sha256;
dkim=fail (body hash mismatch; body probably modified in transit)
header.d=mta.mindbox.ru header.s=mindbox header.a=rsa-sha256
X-Greylist: from auto-whitelisted by SQLgrey-1.8.0
Received-SPF: pass client-ip=185.99.9.135; envelope-from=bounce.b46b070004000000607c60a5@mta.mindbox.ru; helo=mta.mindbox.ru
Received: from mta.mindbox.ru ([185.99.9.135]:29146)
by passat.rdtex.ru with esmtps (TLS1.2:ECDHE_X25519__RSA_SHA256__AES_256_GCM:256)
(Exim 4.94.2)
(envelope-from <bounce.b46b070004000000607c60a5@mta.MINDBOX.ru>)
id 1m0R7C-00HEVp-GA size 24034 maxlen 175
...

Mainlog for frontend Exim (1) contains:

2021-07-05 19:06:34.545 [4107365] 1m0R7C-00HEVp-GA DKIM: d=netology.ru s=mindbox c=relaxed/relaxed a=rsa-sha256 b=1024 [.verification failed - body hash mismatch (body probably modified in transit)]
2021-07-05 19:06:34.545 [4107365] 1m0R7C-00HEVp-GA DKIM: d=mta.mindbox.ru s=mindbox c=relaxed/relaxed a=rsa-sha256 b=1024 [.verification failed - body hash mismatch (body probably modified in transit)]

Manual test of the received mail with perl module Mail::DKIM::Verifier
gives "pass" result for both signatures. If sample mail is re-injected
by SMTP from other host, it passes verification on frontend Exim.

I have 4 frontend relays with almost identical configuration, failures
happen on each of them, randomly. There are no evidence for hardware
problems (segfaults, etc). No evidence of "inherited" memory corruption:
after failure of DKIM verificaion subsequent mails are verified correctly.
So it seems as rarely manifested bug.

I have no ideas how such bug can be located. However, I can share this
sample mail with developers (in private).
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Spurious DKIM failures [ In reply to ]
On 05/07/2021 22:19, Evgeniy Berdnikov via Exim-users wrote:
> No evidence of "inherited" memory corruption:
> after failure of DKIM verificaion subsequent mails are verified correctly.
> So it seems as rarely manifested bug.

I agree, rather difficult to know how to work on this.

You say that subsequent mails are ok; is that subsequent within
a single connection? What about previous ones on the same
connection - does that ever happen?



While there have been several changes in the DKIM code that probably
are not in the binary you are running:

$ git diff --stat exim-4.94.2 master src/dkim.c src/pdkim
src/src/dkim.c | 22 +++++++++++++++-------
src/src/pdkim/pdkim.c | 1 +
src/src/pdkim/signing.c | 3 ++-
3 files changed, 18 insertions(+), 8 deletions(-)

these only amount to a) more taint-tracking b) more efficient memory
use over multi-message connections c) maintaining functionality with
more-recent GnuTLS versions. I don't see any fixes, at least in
dkim-specific code.

--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Spurious DKIM failures [ In reply to ]
On Tue, Jul 06, 2021 at 08:32:36PM +0100, Jeremy Harris via Exim-users wrote:
> You say that subsequent mails are ok; is that subsequent within
> a single connection?

No, I mean accepted by process subsequently forked from the same
listening daemon. I did re-injecting of sample mail to network, it was
accepted by the same host, the same listening process (daemon was not
restarted), and passed with DKIM successfully verified.

> What about previous ones on the same
> connection - does that ever happen?

I can't imagine how it could happen in wild. But one can do testing
with subsequent submissions over one connection.

> While there have been several changes in the DKIM code that probably
> are not in the binary you are running:

I run up-to-date version from Debian/bullseye (package exim4-daemon-heavy
4.94.2-5). Production environment, so the simplest way for me is to wait
for Debian update and look for result.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Spurious DKIM failures [ In reply to ]
Hello.

Returning to topic discussed a month ago...

On Wed, Jul 07, 2021 at 01:21:37AM +0300, Evgeniy Berdnikov wrote:
> On Tue, Jul 06, 2021 at 08:32:36PM +0100, Jeremy Harris via Exim-users wrote:
> > While there have been several changes in the DKIM code that probably
> > are not in the binary you are running:
>
> I run up-to-date version from Debian/bullseye (package exim4-daemon-heavy
> 4.94.2-5). Production environment, so the simplest way for me is to wait
> for Debian update and look for result.

Recently I install 4.95-RC2 on Debian/testing (32bit) and got segfaults
(they are reported in other thread). As for spurious DKIM failures,
they are present in this version too:

----------------------------------------------------------------------------
Authentication-Results: citrine.rdtex.ru (amavisd-new);
dkim=pass (1024-bit key) header.d=netology.ru header.b=JK5fS3uY;
dkim=pass (1024-bit key) header.d=mta.mindbox.ru header.b=aHlP1SF6
Received: from citrine.rdtex.ru ([127.0.0.1])
by localhost (citrine.rdtex.ru [127.0.0.1]) (amavisd-new, port 10024)
with ESMTP id fqfLoPQknlH6 for <xxxxxxxx.xxxxxxxxxx@rdtex.ru>;
Tue, 7 Sep 2021 15:02:37 +0300 (MSK)
[...]
X-Authentication-Results: citrine.rdtex.ru Exim-4.95-RC2;
iprev=pass (mta.mindbox.ru) smtp.remote-ip=185.99.9.136;
dkim=fail (body hash mismatch; body probably modified in transit)
header.d=netology.ru header.s=mindbox header.a=rsa-sha256;
dkim=fail (body hash mismatch; body probably modified in transit)
header.d=mta.mindbox.ru header.s=mindbox header.a=rsa-sha256
Received-SPF: pass (mta.mindbox.ru: 185.99.9.136 is authorized to use
'bounce.9ed50800000092d50000c0d2@mta.mindbox.ru' in 'mfrom' identity
(mechanism 'a' matched)) receiver=citrine.rdtex.ru; identity=mailfrom;
envelope-from="bounce.9ed50800000092d50000c0d2@mta.mindbox.ru";
client-ip=185.99.9.136
----------------------------------------------------------------------------

So, problem still exists.

There is some little progress in attempts to locate it: I found that
this sender (mta.mindbox.ru) have maximum probability to produce fault.
I've removed STARTTLS for its relays and made a traffic capture, it shows
that there is some bulk mailer there, and it 1. uses CHUNKING, 2. send
whole mail in single BDAT, 3. cut off connection without waiting for
status code, 4. mail body is sent without final CRLF. Raw mail body,
extracted from pcap data, passes DKIM test for both signatures.

Removal of TLS layer does not help to prevent DKIM failures. But not all
mails from this sender lead to fault, some mails pass DKIM verification.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Spurious DKIM failures [ In reply to ]
On 07/09/2021 14:44, Evgeniy Berdnikov via Exim-users wrote:
> There is some little progress in attempts to locate it: I found that
> this sender (mta.mindbox.ru) have maximum probability to produce fault.
> I've removed STARTTLS for its relays and made a traffic capture, it shows
> that there is some bulk mailer there, and it 1. uses CHUNKING, 2. send
> whole mail in single BDAT, 3. cut off connection without waiting for
> status code, 4. mail body is sent without final CRLF. Raw mail body,
> extracted from pcap data, passes DKIM test for both signatures.

44a62f5861 looks relevant:

JH/59 DKIM: Fix small-message verification under TLS with chunking. If a
pipelined SMTP command followed the BDAT LAST then it would be
incorrectly treated as part of the message body, causing a verification
fail.

... except for:> Removal of TLS layer does not help to prevent DKIM failures.

Worth checking you have it, though.


On your segfaults: I tried creating a callout/TLS using testsuite
facilities, and am not seeing a fault (on a 32b Debian testing,
with whatever GnuTLS version that has) either with the current
master tip or with tag exim-4.94.2
I think we'll need you to investigate one of your coredumps.

Another test you could try is a build with
#define MISSING_POSIX_MEMALIGN
in OS/os.h-Linux.
If that still coredumps in the same place, it's not a "write
to readonly data" trap; so more likely a null pointer.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Spurious DKIM failures [ In reply to ]
On Tue, Sep 07, 2021 at 03:32:02PM +0100, Jeremy Harris via Exim-users wrote:
> 44a62f5861 looks relevant:
>
> JH/59 DKIM: Fix small-message verification under TLS with chunking. If a
> pipelined SMTP command followed the BDAT LAST then it would be
> incorrectly treated as part of the message body, causing a verification
> fail.

PIPELINING is not advertised. I also studied smtp session in traffic
dump: sender wait for replies and do not send commands ahead.

> ... except for:> Removal of TLS layer does not help to prevent DKIM failures.
>
> Worth checking you have it, though.

There are "P=esmtp" in log records and "with esmtp" in headers.
I read traffic dumps for plain smtp connections.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/