Mailing List Archive

421 lost input connection, not logged?
Hi,

Spent a few minutes today figuring out what happens to a certain client
trying to send mail to our Exim 4.93 #3 MTA since no traces of their IP
could be found in our logs. But tcpdumping shows the session being
terminated after STARTTLS, with Exim sending '421 lost input connection'
over the wire[1].

I very much suspect the issue to be on the client side, as we're not
experiencing other similar issues with TLS in our setup, but i'm
wondering why nothing is logged about this.

Our current log_selector looks like this:

log_selector = +all_parents \
+delivery_size \
+incoming_interface \
+incoming_port \
+smtp_confirmation \
+smtp_protocol_error \
+smtp_syntax_error \
+queue_time \
+deliver_time \
+tls_cipher \
+tls_peerdn \
-retry_defer

And AFAIK this is only adding/removing, not setting, the log_selector
list, so defaults like 'lost_incoming_connection'[2] should still be
active, right?

Any bright ideas?

Regards,
-Sander.

[2]
https://www.exim.org/exim-html-current/doc/html/spec_html/ch-log_files.html#SECTlogselector

[1]
This mail is more about the logging not happening, but if one is so
inclined or has some insight in this, here is the timeline of the lost
connection. This all happens in .082328 seconds according to tcpdump.
The remote MTA (aparently a Win2K12r2 box) issues STARTTLS, my server
says Go ahead, remote MTA sends a packet i can't identify at this moment
but which must be some TLS handshake and it carries the name of my
server, then my server sends a few packets containing my valid wildcard
cert matching the name the remote MTA sent in its packet, then
immediately the connection is 'lost':
0x0020: 5018 01f5 96ff 0000 3432 3120 736d 7470 P.......421.smtp
0x0030: 2e62 6974 2e6e 6c20 6c6f 7374 2069 6e70 .bit.nl.lost.inp
0x0040: 7574 2063 6f6e 6e65 6374 696f 6e0d 0a ut.connection..

--
| /dev/hda1 has been checked 20 times without being mounted, mount forced
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
Am 08.10.20 um 15:55 schrieb Sander Smeenk via Exim-users:
> Hi,
>
> Spent a few minutes today figuring out what happens to a certain client
> trying to send mail to our Exim 4.93 #3 MTA since no traces of their IP
> could be found in our logs. But tcpdumping shows the session being
> terminated after STARTTLS, with Exim sending '421 lost input connection'
> over the wire[1].
>

Your Exim is fine, had that last week with a 4.94 . It was caused by
ANTIVIRUS (Avast if i'm not mistaken) Software intercepting the
transmission, presenting a bogus cert and than the client rejected it
and that was it for the connection.

It did not leave a trace in the logs.

@Devs: Maybe a nice FR to put a "421 client aborted connection from ip
x.x.x.x." in the logs.

best regards,
Marius


--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
On 08/10/2020 15:34, Cyborg via Exim-users wrote:
> @Devs: Maybe a nice FR to put a "421 client aborted connection from ip
> x.x.x.x." in the logs.

Not sure what the significance of the "@" is, nor what an FR might be...

If it was supposed to be an RFE, bugs.exim.org is the place to record
it.
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
On 8 Oct 2020, at 15:34, Cyborg via Exim-users <exim-users@exim.org> wrote:
> @Devs: Maybe a nice FR to put a "421 client aborted connection from ip
> x.x.x.x." in the logs.

...but if the client never managed to actually connect to Exim, there is nothing to log.

You can already add the log selector "smtp_connection" to log the start & end of a session.

At $workplace we use:

log_selector = +smtp_syntax_error +smtp_protocol_error +incoming_interface +subject +pid +queue_time +queue_time_overall +smtp_connection

Graeme
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
Quoting Graeme Fowler via Exim-users (exim-users@exim.org):

> ...but if the client never managed to actually connect to Exim, there is nothing to log.

But the client was connected!
It issued EHLO, STARTTLS and -then- boogered off.

> You can already add the log selector "smtp_connection" to log the start & end of a session.

It would be nice if such situations were logged without me having to log
every TCP SYN to the Exim daemon. ;-)

If this is not a config issue on my side, i might open an RFE on
bugs.exim.org!

Thanks,
-Sndr.
--
| If someone's going through a storm, your silent presence
| is more important than a million empty words ~ Thema Davis
| 4096R/20CC6CD2 - 6D40 1A20 B9AA 87D4 84C7 FBD6 F3A9 9442 20CC 6CD2

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
Hello.

On Thu, Oct 08, 2020 at 03:55:48PM +0200, Sander Smeenk via Exim-users wrote:
> Our current log_selector looks like this:
>
> log_selector = +all_parents \
> +delivery_size \
> +incoming_interface \
> +incoming_port \
> +smtp_confirmation \
> +smtp_protocol_error \
> +smtp_syntax_error \
> +queue_time \
> +deliver_time \
> +tls_cipher \
> +tls_peerdn \
> -retry_defer

I suggest +all, or add +smtp_connection +smtp_incomplete_transaction.

> This mail is more about the logging not happening, but if one is so
> inclined or has some insight in this, here is the timeline of the lost
> connection. This all happens in .082328 seconds according to tcpdump.
> The remote MTA (aparently a Win2K12r2 box) issues STARTTLS, my server
> says Go ahead, remote MTA sends a packet i can't identify at this moment
> but which must be some TLS handshake and it carries the name of my
> server, then my server sends a few packets containing my valid wildcard
> cert matching the name the remote MTA sent in its packet, then
> immediately the connection is 'lost':
> 0x0020: 5018 01f5 96ff 0000 3432 3120 736d 7470 P.......421.smtp
> 0x0030: 2e62 6974 2e6e 6c20 6c6f 7374 2069 6e70 .bit.nl.lost.inp
> 0x0040: 7574 2063 6f6e 6e65 6374 696f 6e0d 0a ut.connection..

If there is no any packet bitween ServerHello (packets with sertificate)
and packet with message "421 smtp.bit.nl lost input connection", then
connection can be either dropped by kernel, or lost due to some bug
inside ssl library.

I propose to add +millisec to log_selector and get trace of syscalls
(with strace -tt), then compare trace with dump to locate the fault
by timestamps. You should catch ECONNREFUSED on read(2), or some
similar error code on read() if kernel drops connection.
--
Eugene Berdnikov

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
Am 08.10.20 um 17:11 schrieb Graeme Fowler via Exim-users:
> ...but if the client never managed to actually connect to Exim, there is nothing to log.
In this case, you could see the messages exim sent for the smtp session
with tcpdump, but it never showed up in the logs.

and honestly, I don't need it to, as shodan and other vul scanners out
there, never intend to send a messages and they are a pest on scanning
mailservers. If any scan of them shows up in the logs, they will produce
a mess IMHO.

Jeremy:  FR = Feature Request ;) I don't consider this a bug, but i
maybe alone with this.


best regards,
Marius

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
On 08/10/2020 17:50, Cyborg via Exim-users wrote:
> Am 08.10.20 um 17:11 schrieb Graeme Fowler via Exim-users:
>> ...but if the client never managed to actually connect to Exim, there is nothing to log.
> In this case, you could see the messages exim sent for the smtp session
> with tcpdump, but it never showed up in the logs.

Ah, but tcpdump taken where? On the server? client? Some middlebox,
which (as Cyborg) pointed out) could be replacing the TLS connection?

And, yes, random drops are an unfortunate feature of today's environment
and do tend to induce log-pollution. Possibly, if this is was which
isn't visible in any way at present, a candidate for log_selector
+protocol_detail ? New since 4.94, that currently logs only
TCP RST received directly after QUIT (a TCP-level protocol violation).

> Jeremy:  FR = Feature Request

Ah. I call them Request For Enhancement (RFE). But the bugzilla
at bugs.exim.org calls them "wishlist".
--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: 421 lost input connection, not logged? [ In reply to ]
Am Do, Okt 8, 2020 at 20:39 schrieb Jeremy Harris via Exim-users
<exim-users@exim.org>:
> On 08/10/2020 17:50, Cyborg via Exim-users wrote:
>> Am 08.10.20 um 17:11 schrieb Graeme Fowler via Exim-users:
>>> ...but if the client never managed to actually connect to Exim,
>>> there is nothing to log.
>> In this case, you could see the messages exim sent for the smtp
>> session
>> with tcpdump, but it never showed up in the logs.
>
> Ah, but tcpdump taken where? On the server? client? Some middlebox,
> which (as Cyborg) pointed out) could be replacing the TLS connection?
>

Directly at the server. The AV programm was on the clients pc.

Gr??e,
Marius

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/