Mailing List Archive

deny senders in RCPT ACL regular expression and wildcards won't match for some reason
Hi, everyone--

I've been tearing my hair out trying to block specific spammers using a
deny senders = file list with many regular expressions in it. It just never
seems to match the spammer's envelope-from address, even though it looks
like it should match perfectly.

Eventually I decided to start exim in debug mode and watch as it actually
processes the ACLs to confirm the failure to match, and yep, it fails. I
saw it fail to match something it should have matched in real time (see
below).

I am running the exim4-daemon-heavy that comes with Ubuntu 20.04. It is
exim 4.93-13.
I started it up with:

# exim4 -bd -d+acl

The config lines for this deny are:

acl_check_rcpt:

### Deny if sender explicitly in blacklist
# $recipients is not available until DATA command
deny senders = ${if exists{CONFDIR/sender-blacklist-envelope-from}\
{CONFDIR/sender-blacklist-envelope-from}\
{}}
message = Rejected RCPT <$local_part@$domain>: Unrouteable address
log_message = Blacklisted - $sender_address
delay = 10s

And here is the excerpt from the file sender-blacklist-envelope-from that
should have blocked the spam I saw.

\N^quicken.loans.*@.*\N
*\N^mutual.of.omaha.*@.*\N*
\N^meetasianlady.*@.*\N

The goal is to block a highly-sophisticated spammer that uses a bunch of
legit email servers (they all have valid DKIM, reverse DNS, etc.) by
blocking keywords in their sender email addresses' local parts. For
instance, the Mutual of Omaha spam always start with:

mutual-of-omaha
mutual_of_omaha
mutual.of.omaha

The spam that failed to match looked like this excerpt of the debug output.
Relevant portions are bolded:

*23314 processing "deny" (/var/lib/exim4/config.autogenerated 316)*
*23314 check senders = ${if
exists{/etc/exim4/sender-blacklist-envelope-from}{/etc/exim4/sender-blacklist-envelope-from}{}}*

...

23314 address match test: subject=mutual_of_omaha_mortgage-me=
domain.com@b92vsh.com pattern=\N^quickenloans.*@.*\N
23314 address match test: subject=mutual_of_omaha_mortgage-me=
domain.com@b92vsh.com pattern=\N^quicken.loans.*@.*\N
*23314 address match test:
subject=mutual_of_omaha_mortgage-me=domain.com@b92vsh.com
<domain.com@b92vsh.com> pattern=\N^mutual.of.omaha.*@.*\N*

...

23314 address match test: subject=mutual_of_omaha_mortgage-me=
domain.com@b92vsh.com pattern=\N^empire.today.*@.*\N
23314 mutual_of_omaha_mortgage-me=domain.com@b92vsh.com in
"/etc/exim4/sender-blacklist-envelope-from"? *no (end of list)*
*23314 deny: condition test failed in ACL "acl_check_rcpt"*

I can't figure out why it fails to match. The regular expression looks
exactly like the documentation says it should look. It starts with a
circumflex, is surrounded by \N to deal with string expansion, has a local
part and a domain part with an @ in between, and is a normal Perl
Compatible Regular Expression (PCRE). Periods should match any character,
so mutual_of_omaha should have been matched. I have tried this pattern too
and it also failed to match:

\N^mutual_of_omaha.*@.*\N

Can anyone help me get this deny senders to match using regular expressions?

Thanks in advance for any help you can provide!
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: deny senders in RCPT ACL regular expression and wildcards won't match for some reason [ In reply to ]
I managed to figure it out.

It was the \N. When I was looking at the debug logs, I noticed that for
another blacklist that used simple wildcard domain blocks, each line of the
log said

no (end of list)

at the end of each line like so:

03:13:13 50032 address match test:
subject=livewire-insurance-start-saving-me=domain.com@foltertankit.com
pattern=*a3a99l3y.com
03:13:13 50032 foltertankit.com in "*a3a99l3y.com"? no (end of list)
03:13:13 50032 address match test:
subject=livewire-insurance-start-saving-me=domain.com@foltertankit.com
pattern=*afd7971a.com
03:13:13 50032 foltertankit.com in "*afd7971a.com"? no (end of list)
03:13:13 50032 address match test:
subject=livewire-insurance-start-saving-me=domain.com@foltertankit.com
pattern=*laxdva.com
03:13:13 50032 foltertankit.com in "*laxdva.com"? no (end of list)

But with my problematic blacklist file full of regular expressions (one
regular expression per line), there was only one "no (end of list)" at the
very end of the last line. Thus, the \N protection against string expansion
was somehow causing exim to treat the entire file as a giant pattern, even
though the log deceptively made it look like each line was being matched
one line at a time like so:

03:13:13 50032 address match test:
subject=livewire-insurance-start-saving-me=domain.com@foltertankit.com
pattern=\N^affiliate.renewal.*@.*\N
03:13:13 50032 address match test:
subject=livewire-insurance-start-saving-me=domain.com@foltertankit.com
pattern=\N^toprated.wines.*@.*\N
03:13:13 50032 address match test:
subject=livewire-insurance-start-saving-me=domain.com@foltertankit.com
pattern=\N^renewal.by.andersen.*@.*\N
03:13:13 50032 address match test:
subject=livewire-insurance-start-saving-me=domain.com@foltertankit.com
pattern=\N^empire.today.*@.*\N
03:13:13 50032 livewire-insurance-start-saving-me=
domain.com@foltertankit.com in "/etc/exim4/sender-blacklist-envelope-from"?
no (end of list)

When I removed all the \N instances from the entire blacklist, every
regular expression started working, each line that didn't match had a "no
(end of list)" at the end of it, and I am now successfully blocking this
network of sophisticated spammers.

I hope this helps someone else.
--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: deny senders in RCPT ACL regular expression and wildcards won't match for some reason [ In reply to ]
On 01/06/2021 10:48, slp tees via Exim-users wrote:
> deny senders = ${if exists{CONFDIR/sender-blacklist-envelope-from}\
> {CONFDIR/sender-blacklist-envelope-from}\
> {}}

> And here is the excerpt from the file sender-blacklist-envelope-from that
> should have blocked the spam I saw.
>
> \N^quicken.loans.*@.*\N
> *\N^mutual.of.omaha.*@.*\N*
> \N^meetasianlady.*@.*\N

http://exim.org/exim-html-current/doc/html/spec_html/ch-domain_host_address_and_local_part_lists.html#SECTfilnamlis

"no expansion of the data from the file takes place"

You don't want those \N pairs; they stay as part of the result data.--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: deny senders in RCPT ACL regular expression and wildcards won't match for some reason [ In reply to ]
On 2021-06-01 slp tees via Exim-users <exim-users@exim.org> wrote:
[...]
> acl_check_rcpt:

> ### Deny if sender explicitly in blacklist
> # $recipients is not available until DATA command
> deny senders = ${if exists{CONFDIR/sender-blacklist-envelope-from}\
> {CONFDIR/sender-blacklist-envelope-from}\
> {}}
> message = Rejected RCPT <$local_part@$domain>: Unrouteable address
> log_message = Blacklisted - $sender_address
> delay = 10s

> And here is the excerpt from the file sender-blacklist-envelope-from that
> should have blocked the spam I saw.

> \N^quicken.loans.*@.*\N
> *\N^mutual.of.omaha.*@.*\N*
> \N^meetasianlady.*@.*\N
[...]

Hello,

The lines in the file are not expanded, so quoting with \N should not be
necessary. The second line ("*\N^mutual.of.omaha.*@.*\N*") does not
start with ^ but with "*" and is therefore not interpreted as a regular
expression.

This would probably work:
^mutual.of.omaha.*@.*

cu Andreas
--
`What a good friend you are to him, Dr. Maturin. His other friends are
so grateful to you.'
`I sew his ears on from time to time, sure'

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/