Mailing List Archive

Sieve filter not working properly anymore, how to debug?
Hello,

I'm setting up a new mail server with Exim and try to get the Sieve
filter working as it does on another server. But it doesn't. And I don't
know why.

Here's a sieve script for demo@test.local:

if header :contains ["Subject"] "Newsletter" {
redirect "demo2@test.local";
}

I'm sending a message from demo3@test.local to demo@test.local with the
subject "Newsletter". The message is delivered in the inbox of the
original recipient and not forwarded.

Here's another sieve script:

redirect "demo2@test.local";
stop;

The same message is now both delivered in the original inbox and
forwarded to demo2. It looks like this time the filter action was at
least considered, but the implicit keep should not be effective, but is.

The Exim version is 4.93 on Ubuntu Linux 20.04.

This is a part of my Exim config. All addresses with maildirs and sieve
data is stored in a database.

> SQL_FILTER = \
> select E'# Sieve filter\n' || sieve_filter \
> from mail_entries \
> where local = '${quote_pgsql:$local_part}' and domain = '${quote_pgsql:$domain}' and maildir <> '' and sieve_filter <> ''
>
> begin routers
>
> virtual_user_filter:
> driver = redirect
> allow_fail
> allow_defer
> allow_filter
> user = Debian-exim
> data = ${lookup pgsql{SQL_FILTER}{$value}}
> address_data = ${lookup pgsql{SQL_QUOTA}{$value}fail}
> file_transport = address_directory
>
> begin transports
>
> address_directory:
> driver = appendfile
> directory = ${if eq{$address_file}{inbox} \
> {${lookup pgsql{SQL_MAILDIR}{$value}fail}} \
> {${if eq{${substr_0_1:$address_file}}{/} \
> {$address_file} \
> {${lookup pgsql{SQL_MAILDIR}{$value}fail}/.$address_file} \
> }} \
> }
> user = mail
> maildir_format
> delivery_date_add
> envelope_to_add
> return_path_add

The mainlog doesn't show more than the in and out events. There are
especially no errors. (I can have errors when I leave out the Sieve
filter comment prefix or use an invalid database query or break the
database permissions. I've seen all of this but they're fixed and the
messages are gone.)

How can I debug or fix the Sieve filter in Exim? I thought that the
config I have here is about the same as what's already working with the
Exim from Ubuntu 16.04.

-Yves

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Sieve filter not working properly anymore, how to debug? [ In reply to ]
On 07/10/2020 14:14, Yves Goergen via Exim-users wrote:
> How can I debug [...] Sieve filter in Exim? I thought that the

Command line "-d" facilities. Depending how you are submitting
a test message, you may need to run your daemon in foreground
mode.

http://exim.org/exim-html-current/doc/html/spec_html/ch-the_exim_command_line.html#SECID39

--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Sieve filter not working properly anymore, how to debug? [ In reply to ]
On Wed, 7 Oct 2020, Yves Goergen via Exim-users wrote:

> Hello,
>
> I'm setting up a new mail server with Exim and try to get the Sieve filter
> working as it does on another server. But it doesn't. And I don't know why.

> The Exim version is 4.93 on Ubuntu Linux 20.04.
>
> This is a part of my Exim config. All addresses with maildirs and sieve data
> is stored in a database.
>
>> SQL_FILTER = \
>> select E'# Sieve filter\n' || sieve_filter \
>> from mail_entries \
>> where local = '${quote_pgsql:$local_part}' and domain =
>> '${quote_pgsql:$domain}' and maildir <> '' and sieve_filter <> ''


> The mainlog doesn't show more than the in and out events. There are
> especially no errors. (I can have errors when I leave out the Sieve filter
> comment prefix or use an invalid database query or break the database
> permissions. I've seen all of this but they're fixed and the messages are
> gone.)
>
> How can I debug or fix the Sieve filter in Exim? I thought that the config I
> have here is about the same as what's already working with the Exim from
> Ubuntu 16.04.

A new concept "Tainting" was introduced to exim between Ubuntu 16.04 and 20.04.
quote_pgsql may untaint $local_part (I am not familiar with quote_pgsql),
but in many configurations $local_part causes taintedness and failures;
the fix in some cases is to replace it with $local_part_data *and* ensure
that the router does check_local_user.

--
Andrew C. Aitchison Kendal, UK
andrew@aitchison.me.uk

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Sieve filter not working properly anymore, how to debug? [ In reply to ]
I don't know what "tainting" means and an English dictionary lookup
didn't make it more plausible. But as you're referring to the
'quote_pgsql' function I guess you mean escaping of SQL query
parameters. This certainly does not cause issues here because the local
part is "demo" or "demo2" in this case, and this concept certainly isn't
new either (I used quote_mysql in my Ubuntu 16.04 setup) and absolutely
required since the invention of SQL queries.

Also, modifying the Sieve text does have an effect and at least once
also forwards the message to the address specified in the script. So the
filter must be accessible. It's just not interpreted by Exim in the
expected way.

-Yves


-------- Ursprüngliche Nachricht --------
Von: Andrew C Aitchison via Exim-users <exim-users@exim.org>
Gesendet: Mittwoch, 7. Oktober 2020, 16:02 MESZ
Betreff: [exim] Sieve filter not working properly anymore, how to debug?

On Wed, 7 Oct 2020, Yves Goergen via Exim-users wrote:

Hello,

I'm setting up a new mail server with Exim and try to get the Sieve filter
working as it does on another server. But it doesn't. And I don't know why.

The Exim version is 4.93 on Ubuntu Linux 20.04.

This is a part of my Exim config. All addresses with maildirs and sieve data
is stored in a database.

SQL_FILTER = \
select E'# Sieve filter\n' || sieve_filter \
from mail_entries \
where local = '${quote_pgsql:$local_part}' and domain =
'${quote_pgsql:$domain}' and maildir <> '' and sieve_filter <> ''


The mainlog doesn't show more than the in and out events. There are
especially no errors. (I can have errors when I leave out the Sieve filter
comment prefix or use an invalid database query or break the database
permissions. I've seen all of this but they're fixed and the messages are
gone.)

How can I debug or fix the Sieve filter in Exim? I thought that the config I
have here is about the same as what's already working with the Exim from
Ubuntu 16.04.

A new concept "Tainting" was introduced to exim between Ubuntu 16.04 and
20.04.
quote_pgsql may untaint $local_part (I am not familiar with quote_pgsql),
but in many configurations $local_part causes taintedness and failures;
the fix in some cases is to replace it with $local_part_data *and* ensure
that the router does check_local_user.



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Sieve filter not working properly anymore, how to debug? [ In reply to ]
On 07/10/2020 15:56, Yves Goergen via Exim-users wrote:
> I don't know what "tainting" means

http://exim.org/exim-html-current/doc/html/spec_html/ch-string_expansions.html

Third paragraph.

--
Cheers,
Jeremy

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Sieve filter not working properly anymore, how to debug? [ In reply to ]
On 2020-10-07 15:14, Yves Goergen wrote:

> I'm setting up a new mail server with Exim and try to get the Sieve
> filter working as it does on another server. But it doesn't. And I
> don't know why.
>
> Here's a sieve script for demo@test.local:
>
> if header :contains ["Subject"] "Newsletter" {
> redirect "demo2@test.local";
> }
>
> I'm sending a message from demo3@test.local to demo@test.local with
> the subject "Newsletter". The message is delivered in the inbox of the
> original recipient and not forwarded.
>
> Here's another sieve script:
>
> redirect "demo2@test.local";
> stop;
>
> The same message is now both delivered in the original inbox and
> forwarded to demo2. It looks like this time the filter action was at
> least considered, but the implicit keep should not be effective, but
> is.

I find Yves' questions important, and I think we should not redirect
this thread into another long discussion of tainting.

--
Ian

--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/
Re: Sieve filter not working properly anymore, how to debug? [ In reply to ]
Thanks for the pointer. I also found other interesting command line
options like the filter test. It also needed the -v or -d option
together with -bf to actually see the promised result. And that looked
all good.

So I went on and tried to test the filter loaded the regular way from
the database with the -bm option. Together with -v I found the message

> Sieve error: syntax error in line 2

somwhere in the terminal window.

When I commented out the second line (after the comment in the first
line), the reported line number changed to 3. Somehow that reminded me
of line endings. The database is fed from a web form entered in a
browser on Windows, processed by an ASP.NET Core application running on
Linux. I added code to filter out any \r characters, and that actually
did the trick. Multiline web forms seem to have CRLF line endings, and
the Exim documentation says that it wants LF-only filters.

So this is resolved, the filter text now contained the wrong line
endings. The relevant change lies in my config tooling.

As a small wish: Those Sieve syntax error messages are a bit hard to
get. It would be nice if they could be included in the regular log
files. That would make debugging Sieve filters a lot easier. There
doesn't seem to be a log selector for that.

-Yves


-------- Ursprüngliche Nachricht --------
Von: Jeremy Harris via Exim-users <exim-users@exim.org>
Gesendet: Mittwoch, 7. Oktober 2020, 15:45 MESZ
Betreff: [exim] Sieve filter not working properly anymore, how to debug?

On 07/10/2020 14:14, Yves Goergen via Exim-users wrote:
How can I debug [...] Sieve filter in Exim? I thought that the

Command line "-d" facilities. Depending how you are submitting
a test message, you may need to run your daemon in foreground
mode.

http://exim.org/exim-html-current/doc/html/spec_html/ch-the_exim_command_line.html#SECID39



--
## List details at https://lists.exim.org/mailman/listinfo/exim-users
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/