Mailing List Archive: [Bug 428] Greatly expand ratelimit features

[Bug 428] Greatly expand ratelimit features

Jul 29, 2007, 12:15 PM

Post #1 of 5 (980 views)

------- You are receiving this mail because: -------
You are the QA contact for the bug.

http://bugs.exim.org/show_bug.cgi?id=428

Dave Lugo <dlugo@etherboy.com> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |dlugo@etherboy.com

--- Comment #11 from Dave Lugo <dlugo@etherboy.com> 2007-07-29 20:15:30 ---
As suggested by Marc Perkel, I'm looking at using the ratelimit
feature for automated whitelisting.

I've noticed some odd behavior at odds with the docs and what
I'm trying to do.

(btw - I've verified this both pre and post patch- example below is with
the patch)

When a user at one of my provisioned domains sends an email
outbound, this is called in the rcpt acl:

ratelimit = 0 / 52w / per_cmd / strict / noupdate
${lc:$sender_address}:${lc:${extract{1}{+}{$local_part}}}@${lc:$domain}

(it will likely wrap above, in my config it's all on one line).

When that fires, I see this (as an example):

root@spot> /var/www/sbin/exim_dumpdb /var/www/spool ratelimit
29-Jul-2007 14:53:54.151117 rate: 0.000 key:
52w/per_cmd/strict/dlugo@etherboy.com:david.lugo@acs-inc.com

Shouldn't that be a rate of '1'? (at least, that's what I want it to be). So
I looked in spec.txt a bit, and saw this re per_cmd:

"The per_cmd option causes Exim to recompute the rate every time the condition
is processed. "

I then tried calling ratelimit multiple times (first, rm'ing the current
ratelimit db). A
single message traversing this:

accept logwrite = Sender rate pre1: $sender_rate
ratelimit = 0 / 52w / per_mail / strict / noupdate /
${lc:$sender_address}:${lc:${extract{1}{+}{$local_part}}}@${lc:$domain}
logwrite = Sender rate pre2: $sender_rate
ratelimit = 0 / 52w / per_mail / strict /
${lc:$sender_address}:${lc:${extract{1}{+}{$local_part}}}@${lc:$domain}
logwrite = Sender rate pre3: $sender_rate
ratelimit = 0 / 52w / per_mail / strict /
${lc:$sender_address}:${lc:${extract{1}{+}{$local_part}}}@${lc:$domain}

(sorry about the linewraps)

... just logs this:

Jul 29 14:53:54 spot exim[32422]: Sender rate pre1:
Jul 29 14:53:54 spot exim[32422]: Sender rate pre2: 0.0
Jul 29 14:53:54 spot exim[32422]: Sender rate pre3: 0.0

Even if I specify the same rcpt twice in the same mail sent by my
client (and verified by the logs here), the initial rate recorded
appears to remain 0.0. The current rate is incremented only on
a second (and successive) email(s) sent with the same from/to pair.

So, to autoadd a whitelist entry, the local user has to send two separate
emails. (seems silly to me :)

If the local user has sent only one email, using noupdate on return mail
will just see '0.0' as the rate. I suppose I could test for the rate being
blank, but I wonder if it's better if the initial rate recorded should just
be 1.0. Another solution would be having each successive call of per_cmd
for the same email actually increment the counter. (at least, that's what
the docs seem to suggest... :)

Thanks,

Dave

--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[Bug 428] Greatly expand ratelimit features [ In reply to ]

dlugo at etherboy

Jul 29, 2007, 12:18 PM

Post #2 of 5 (955 views)

Permalink

------- You are receiving this mail because: -------
You are the QA contact for the bug.

http://bugs.exim.org/show_bug.cgi?id=428

--- Comment #12 from Dave Lugo <dlugo@etherboy.com> 2007-07-29 20:18:24 ---
correction to my last post:

> When a user at one of my provisioned domains sends an email
> outbound, this is called in the rcpt acl:
>
> ratelimit = 0 / 52w / per_cmd / strict / noupdate
> ${lc:$sender_address}:${lc:${extract{1}{+}{$local_part}}}@${lc:$domain}
>

s//\noupdate//

--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[Bug 428] Greatly expand ratelimit features [ In reply to ]

dlugo at etherboy

Jul 29, 2007, 2:26 PM

Post #3 of 5 (958 views)

Permalink

------- You are receiving this mail because: -------
You are the QA contact for the bug.

http://bugs.exim.org/show_bug.cgi?id=428

--- Comment #13 from Dave Lugo <dlugo@etherboy.com> 2007-07-29 22:26:04 ---
Another followup...

I changed 0.0 to 1.0 in this (~line 2275 in acl.c):

if (dbd == NULL)
{
HDEBUG(D_acl) debug_printf("ratelimit initializing new key's data\n");
dbd = store_get(sizeof(dbdata_ratelimit));
dbd->time_stamp = tv.tv_sec;
dbd->time_usec = tv.tv_usec;
dbd->rate = 1.0;
^^^^^^^^^^^^^^
}

... which works nicely to add a new record with a count of '1'.

Problem is, when I do this to see if a WL entry should fire:

ratelimit = 1.0 / 52w / per_cmd / strict / noupdate /
${lc:$local_part}@${lc:$domain}:${lc:$sender_address}

... it always returns true, even for stuff that isn't in the ratelimit db.

I guess that ratelimit uses the default 1.0 for adding a null record, even when
doing noupdate. Shouldn't ratelimit return 0 if 0.0 or null if the record
isn't found
*and* noupdate is being used?

(my apologies in advance if I'm confused)

Thanks,

Dave

--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[Bug 428] Greatly expand ratelimit features [ In reply to ]

dot at dotat

Jul 30, 2007, 11:22 AM

Post #4 of 5 (967 views)

Permalink

------- You are receiving this mail because: -------
You are the QA contact for the bug.

http://bugs.exim.org/show_bug.cgi?id=428

Tony Finch <dot@dotat.at> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |dot@dotat.at

--- Comment #14 from Tony Finch <dot@dotat.at> 2007-07-30 19:22:28 ---
A rate of 0 is correct for the first message. The essence of the computation
is:

interval = this_time - prev_time;
i_over_p = interval / period;
a = exp(-i_over_p);
new_rate = (1 - a) / i_over_p + a * old_rate;

For the first message, the interval is effectively infinite, so

interval = inf;
i_over_p = inf / period = inf;
a = exp(-i_over_p) = exp(-inf) = 0;
new_rate = (1 - a) / i_over_p + a * old_rate
= 1 / inf + 0 * old_rate
= 0;

This has the advantage that you can set the rate limit arbitrarily low.

I'm not sure whether or not the noupdate in your previous posts is a red
herring. The debugging output you posted is consistent with a stray noupdate. I
can't understand what you are trying to suggest in your last post.

--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##

[Bug 428] Greatly expand ratelimit features [ In reply to ]

dlugo at etherboy

Jul 30, 2007, 11:28 AM

Post #5 of 5 (952 views)

Permalink

------- You are receiving this mail because: -------
You are the QA contact for the bug.

http://bugs.exim.org/show_bug.cgi?id=428

--- Comment #15 from Dave Lugo <dlugo@etherboy.com> 2007-07-30 19:28:02 ---
On Mon, 30 Jul 2007, Tony Finch wrote:
>
> --- Comment #14 from Tony Finch <dot@dotat.at> 2007-07-30 19:22:28 ---
> A rate of 0 is correct for the first message. The essence of the computation
> is:
>
> interval = this_time - prev_time;
> i_over_p = interval / period;
> a = exp(-i_over_p);
> new_rate = (1 - a) / i_over_p + a * old_rate;
>
> For the first message, the interval is effectively infinite, so
>
> interval = inf;
> i_over_p = inf / period = inf;
> a = exp(-i_over_p) = exp(-inf) = 0;
> new_rate = (1 - a) / i_over_p + a * old_rate
> = 1 / inf + 0 * old_rate
> = 0;
>
> This has the advantage that you can set the rate limit arbitrarily low.
>
> I'm not sure whether or not the noupdate in your previous posts is a red
> herring. The debugging output you posted is consistent with a stray noupdate. I
> can't understand what you are trying to suggest in your last post.
>

I think most of the problem was me confusing myself, and a bad
paste into one of the comments in this bug. <sorry>

I ended up getting it working fine (see my post on exim-dev).

The current noupdate patch works great for what I want to do.

Best regards,

Dave

--
Configure bugmail: http://bugs.exim.org/userprefs.cgi?tab=email

--
## List details at http://lists.exim.org/mailman/listinfo/exim-dev Exim details at http://www.exim.org/ ##