Mailing List Archive

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Monday, March 11 at 09:50 PM, quoth Vahid Moghaddasi:
> I am using a Perl script for DKIM signing of our mass e-mail
> broadcast which of course take more CPU time than I would want to,
> and now there is DMARC http://www.dmarc.org/ authentication method
> that we need to use for many dmains. My question is, how do I
> incorporate DMARC into qmail? Also, what DKIM program is preffered
> to use with qmail. Thanks,

DMARC isn't a new authentication method. If you look at the "Overview"
page of the site you sent, at the bottom, they have the 5 steps
necessary to deploy DMARC. The first two are to ensure you have DKIM
and SPF working (which you've already done). The third is to publish a
DNS record (which has nothing to do with qmail). The fourth is to
analyze the data reports you get back from recipients as part of the
DMARC program (which has nothing to do with qmail). And the fifth is
to update your DMARC policy flags in DNS as necessary (which has
nothing to do with qmail). So, unless you know something I don't, it
doesn't look like you need to do anything with qmail to employ DMARC.

Now, as far as what DKIM program is preferred to use with qmail...
qmail isn't quite so centralized as all that. There are several
different options, and the preference over which to use is largely up
to the fellow setting it up (i.e. you). I wrote one wrapper script in
perl (maybe it's the one you're using), and I believe there are
others. It's possible that using a patch (such as the ones from
Brandon Turner or Manvendra Bhangui or Mihai Secasiu) will provide you
better performance than a Perl-based wrapper, but, in my opinion, the
overhead of using a perl script is probably insignificant: the fact
that you're computing a cryptographic hash of *every* message is the
real CPU eater. However, CPU cycles are pretty cheap these days,
especially compared to I/O (i.e. the cost of sending the email out
over the network). I'm curious what problems you've experienced as a
result of the CPU overhead of signing your outbound messages.

~Kyle
- --
Our lives begin to end the day we become silent about things that
matter.
-- Martin Luther King Jr.
-----BEGIN PGP SIGNATURE-----
Comment: Thank you for using encryption!

iQIcBAEBCAAGBQJRPynQAAoJECuveozR/AWe0d0P/2LMCKGAbe3sIQanx4taj9rz
afVQ2+EqMrCT7QxDnKq/gmhvBTtO3i6Jr1jEegUPDICEtKIdjHO1xAyTY5l6dd/W
G2BwZ060MKnq3q+Ua3G1tBKExxJDhzZYqMHaW0JTdWlP0aG0mfBezNAlZb+NMsFi
qX7x+ZmgAFOZpFNTET4MjzryixLvVtaRxVu/eDluLjaJyk7GT5rEBVseWRXovNfD
4IEhU0Bc9UCr3JXh/aXGhgEYFNhspoq5AQEFNQcR9zS4Xf8t0PCITJ3H7BnMmTHN
AwdECrdG7vBFcrHBxhfrwkSIfCykct43fOaq5zigMkdeBTIELBYgh8Yow4AWvp/k
O9lJnzgMre+0Fp+mPFDRKRi97miTPt+zbaE9uEz+65GteINf3zGnbY5E5/3HKwUi
vhxK60m6bwhVJefA/Roc4WsXJeFIZY3NnfafN62FtRp0djBqb6pvfrWt3mYcQmLK
CDRv/iLalCZUfBE7rHgVo1afajCiSM8g+Ib7UNL5c8lt85hX5uWpqlQj7yG7c5rg
JbRmlfRdh693JeHzdUh4ckp7hBbIsCYaWS1HLOc12tSp7gJMmjueYg292mCF1Wmu
l5HwF4cp9ApBTMSHYH1d6SWCnrOhkp2yjBCOh2in4Odz/MVvo8miUYJfRqclCfS8
kNsUdjhPjHKOes86s8Xp
=8HSh
-----END PGP SIGNATURE-----

>nothing to do with qmail). So, unless you know something I don't, it
>doesn't look like you need to do anything with qmail to employ DMARC.

If you want to send (as opposed to receive) DMARC reports, it's a lot
of work since you need to log DKIM and SPF results for incoming mail
in a database, then generate reports from that.

I'm slowly looking at putting SPF and DKIM checking modules into
mailfront (which is a lot more tractable than a heavily hacked
qmail-smtpd) but don't hold your breath,

>It's possible that using a patch (such as the ones from
>Brandon Turner or Manvendra Bhangui or Mihai Secasiu) will provide you
>better performance than a Perl-based wrapper, but, in my opinion, the
>overhead of using a perl script is probably insignificant:

Agreed. I've been using a perl shim in front of qmail-queue for a
long time, and it's been plenty fast.

On Tue, Mar 12, 2013 at 9:12 AM, Kyle Wheeler <kyle-qmail@memoryhole.net>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> DMARC isn't a new authentication method. If you look at the "Overview"
> page of the site you sent, at the bottom, they have the 5 steps
> necessary to deploy DMARC. The first two are to ensure you have DKIM
> and SPF working (which you've already done). The third is to publish a
> DNS record (which has nothing to do with qmail). The fourth is to
> analyze the data reports you get back from recipients as part of the
> DMARC program (which has nothing to do with qmail). And the fifth is
> to update your DMARC policy flags in DNS as necessary (which has
> nothing to do with qmail). So, unless you know something I don't, it
> doesn't look like you need to do anything with qmail to employ DMARC.
>

Great, will do some more reading and see what I have to put in DNS etc..


>
> Now, as far as what DKIM program is preferred to use with qmail...
> qmail isn't quite so centralized as all that. There are several
> different options, and the preference over which to use is largely up
> to the fellow setting it up (i.e. you). I wrote one wrapper script in
> perl (maybe it's the one you're using), and I believe there are
> others. It's possible that using a patch (such as the ones from
> Brandon Turner or Manvendra Bhangui or Mihai Secasiu) will provide you
> better performance than a Perl-based wrapper, but, in my opinion, the
> overhead of using a perl script is probably insignificant: the fact
> that you're computing a cryptographic hash of *every* message is the
> real CPU eater. However, CPU cycles are pretty cheap these days,
> especially compared to I/O (i.e. the cost of sending the email out
> over the network). I'm curious what problems you've experienced as a
> result of the CPU overhead of signing your outbound messages.
>
> I am not experiencing much problem at the moment since we are only
required by AOL to use DKIM but I am worried that maybe in the future, if
every domain required DKIM we might have problem.
All the e-mail destine to domains which require DKIM from us (AOL for now)
are being send to one server and that server signs the e-mails with DKIM
and sends them out. There is no incoming e-mail, all out going. During the
pick e-mail activity (end of month or so) the CPU spikes high on that one
server that is sending mail to AOL.
During pick e-mail activity, we easily send more that 1 million e-mails per
day, of course not all are AOL but a good 20K are.

Would you advise that we sign every e-mail to every domain with DKIM
instead of having dedicated servers to deliver e-mail only to domain who
require DKIM?
Thank you Kyle.

Manvendra, thank you, I will try your C program but as I understand, it
needs to be patched into qmail and compiled rather than being a wrapper
similar to the Perl script.

John, We only send mail out, nothing comes in, except bounces.
Thank you all.

On Mar 12, 2013, at 9:46 PM, Vahid Moghaddasi <vahid.moghaddasi@gmail.com> wrote:

> Would you advise that we sign every e-mail to every domain with DKIM
> instead of having dedicated servers to deliver e-mail only to domain who
> require DKIM?

Is your DKIM signing script a front-end for qmail-queue, or qmail-remote? I've seen both techniques. If you're using mailing-list software, like ezmlm, that queues a single message with many recipients, you would save CPU cycles if you sign the single copy going into qmail-queue instead of each individual message sent by qmail-remote.
--
Matt Simpson
Tatertown, KY

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tuesday, March 12 at 09:59 PM, quoth Matt Simpson:
>
> On Mar 12, 2013, at 9:46 PM, Vahid Moghaddasi <vahid.moghaddasi@gmail.com> wrote:
>
>> Would you advise that we sign every e-mail to every domain with DKIM
>> instead of having dedicated servers to deliver e-mail only to domain who
>> require DKIM?
>
> Is your DKIM signing script a front-end for qmail-queue, or
> qmail-remote? I've seen both techniques. If you're using
> mailing-list software, like ezmlm, that queues a single message
> with many recipients, you would save CPU cycles if you sign the
> single copy going into qmail-queue instead of each individual
> message sent by qmail-remote.

Many mailing list software packages, including ezmlm, also have
options to "customize" the outbound email message per recipient. This
can be invaluable for several reasons, for example, when dealing with
a spam report that provides the message marked as spam (e.g. synacore
reports) to know which recipient doesn't want to be on your mailing
list any more. Whatever the reason, if customized email is important,
signing before it gets to the queue doesn't help. :(

~Kyle
- --
Life is too important to be taken seriously.
-- Oscar Wilde
-----BEGIN PGP SIGNATURE-----
Comment: Thank you for using encryption!

iQIcBAEBCAAGBQJRP+iQAAoJECuveozR/AWe4Z0QAKhUcEs0Fd3mKAHeGx2an2qB
wYiNWvRPMVJN343GmzmxhEWnidZX0sXX4T5JBGPjlWkgPT03Yoo6iCpZacog+2Vv
LsI0M/BLQBN0JVo4NJ4qTP8Z9cEXFy8q5IBXCb0lPBYNs9FjDQiP1rDNRKc4lnvn
PUVgZFtH4/JfJDKY3NFSnNeraSsPl10rXsT+DCN5EhBbpX70mw/gd5Mo+N5ioumU
7QXtIMw7O7piDShBDaUbipz6wGJHrGldFz9aYGDD67gkYOgSqnolchJ6kpvy7vId
lRCTzWeeszIL35FEkuxX6tBjOsAhYEPy4zJ6sRJSdlA6ZruTQiSKgnh/O3YSX70L
nbM3aHZgXY5rWqhRUbscv9FSK0Yb35nCja2jaRfBJA30CVQLgRR3LKoQabejqkDG
AeBV9onw6uPcWClDrFujn56TOFkvckOGBuxtjExG7XRrDxnHoYSwvuqjaUvKEJa9
VVCRXS0bSsSGtE2FIds3Qw7SUPiecjG1KyVWcyGK28oziOmZiSausgKn4fe7x/Pq
DMqzmTnx0CH8WfFj92SCuqK4EGAnIi3ojsg4ijEHyjn9eVZUQI7VJj0UHY9dx7ok
b3BEWc9S10p0Q9J/hib3e5ioK4xOYkW5LsZy275R3QVDqXegkGutwcFXYiWL12XH
aRONqzcfFv1Jmt2+n4Z7
=ascc
-----END PGP SIGNATURE-----

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

On Tuesday, March 12 at 09:46 PM, quoth Vahid Moghaddasi:
> I am not experiencing much problem at the moment since we are only
> required by AOL to use DKIM but I am worried that maybe in the
> future, if every domain required DKIM we might have problem. All the
> e-mail destine to domains which require DKIM from us (AOL for now)
> are being send to one server and that server signs the e-mails with
> DKIM and sends them out. There is no incoming e-mail, all out going.
> During the pick e-mail activity (end of month or so) the CPU spikes
> high on that one server that is sending mail to AOL. During pick
> e-mail activity, we easily send more that 1 million e-mails per day,
> of course not all are AOL but a good 20K are.

Doing some rough calculations, there are 86400 seconds in a day, so if
you send 1 million emails per day, that's a maximum of 0.0864 seconds
per email, assuming they all had to happen one after the other.
However, on a multi-core machine, you have more flexibility. For
example, on my 6-year-old (or so) 8-core 2.4GHz Xeon, my perl-based
DKIM signer took 0.176 seconds to compute a signature. That means it
can process just over 490,000 emails per day using a single core. If
all eight cores are pegged, my little server can process nearly 4
million messages per day. Granted, half of those cores are
hyperthreaded cores, but even if we halve the performance, my old
server can sign approximately 2 million messages per day. Now,
granted, I'm not leaving any time for the other things that have to be
done (I/O, for example), but you get the idea.

I imagine your server is faster than mine, possibly newer. I'd be
surprised if it choked on 1 million signatures per day. If you had a
balanced approach, with multiple servers sending messages instead of
just one, I don't think you'd have any problem at all. You might have
some higher latencies, especially if you tried to shove all of the
emails through a single server, but it sounds like you already have a
bunch of email processing machines to handle your load.

Of course, your mileage may vary, and that's just rough calculating.

~Kyle
- --
A tyrant must put on the appearance of uncommon devotion to religion.
Subjects are less apprehensive of illegal treatment from a ruler whom
they consider God-fearing and pious. On the other hand, they do less
easily move against him, believing that he has the gods on his side.
-- Aristotle
-----BEGIN PGP SIGNATURE-----
Comment: Thank you for using encryption!

iQIcBAEBCAAGBQJRP+x0AAoJECuveozR/AWeH/wP/1ztY4MDeWnCcr3a/kpebtnK
CW2YkqHLrEKWIV+ESrHVIo1DvX44MH8gPzMtZoOlSDwavG5tHHConfJoNjYueqNj
4Y+r9kC0ZEHvDdOIHKFOHxuwV8ef7OhzLQ3i8FKQHyVCmQpXH6K79YUmfiayg9ob
e3+bMjxAiJE0sRUib6g7SUEOXXzJy3RufJgsXd6idm9wOztlAPtNGEi/MwkcmZEt
UDF0RjeuDbyGOM4rSrTXn+th6HLk0zZZw2vGFb9ZeSj7RjCwZN8oIBoH0xTVKvJU
re1213/LR8hUIBAOGBdwSi6i5BcS8vZKfvFVL+EG3Wtm+1qyfj6P8ztAo87V0PTS
P2ZcNfE0VlaIzfFuufNPvDIEgMh4UymU9ZeWDYsRQFQya81+xAWqoPL3mpP6Ir1e
iR9E1YA0guCaVRiQ7NZ9pUTjqrt6ol0OCSo2rqN7szIdwr6P3fraCD5ODqKrc4gD
6kC1AIRmGqQkFD0xpmt5C/9ns1d3sQ9fDshJZdF5p5k3imL1LrimyWAvuEqkCdjD
d0L4SOiM4XB4pB64HtAVVeETMQiw/2tbL6hCVdFOaa+cj0E2ZaFAxr162WgBs+1G
hJ3sapTVCwFHxlxktqP7uPkjHlFh2gFzK8+G58J63YjE8kIPIF930Ph6xcE6RN13
3jjoNmoZFLCgNus+byCW
=O7mA
-----END PGP SIGNATURE-----

On 13 Mar, 2013, at 9:46 AM, Vahid Moghaddasi <vahid.moghaddasi@gmail.com> wrote:

>
> On Tue, Mar 12, 2013 at 9:12 AM, Kyle Wheeler <kyle-qmail@memoryhole.net> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
> DMARC isn't a new authentication method. If you look at the "Overview"
> page of the site you sent, at the bottom, they have the 5 steps
> necessary to deploy DMARC. The first two are to ensure you have DKIM
> and SPF working (which you've already done). The third is to publish a
> DNS record (which has nothing to do with qmail). The fourth is to
> analyze the data reports you get back from recipients as part of the
> DMARC program (which has nothing to do with qmail). And the fifth is
> to update your DMARC policy flags in DNS as necessary (which has
> nothing to do with qmail). So, unless you know something I don't, it
> doesn't look like you need to do anything with qmail to employ DMARC.
>
> Great, will do some more reading and see what I have to put in DNS etc..
>
>
> Now, as far as what DKIM program is preferred to use with qmail...
> qmail isn't quite so centralized as all that. There are several
> different options, and the preference over which to use is largely up
> to the fellow setting it up (i.e. you). I wrote one wrapper script in
> perl (maybe it's the one you're using), and I believe there are
> others. It's possible that using a patch (such as the ones from
> Brandon Turner or Manvendra Bhangui or Mihai Secasiu) will provide you
> better performance than a Perl-based wrapper, but, in my opinion, the
> overhead of using a perl script is probably insignificant: the fact
> that you're computing a cryptographic hash of *every* message is the
> real CPU eater. However, CPU cycles are pretty cheap these days,
> especially compared to I/O (i.e. the cost of sending the email out
> over the network). I'm curious what problems you've experienced as a
> result of the CPU overhead of signing your outbound messages.
>
> I am not experiencing much problem at the moment since we are only required by AOL to use DKIM but I am worried that maybe in the future, if every domain required DKIM we might have problem.
> All the e-mail destine to domains which require DKIM from us (AOL for now) are being send to one server and that server signs the e-mails with DKIM and sends them out. There is no incoming e-mail, all out going. During the pick e-mail activity (end of month or so) the CPU spikes high on that one server that is sending mail to AOL.
> During pick e-mail activity, we easily send more that 1 million e-mails per day, of course not all are AOL but a good 20K are.
>
> Would you advise that we sign every e-mail to every domain with DKIM instead of having dedicated servers to deliver e-mail only to domain who require DKIM?
> Thank you Kyle.
>
> Manvendra, thank you, I will try your C program but as I understand, it needs to be patched into qmail and compiled rather than being a wrapper similar to the Perl script.
>
> John, We only send mail out, nothing comes in, except bounces.
> Thank you all.
>

I have tried Kyle's wrapper for qmail-remote but the Dell R410 server just become very very sluggish when sending 70,000+ mails per time,. I have no choice but using Brandon's patch instead. (http://www.bltweb.net/qmail/)

Kinglok, Fong

On Mar 12, 2013, at 10:46 PM, Kyle Wheeler <kyle-qmail@memoryhole.net> wrote:

> Many mailing list software packages, including ezmlm, also have
> options to "customize" the outbound email message per recipient. .. Whatever the reason, if customized email is important,
> signing before it gets to the queue doesn't help. :(

That's a good point, and if you customize messages, you might need to sign each customized message instead of the original, depending on the type of cusomization.

In my case, the only customization I do is an additional header (I use ezmlm, and I add a header with the individual unsubscribe address for each user). The signing script I use allows specific headers to be eliminated from signing, so I sign in a qmail-queue frontend and ignore the List-Unsubscribe header. But if you're adding something like a customized message trailer, that won't work and you will have to sign each message in qmail-remote.
--
Matt Simpson
Tatertown, KY

On Tue, Mar 12, 2013 at 11:03 PM, Kyle Wheeler <kyle-qmail@memoryhole.net>wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA256
>
>
> Doing some rough calculations, there are 86400 seconds in a day, so if
> you send 1 million emails per day, that's a maximum of 0.0864 seconds
> per email, assuming they all had to happen one after the other.
> However, on a multi-core machine, you have more flexibility. For
> example, on my 6-year-old (or so) 8-core 2.4GHz Xeon, my perl-based
> DKIM signer took 0.176 seconds to compute a signature. That means it
> can process just over 490,000 emails per day using a single core. If
> all eight cores are pegged, my little server can process nearly 4
> million messages per day. Granted, half of those cores are
> hyperthreaded cores, but even if we halve the performance, my old
> server can sign approximately 2 million messages per day. Now,
> granted, I'm not leaving any time for the other things that have to be
> done (I/O, for example), but you get the idea.
>
> I imagine your server is faster than mine, possibly newer. I'd be
> surprised if it choked on 1 million signatures per day. If you had a
> balanced approach, with multiple servers sending messages instead of
> just one, I don't think you'd have any problem at all. You might have
> some higher latencies, especially if you tried to shove all of the
> emails through a single server, but it sounds like you already have a
> bunch of email processing machines to handle your load.
>
> Of course, your mileage may vary, and that's just rough calculating.
>
My server might be a bit faster, I have a few Oracle/Sun Sparc T4-4
servers, each has 4 cpu and each cpu has 8 core, each core has 8 threads,
total of 256 threads. I am also thinking of porting to HP DL980 but it is
way too expensive for what it offers.

I am using "Manuel Mausz" qmail-remote perl wrapper for DKIM, not sure if
this is the best one but I found this a few years ago which was easy to
implement and has been working fine so far, except some cpu spike sometimes.

Thank you all for the discussion and help.
Vahid.