Mailing List Archive

TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send
There have been two things that I have thought of doing differently in
qmail for a very long time.

1. Avoid scanning todo directory for new email. qmail-queue has the
information of a new email. Can this be transmitted to qmail-send
efficiently without the need for scanning todo directory?

2. Send multiple emails using as few connections as possible for remote
deliveries rather a making new connection for every email to be
delivered to remote hosts. This will be my next project and I
think this will be challenging because too much to deal with tcp/ip
sockets, timeouts, handling reconnection and stuff. Not sure I will
succeed, but I will give it a try.

Eliminating lock/trigger mechanism.

In my experience I have found this to be costly in a very high injection
environment. Recently I have done lot of tests and high injection rates
always has a drastic impact on the qtime value (see zoverall script
from qmailanalog). However with the current hardwares, I haven't
encountered the Silly Qmail Syndrome which was often seen in the
hardware of the 90s.

So I thought of a way to do away with the lock/trigger mechanism and
henceavoid the need of scanning the todo directory for newly injected
mails byusing POSIX message queues in qmail-queue to communicate
with qmail-send.

To achieve this qmail-queue was re-written to use POSIX message queues and
POSIX shared memory to communicate with qmail-send. Here qmail-queue simply
communicates the inode number of the new message to qmail-send and
qmail-send classifies it as local or remote for delivery. The entire todo
scan using opendir(), readdir() is avoided. Additionally, qmail-send was
modified to write the current local and remote concurrency to shared
memory. This enabled a separate process to look at the shared memory
segment and dynamically increase the queue count based on the incoming
injection rate. Having the current concurrency also enables to look at the
queue without having to scan the queue directories (mess, local, remote
and todo). Just by setting the environment variable DYNAMIC_QUEUE to 1
enables point 1 and 2. Initially, this method gave worse performance than
the lock/trigger method, but after ironing out the issues with using IPC,
I'm getting at least 40% improvement. To measure the performance a script
using qmailanalog was written which could fire multiple processes to
inject mail using qmail-inject. I tested various distributions netqmail,
notqmail, s/qmail, indimail-mta, netqmail with exttodo patch. In this
process, I discovered that it is a huge benefit to have a separate processor
for todo directory. netqmail with exttodo patch, s/qmail and indimail-mta
benefits a lot in managing a very low qtime. However the biggest impact on
delivery rate is from the fsync() calls made in qmail-queue.c, qmail-send.c,
qmail-local.c. In tests conducted, fsync made deliveries 8x slower. So if
you have a stable power supply and an OS like Linux (which I haven't seen
crashing for more than 10 years and counting), one can disable fsync.

Having the local and concurrency in shared memory also allows to have a
tmux script named 'qtop' which displays the local and remote concurrency
for each queue in top window and qmail-send logs in the bottom window. This
is done without using disk IO to list files in todo, local, remote subdirs. In
contrast, using qmail-qread drastically reduces injection and delivery
rates when injecting emails at a high rate.

There are few other things that this release achieves

1 dynamically Increase the number of queues based on the injection
rate.

2 Rate limit remote deliveries. This used to be the biggest challenge for
me when dealing with high volume delivery to sites like yahoo, aol and
roadrunner.

3 A much smaller MTA, just one binary that would handle both local, remote
delivery for small board computers. A downside is that it doesn't have the
trust partitioning that having different processes to handle different jobs.

To achieve 1, a program qmonitor was written to look at the concurrency
values stored in the shared memory. When the queue reaches the
concurrency limit, a new queue is created with its own qmail-send process
to process the queue. The current number of queues is also maintained in
shared memory. This allows qmail-queue to know that a new queue is
available for use.

To achieve 2, prioq.c, qmail-send was modified and a new mta was written,
named slowq-send. This special qmail-send looks at queue/ratelimit
directory for a file named 'domain'. Here 'domain' is the destination
domain and it contains a simple expression. e.g. 10/3600 means allow only
10 emails per hour to be delivered to this domain.

To achieve 3, a new MTA was written combining the code of qmail-send,
qmail-lspawn.c, qmail-rspawn.c. This binary known as qmta-send doesn't have
any trust partitioning. But it allows one to run just one systemd service
and one binary to do both local and remote deliveries.

The code has been released as part of indimail-mta 3.0.0 release. It has
been in the works since Oct 2021. It has been difficult and hence I believe
that I will keep on finding bugs as I go along. But sharing this to just
let you folks know about this.

https://github.com/mbhangui/indimail-mta/releases/tag/v3.0.0

Few other things that this release has
1. envdir can use multiple directories. You just need to create
soft links. This allows one to have a directory with global
variables for all supervised services
2. softlimit now has a -q option to set message queue limits.
3. ucspi-tcp, qmail-smtpd, qmail-remote is openssl 3.0.0 ready
4. djb has this comment on line 248 in function comm_canwrite()
in qmail-send.c
/* XXX: could allow a bigger buffer; say 10 recipients */
I tried this by having an environment variable TODO_CHUNK_SIZE
to allow a variable buffer. This does have a positive impact on delivery
rates at the cost of a delayed start.

All of this was benchmarked and documented here
https://github.com/mbhangui/indimail-mta/tree/master/indimail-mta-x/qmail-perf
The google sheet also has the raw data but those sheets are hidden.
Once can enable them using the unhide option in the google sheet menu.
The tests were done on an aging 2012 sony vaio laptop. But this is what I have.
Better hardware could show different results. Also FreeBSD showed
very good inject speeds for netqmail, notqmail, s/qmail but my
FreeBSD is on virtualbox and most probably the fsync() calls do not
actually do fsync(). If someone has a spare box available and
can give login access for a month, I can redo the tests on FreeBSD.

One of the few things I discovered was
1. The conf split doesn't have much impact. Increasing
conf split gives a very marginal improvement on
inject and delivery speeds.
2. dynamically linked qmail-queue reduces injection speeds.
This was a big reason why indimail-mta had the worst
injection rates. I have now made qmail-queue statically
linked in indimail-mta.
3. zfs for the queue directory gives the best performance
if fsync is enabled. ext4 gives the best performance if
fsync is disabled. This is for file creation speed
4. zfs gives the worst performance for file deletion
5. file access times are more or less similar for ext4
and zfs
6. The access times and deletion times doesn't have
high granularity in the tests because I used the
unix time command which gives times only
upto two decimal places.
7. I did the tests on FreeBSD too and got good
speeds. But my FreeBSD is on a virtualbox
and I think the virtual box disk io is cheating on
fsync(). One of these days I will carry out by
installing FreeBSD natively on a separate hard
disk.
8. Having a separate process to scan the todo drastically improves
the qtime. It is instantly visible if you use the zoverall script from
qmailanalog.
9. external todo processor has a remarkable impact on the
local concurrency. The concurrency never reaches high
values with high inject rates.
10. For a large amount of mail injection, it is almost impossible to
beat the speed of netqmail.

--
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Manvendra Bhangui <mbhangui@gmail.com> wrote:
>
> 1. Avoid scanning todo directory for new email. qmail-queue has the
> information of a new email. Can this be transmitted to qmail-send
> efficiently without the need for scanning todo directory?

As you noted, I would have concerns about the weakening (shared memory etc) or
removal (combining multiple qmail programs) of the partitioning between
different parts of qmail.

ext-todo does work quite well. Have you found that with that, on modern
hardware, that the todo scanning is still a bottleneck for the system? I'm
not saying I doubt you, just that I haven't run into it myself lately.

Regarding combining multiple qmail programs to run on tiny embedded systems
and whatnot -- I don't think running a full MTA is a sensible choice there.
Tiny systems should be using nullmailer or ssmtp or some other equivalent to
send all their mail to a smarthost, where the real MTA lives.

> 2. Send multiple emails using as few connections as possible for remote
> deliveries rather a making new connection for every email to be
> delivered to remote hosts. This will be my next project and I
> think this will be challenging because too much to deal with tcp/ip
> sockets, timeouts, handling reconnection and stuff. Not sure I will
> succeed, but I will give it a try.

qmail's architecture makes it difficult - you have to have qmail-send change
to supplying multiple messages and recipients to qmail-remote, which is
significant surgery.

The traditional solution to this for dealing with mail to the big hosts --
Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in a
maildir and periodically run maildirsmtp, which then delivers all accumulated
mail over a single persistent connection. It works quite well. The number of
huge mail providers which really benefit from this type of optimization is
reasonably small, so this is a workable solution.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
On Wed, 18 May 2022 at 21:17, Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:
> As you noted, I would have concerns about the weakening (shared memory etc) or
> removal (combining multiple qmail programs) of the partitioning between
> different parts of qmail.

Yes. That's the concern. The trust partitioning gets eliminated by
combining the functionality of lspawn, rspawn, qmail-send in one
binary. This is the reason I run only on my raspberry pis which run
only on my internal network. They are non-critical just running music
player daemon, librespotify and airplay service.

> ext-todo does work quite well. Have you found that with that, on modern
> hardware, that the todo scanning is still a bottleneck for the system? I'm
> not saying I doubt you, just that I haven't run into it myself lately.

I did mention that. I have never been able to simulate silly qmail
syndrome since ages. But exttodo has a drastic improvement on the
'qtime' parameter that zoverall measures. It was visible in all my
tests. The effect of the exttodo is the mail stays in the queue for
shorter period of time compared to without the exttodo.

> Regarding combining multiple qmail programs to run on tiny embedded systems
> and whatnot -- I don't think running a full MTA is a sensible choice there.
> Tiny systems should be using nullmailer or ssmtp or some other equivalent to
> send all their mail to a smarthost, where the real MTA lives.
>
Indeed. But this was more like a fun project and nothing else. Since
I'm very familiar with the qmail code, I prefer to run this on my
raspberry PI boards that I use for my music listening. The thing is
that I have automated the whole process of updating the code on git
and have the binaries built automatically on openSUSE build service.
My raspberry pi have a script running under supervise which do apt-get
update.

> > 2. Send multiple emails using as few connections as possible for remote
> > deliveries rather a making new connection for every email to be
> > delivered to remote hosts. This will be my next project and I
> > think this will be challenging because too much to deal with tcp/ip
> > sockets, timeouts, handling reconnection and stuff. Not sure I will
> > succeed, but I will give it a try.
>
> qmail's architecture makes it difficult - you have to have qmail-send change
> to supplying multiple messages and recipients to qmail-remote, which is
> significant surgery.

I do visualize this to be very difficult and almost impossible but
still want to give it a try just for the love of programming.

> The traditional solution to this for dealing with mail to the big hosts --
> Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in a
> maildir and periodically run maildirsmtp, which then delivers all accumulated
> mail over a single persistent connection. It works quite well. The number of
> huge mail providers which really benefit from this type of optimization is
> reasonably small, so this is a workable solution.

Let me think of this approach and see if I can come up with a solution
that is simple and achieves the object of rate controlled delivery
which is easily configurable by the end user.

--
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Manvendra Bhangui <mbhangui@gmail.com> wrote:
> On Wed, 18 May 2022 at 21:17, Charles Cazabon
>
> > Have you found that with that, on modern hardware, that the todo scanning
> > is still a bottleneck for the system? I'm not saying I doubt you, just
> > that I haven't run into it myself lately.
>
> I did mention that. I have never been able to simulate silly qmail syndrome
> since ages.

I wasn't specifically asking about that syndrome -- more just that, if you
have a qmail install today, with ext-todo, on reasonable modern hardware, do
you still run into issues where high injection rates result in slow deliveries
(from any cause)?

I'm trying to see if that's why you propose eliminating what you call the lock
/ trigger mechanism -- or whether you just do it "for fun".

> > The traditional solution to this for dealing with mail to the big hosts --
> > Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in
> > a maildir and periodically run maildirsmtp, which then delivers all
> > accumulated mail over a single persistent connection.
>
> Let me think of this approach and see if I can come up with a solution that
> is simple and achieves the object of rate controlled delivery which is
> easily configurable by the end user.

Do you even need to rate-limit if you've only got one connection to Gmail etc
open at a time? maildirsmtp processes messages serially.

If you really wanted to add delays between messages sent with maildirsmtp, it
would be a very simple change to the code. Although I would think Gmail etc
would be pretty agressive about dropping idle incoming SMTP connections, so I
doubt it's necessary or desirable.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
On Thu, 19 May 2022 at 00:34, Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:
>
> Manvendra Bhangui <mbhangui@gmail.com> wrote:
> > On Wed, 18 May 2022 at 21:17, Charles Cazabon
> >
> > > Have you found that with that, on modern hardware, that the todo scanning
> > > is still a bottleneck for the system? I'm not saying I doubt you, just
> > > that I haven't run into it myself lately.
> >
> > I did mention that. I have never been able to simulate silly qmail syndrome
> > since ages.
>
> I wasn't specifically asking about that syndrome -- more just that, if you
> have a qmail install today, with ext-todo, on reasonable modern hardware, do
> you still run into issues where high injection rates result in slow deliveries
> (from any cause)?

Few Users are never happy with whatever speed you give them. The whole
point was to try this and also to improve the delivery rate for a few
folks who use this for marketing emails which sometimes amount to more
than 100k emails per day. I have no means to do a large number of
remote deliveries. So a benchmark was done for local deliveries using
my old laptop. The results were a bit disappointing because it gave
marginal improvement for the effort put in to write the code.

netqmail with exttodo - 31.08 emails/sec with qtime of 0.34 secs
netqmail - 33.17 emails/sec with qtime of 36.56 secs
indimail-mta lock/trigger with fsync - 40.69 emails/sec with qtime of 0.2 secs
indimail-mta ipc with fsync - 40.42 emails/sec with qtime of 0.38 secs
indimail-mta lock/trigger with fsync but delivery in chunks - 44.7
emails/sec with qtime of 9.34 secs
indimail-mta ipc with fsync but delivery in chunks - 71.77 emails/sec
with qtime of 15.44 secs
indimail-mta lock/trigger without fsync - 167.72 emails/sec with qtime
of 7.49 secs
indimail-mta ipc without fsync - 170.14 emails/sec with qtime of 3.65
secs with qtime of 3.65 secs
indimail-mta lock/trigger without fsync, but delivery in chunks -
174.6 emails/sec with qtime of 6.66 secs
iindimail-mta ipc without fsync, but delivery in chunks - 179.11
emails/sec with qtime of 7.14 secs

Regarding delivery in chunks, see djb's comment in
qmail-1.03/qmail-send.c on line 248, comm_canwrite() function. For
marketing campaigns, one of my users uses this method where qmail-send
picks up deliveries in chunks of 5000 emails. delivery doesn't start
till qmail-send has 5000 emails to schedule in one go. The del_start()
function just returns if the comm_buff doesn't have that many emails
within an elapsed period of time (configurable).

> I'm trying to see if that's why you propose eliminating what you call the lock
> / trigger mechanism -- or whether you just do it "for fun".
>

I'm not proposing it. It was something that I felt like doing and I
felt it would reduce the time because scanning of todo directory is
totally avoided. qmail-queue communicates the inode to qmail-send,
avoiding the need for qmail-send to do opendir, readdir of the todo
subdir every few seconds. I thought that it would reduce the time it
takes to finish marketing campaigns, promotions, newsletters and
stuff. In this exercise what I observed is that the fsync() calls in
qmail-queue.c, qmail-send.c, qmail-local.c have the biggest impact.
The fsync calls make qmail crash-proof at the cost of delivery rates
and increased electricity consumption. Only after the benchmarking was
I able to find out that using Posix IPC gives very marginal
improvement at the cost of system complexities managing various ipc
limits (sysctl stuff). The IPC method requires some more work to be
done on my side. I now have the ability to increase the number of
qmail-send processes based on the workload. The thing is that I have
to try this out with a higher workload and that is something pending
at my end due to lack of good hardware at home. One of my users has
promised me access to his hardware, so more tests when I get access to
his hardware.

> > > The traditional solution to this for dealing with mail to the big hosts --
> > > Gmail, Yahoo, Yandex, whatever -- is to put all mail for those domains in
> > > a maildir and periodically run maildirsmtp, which then delivers all
> > > accumulated mail over a single persistent connection.
> >
> > Let me think of this approach and see if I can come up with a solution that
> > is simple and achieves the object of rate controlled delivery which is
> > easily configurable by the end user.
>
> Do you even need to rate-limit if you've only got one connection to Gmail etc
> open at a time? maildirsmtp processes messages serially.
>
> If you really wanted to add delays between messages sent with maildirsmtp, it
> would be a very simple change to the code. Although I would think Gmail etc
> would be pretty agressive about dropping idle incoming SMTP connections, so I
> doubt it's necessary or desirable.
>
IME, gmail does better than Yahoo. Yahoo does aggressive rate limiting
and the concurrency slots get used for mail delivery attempts which
anyway goes into deferral. So with the rate limiting code, qmail-send
just doesn't pick up mails for Yahoo, gmail, etc and uses the
available concurrency slot for delivering mails to hosts that are not
rate limiting. The rate limiting code also records for each domain the
delivery rate. When the delivery rate falls below a number, only then
qmail-send picks up the email and communicates to qmail-rspawn to make
the remote delivery attempt. The whole process of rate limiting is
done by a utility named drate which the end user uses. e.g. the
command

# drate -d yahoo.com -r 100/3600

rate limits emails to yahoo at 100 emails per hour. Having this kind
of scheme allows emails to non-big sites to be delivered without
delays. Without the rate limit code, delivery to all sites gets
delayed by delivery attempts to the big sites.
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Manvendra Bhangui <mbhangui@gmail.com> wrote:
> >
> > Do you even need to rate-limit if you've only got one connection to Gmail
> > etc open at a time? maildirsmtp processes messages serially.
> > doubt it's necessary or desirable.
>
> [...] The rate limiting code also records for each domain the
> delivery rate. When the delivery rate falls below a number, only then
> qmail-send picks up the email and communicates to qmail-rspawn to make
> the remote delivery attempt.

I would be curious to see if Yahoo rate-limits incoming mail with the
maildirsmtp method. With only one connection open at a time, and that
connection being re-used serially (rather than establishing a new connection
for each message), it would not surprise me if their limits were much more
generous, or even not applied at all.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Hi Manvendra,

thanks for your continuous support of qmail, your indimail project, and
your thoughts and observations sharing with us.

May I comment some of your items?

Am Mittwoch, dem 18.05.2022 um 20:22 +0530 schrieb Manvendra Bhangui:
> There have been two things that I have thought of doing differently
> in
> qmail for a very long time.
>
> 1. Avoid scanning todo directory for new email. qmail-queue has the
>    information of a new email. Can this be transmitted to qmail-send
>    efficiently without the need for scanning todo directory?
>

This was certainly a design decision of DJB using the file system for
both persistent storage (store&forward) and by the same token for
controlling the delivery and state. Here, we are 25 years back in time.
Given today's hardware, a revision makes sense. Further, we see
different VFSs now and of course no rotating disks any more.

It would be a challenge to provide better queuing (and faster delivery)
using the same API for eg. qmail-smtpd (and others feeding or draining
the queue).


> 2. Send multiple emails using as few connections as possible for
> remote
>    deliveries rather a making new connection for every email to be
>    delivered to remote hosts. This will be my next project and I
>    think this will be challenging because too much to deal with
> tcp/ip
>    sockets, timeouts, handling reconnection and stuff. Not sure I
> will
>    succeed, but I will give it a try.

Same for my s/qmail; which is on my list of todos. Have a look at the
smtproutes case. 
Actually, a redesign of the queue would help in that respect. Perhaps
defining a useful control message queue protocol?


> There are few other things that this release achieves
>
> 1 dynamically Increase the number of queues based on the injection
>   rate.

My thoughts (about s/qmail) was a bit different: Using particular
delivery channels for authenticated mails; potentially with encrypted
messages in the queue.

>
> 2 Rate limit remote deliveries. This used to be the biggest challenge
> for
>   me when dealing with high volume delivery to sites like yahoo, aol
> and
>   roadrunner.

Hm, rate-limiting seems not to be concern for my users.

---

Just one more question: The advantace of netqmail arises from the fact
that it minimizes the fsync() calls?

Thanks again and also to Charles for his helpful comments.

Regards.
--eh.

--
Dr. Erwin Hoffmann | www.fehcom.de
PGP key-id: 20FD6E671A94DC1E
PGP key-fingerprint: 8C6B 155B 0FDA 64F1 BCCE A6B9 20FD 6E67 1A94 DC1E
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
On Thu, 19 May 2022 at 19:30, Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:
>
> Manvendra Bhangui <mbhangui@gmail.com> wrote:
> > [...] The rate limiting code also records for each domain the
> > delivery rate. When the delivery rate falls below a number, only then
> > qmail-send picks up the email and communicates to qmail-rspawn to make
> > the remote delivery attempt.
>
> I would be curious to see if Yahoo rate-limits incoming mail with the
> maildirsmtp method. With only one connection open at a time, and that
> connection being re-used serially (rather than establishing a new connection
> for each message), it would not surprise me if their limits were much more
> generous, or even not applied at all.

Interesting. Yahoo could be rate limiting because of multiple
connections from the same IP, which makes use of maildirsmtp
attractive. So it may not be the number of mails sent. So what you are
saying basically is

1. have the emails to these domains delivered locally to a Maildir.
2. run maildirsmtp supplying the appropriate prefix. I guess I can run
as many maildirsmtp as the number of such domains. So if I have
yahoo.com and gmail.com, I can run two maildirsmtp in parallel? Right?

I will run this idea with my user and try this and see if this can
improve the delivery process. Thank you
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
On Thu, 19 May 2022 at 19:56, Erwin Hoffmann <feh@fehcom.de> wrote:
> May I comment some of your items?
>
Sure Erwin. You don't require my permission for that :)

...

> > 2. Send multiple emails using as few connections as possible for
> > remote
> > deliveries rather a making new connection for every email to be
> > delivered to remote hosts. This will be my next project and I
> > think this will be challenging because too much to deal with
> > tcp/ip
> > sockets, timeouts, handling reconnection and stuff. Not sure I
> > will
> > succeed, but I will give it a try.
>
> Same for my s/qmail; which is on my list of todos. Have a look at the
> smtproutes case.

I looked up your site and couldn't find details on this. Any specific
link to this? Maybe you are talking about distributed queueing QMQ.
This I think is similar to my concept of multiple local queues which
could be distributed across different disks across different
controllers, etc. How do you distribute the load across the clusters?

> Actually, a redesign of the queue would help in that respect. Perhaps
> defining a useful control message queue protocol?
...
> My thoughts (about s/qmail) was a bit different: Using particular
> delivery channels for authenticated mails; potentially with encrypted
> messages in the queue.

I look forward to some ideas on this. I did use shared memory segments
to store the concurrency. qmail-send increments this number when a
mail is added for delivery and decrements it when it gets deferred,
delivered or bounced. This has allowed me to write a script named qtop
displaying the current local and remote concurrency instantly for all
my queues
--
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
On Thu, 19 May 2022 at 20:18, Ramesh T S <rameshts@rediff.co.in> wrote:
>
> The bullet mail stuff has limits on 1 connection with number of emails per connection value could be in the range of 50 ~ 100. Alternatively multiple ip's in that system can benefit in the sending logic, Random ip selections to connect with and a rate limitation on the number of mails per connection.

I have this control file named outgoingip in indimail-mta. One can
have multiple ip listed in this control file. Based on the time,
qmail-remote uses the modulus operator to choose an outgoing ip for
remote deliveries. This control file can also be
outgoingip.example.com which makes qmail-remote use the control file
only for deliveries to example.com.
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Hi,

Am Donnerstag, dem 19.05.2022 um 20:51 +0530 schrieb Manvendra Bhangui:
> On Thu, 19 May 2022 at 19:56, Erwin Hoffmann <feh@fehcom.de> wrote:
> > May I comment some of your items?
> >
> Sure Erwin. You don't require my permission for that :)
>
> ...
>
> > > 2. Send multiple emails using as few connections as possible for
> > > remote
> > >    deliveries rather a making new connection for every email to
> > > be
> > >    delivered to remote hosts. This will be my next project and I
> > >    think this will be challenging because too much to deal with
> > > tcp/ip
> > >    sockets, timeouts, handling reconnection and stuff. Not sure I
> > > will
> > >    succeed, but I will give it a try.
> >
> > Same for my s/qmail; which is on my list of todos. Have a look at
> > the
> > smtproutes case.
>

in case to setup smtproutes

:mailrelay.example.com;port|user|passwd

qmail-remote (at least for s/qmail) delivers all messages to that
mailrelay in one (authenticated) SMTP transaction irrespectively of the
recipient domain: SMTP pipelining with one message (body) transmission
only.


> I looked up your site and couldn't find details on this. Any specific
> link to this? Maybe you are talking about distributed queueing QMQ.
> This I think is similar to my concept of multiple local queues which
> could be distributed across different disks across different
> controllers, etc. How do you distribute the load across the clusters?

No, QMQ is not involved here; just plain qmail-remote + qmail-rspawn.

Regards.
--eh.

--
Dr. Erwin Hoffmann | www.fehcom.de
PGP key-id: 20FD6E671A94DC1E
PGP key-fingerprint: 8C6B 155B 0FDA 64F1 BCCE A6B9 20FD 6E67 1A94 DC1E
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Manvendra Bhangui <mbhangui@gmail.com> wrote:
> On Thu, 19 May 2022 at 19:30, Charles Cazabon wrote:
> >
> > I would be curious to see if Yahoo rate-limits incoming mail with the
> > maildirsmtp method. With only one connection open at a time, and that
> > connection being re-used serially (rather than establishing a new connection
> > for each message), it would not surprise me if their limits were much more
> > generous, or even not applied at all.
>
> Interesting. Yahoo could be rate limiting because of multiple
> connections from the same IP, which makes use of maildirsmtp
> attractive.

Exactly. If you're delivering 150 messages to them with unpatched qmail,
you're opening 150 connections. Those may not all be in parallel, but it's
likely a number of them might. Yahoo might be doing rate-limiting primary on
"connections from <IP> per <time>" instead of/in addition to "messages from
<IP>..." or "messages from <DOMAIN>...".

> 1. have the emails to these domains delivered locally to a Maildir.
> 2. run maildirsmtp supplying the appropriate prefix.

Yes. You set up yahoo.com or gmail.com or whatever as a virtualdomain,
probably controlled by the alias user with a suitable prefix. Then
periodically run maildirsmtp on each of those maildirs, configured to strip
the appropriate prefix.

> I guess I can run as many maildirsmtp as the number of such domains. So if I
> have yahoo.com and gmail.com, I can run two maildirsmtp in parallel? Right?

Absolutely. My preference is to run them as a daemontools service so that you
run them with a fixed down period between connections to that domain, and you
cdon't accidentally end up with two trying to process the same maildir at once
(a la cron), but if you choose cron you can always avoid the problem with
setlock or similar.

For anyone not quite sure what configuration I've been talking about...

.../qmail/control/virtualdomains:

gmail.com:alias-gmail
yahoo.com:alias-yahoo
[...]

~alias/.qmail-gmail-default:

./outgoing-gmail/

/var/service/mail-gmail/run script:

#!/bin/sh

maildirsmtp ~alias/outgoing-gmail \
alias-gmail- gmail-smtp-in.l.google.com myhostname.example.org

sleep 300

> I will run this idea with my user and try this and see if this can
> improve the delivery process. Thank you

You're welcome. Please post your results - I haven't done this comparison in
a donkey's age, and would be interested in how well it works now.

Charles

P.S. any other old hands here find their fingers typing "qmail" instead of
"gmail"?

--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
It appears that Manvendra Bhangui <mbhangui@gmail.com> said:
>1. have the emails to these domains delivered locally to a Maildir.
>2. run maildirsmtp supplying the appropriate prefix. I guess I can run
>as many maildirsmtp as the number of such domains. So if I have
>yahoo.com and gmail.com, I can run two maildirsmtp in parallel? Right?

Right, so long as they have a different prefix. I agree that you get
rate limited on parallel deliveries, not sequential ones.

Another possibility is to use multiple instances of qmail. My main instance
is in /var/qmail, a second one in /var/qmail-straw for the recipients who
want to get their mail through a straw rather than the usual firehose.

The concurrencyremote in the main instance is 30, in qmail-straw is 3.
I have virtualdomains entries in the main instance that deliver mail
to a tiny script that requeues them in qmail-straw. Works great.

By the way, I have found that Yahoo and Comcast need the straw treatment
but not Gmail.

R's,
John
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Thus said Manvendra Bhangui on Thu, 19 May 2022 19:59:02 +0530:

> I can run two maildirsmtp in parallel? Right?

Definitely. You only want one maildir per domain in this case because
you aren't delivering all email to one relay. So in virtualdomains:

gmail.com:serial-gmail
yahoo.com:serial-yahoo

Then in ~serial/.qmail-gmail-default:

./maildir/gmail/

And in ~serial/.qmail-yahoo-default:

./maildir/yahoo/

And of course (as "serial" user)

maildirmake ~/maildir/gmail
maildirmake ~/maildir/yahoo

Then you just need to run (as "serial" user):

maildirsmtp ~/maildir/gmail serial-gmail- $(dnsmx gmail.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)

maildirsmtp ~/maildir/yahoo serial-yahoo- $(dnsmx yahoo.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)

The commands for emptying the maildirs can be put in a cronjob. For
parallelism just have multiple crontab entries. You may want to make the
MX lookup more robust.

This example is adapted from TOISP in the serialmail sources.

Andy
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Awesome thanks Andy. That was a very detailed on post how to setup
maildirsmtp and will go a long way for anyone who wants to setup
maildirsmtp.

On Fri, 20 May 2022 at 08:58, Andy Bradford
<amb-sendok-1655608419.pihkcaidealnncpjikcf@bradfords.org> wrote:
>
> Thus said Manvendra Bhangui on Thu, 19 May 2022 19:59:02 +0530:
>
> > I can run two maildirsmtp in parallel? Right?
>
> Definitely. You only want one maildir per domain in this case because
> you aren't delivering all email to one relay. So in virtualdomains:
>
> gmail.com:serial-gmail
> yahoo.com:serial-yahoo
>
> Then in ~serial/.qmail-gmail-default:
>
> ./maildir/gmail/
>
> And in ~serial/.qmail-yahoo-default:
>
> ./maildir/yahoo/
>
> And of course (as "serial" user)
>
> maildirmake ~/maildir/gmail
> maildirmake ~/maildir/yahoo
>
> Then you just need to run (as "serial" user):
>
> maildirsmtp ~/maildir/gmail serial-gmail- $(dnsmx gmail.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)
>
> maildirsmtp ~/maildir/yahoo serial-yahoo- $(dnsmx yahoo.com | sort -k 1n | head -1 | awk '{ print $2 }') $(hostname)
>
> The commands for emptying the maildirs can be put in a cronjob. For
> parallelism just have multiple crontab entries. You may want to make the
> MX lookup more robust.
>
> This example is adapted from TOISP in the serialmail sources.
>
> Andy
>


--
Regards Manvendra - http://www.indimail.org
GPG Pub Key
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xC7CBC760014D250C
Re: TLDR; Using POSIX message queues in qmail-queue to communicate with qmail-send [ In reply to ]
Thus said Manvendra Bhangui on Fri, 20 May 2022 09:02:03 +0530:

> Awesome thanks Andy. That was a very detailed on post how to setup
> maildirsmtp and will go a long way for anyone who wants to setup
> maildirsmtp.

Apparently I hadn't yet seen Charles' very similar set of instructions
or I may not have responded.

I too would actually recommend using daemontools to get a better
experience than cronjob, but as he said, the same can be accomplished
with setlock.

Andy