Mailing List Archive

Quarantining an HTML email, later injecting it - not displaying properly
People,

I think this a problem that I have created rather than a problem with my
email client (roundcubemail):

- I use a netqmail setup and make use of qmail-qfilter to test incoming
mails, if they are suspicious, I reject them but also copy them line by
line to a file in a "quarantine" dir

- Later, if I find the mail/file is OK I inject it - to have it properly
delivered - but I frequently have problems with HTML mail not displaying
anything and I end having to mess around with the text file and
re-inject it before I get something that I can read - usually I just
strip out everything except the plain text but in this case I want to
see the full HTML mail

- I presume it has something to do with what qmail does to process the
mail before it puts it into the Maildir dir? It seems the issue is to
do with Content-Type boundaries or something

I attach a problem mail - if some guru can tell me if this is a qmail
problem or otherwise point me in the right direction it would be
appreciated . .

Thanks,

Phil.

--
Philip Rhoades

GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au
Re: Quarantining an HTML email, later injecting it - not displaying properly [ In reply to ]
Philip Rhoades <phil@pricom.com.au> wrote:
>
> - Later, if I find the mail/file is OK I inject it - to have it
> properly delivered - but I frequently have problems with HTML mail
[...]
> - I presume it has something to do with what qmail does to process
> the mail before it puts it into the Maildir dir? It seems the issue
> is to do with Content-Type boundaries or something

No, that's got to be a misdiagnosis, as qmail neither knows nor cares about
any of that. In particular, the process of writing messages to the maildir
(if you're not using an external MDA in place of qmail-local) simply writes
out a Return-path: and Delivered-to: header, then copies the message to the
file as a string of bytes. qmail doesn't even parse the message during
delivery, much less care about its MIME formatting. Messages can be parsed
during message injection (though not if they're received by SMTP in a vanilla
qmail setup) and broken MIME structure could cause problems there.

How, exactly, are you re-injecting these quarantined messages? And how are
you saving them in the first place? Those would seem to be the likely
locations for a problem due to parsing the message content.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: Quarantining an HTML email, later injecting it - not displaying properly [ In reply to ]
On 2014-04-11 04:04, Charles Cazabon wrote:
> Philip Rhoades <phil@pricom.com.au> wrote:
>>
>> - Later, if I find the mail/file is OK I inject it - to have it
>> properly delivered - but I frequently have problems with HTML mail
> [...]
>> - I presume it has something to do with what qmail does to process
>> the mail before it puts it into the Maildir dir? It seems the issue
>> is to do with Content-Type boundaries or something
>
> No, that's got to be a misdiagnosis, as qmail neither knows nor cares
> about
> any of that. In particular, the process of writing messages to the
> maildir
> (if you're not using an external MDA in place of qmail-local) simply
> writes
> out a Return-path: and Delivered-to: header, then copies the message to
> the
> file as a string of bytes. qmail doesn't even parse the message during
> delivery, much less care about its MIME formatting.


Right.


> Messages can be parsed
> during message injection (though not if they're received by SMTP in a
> vanilla
> qmail setup) and broken MIME structure could cause problems there.


OK, that seems like a possibility . .


> How, exactly, are you re-injecting these quarantined messages?


/var/qmail/bin/qmail-inject < filename


> And how are
> you saving them in the first place? Those would seem to be the likely
> locations for a problem due to parsing the message content.


I think you must be right - it is a Ruby script that parses the incoming
mail that is intercepted by qmail-qfilter and, if it is not acceptable,
written to a file, line by line - and the incoming SMTP message rejected
with a return message (as opposed to other messages that I KNOW are no
good and which are dropped silently).

However, from memory, the only changes I make to the incoming file are
to concatenate header lines that are a continuation of the previous
header line (ie they are indented) so any MIME separator things should
be unchanged . . I suppose I could test by temporarily turning off the
spam filter and getting my friend to do another edit on the Google Doc
and cause another mail to be sent and see what that looks like . .

I don't like HTML mail and I have only ever wanted to see the plain text
in the past so it was easy to fix but now I want to see what the HTML
mail looks like.

Thanks,

Phil.
--
Philip Rhoades

GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au
Re: Quarantining an HTML email, later injecting it - not displaying properly [ In reply to ]
Philip Rhoades <phil@pricom.com.au> wrote:
> On 2014-04-11 04:04, Charles Cazabon wrote:
> > qmail doesn't even parse the message during delivery, much less care about
> > its MIME formatting.
>
> Right.
>
> > Messages can be parsed during message injection (though not if they're
> > received by SMTP in a vanilla qmail setup) and broken MIME structure could
> > cause problems there.
>
> OK, that seems like a possibility . .
>
> >How, exactly, are you re-injecting these quarantined messages?
>
> /var/qmail/bin/qmail-inject < filename

That can be problematic. qmail-inject isn't designed to deal with
arbitrarily-badly-written messages from random MUAs on the 'net, and it can
fail to parse some badly misformatted messages, resulting in no queued mail.
It does exit nonzero in this case, so if whatever is calling qmail-inject is
well-written, it shouldn't lose mail.

You may wish to use new-inject from the mess822 package instead. It is
designed to cope with really badly malformatted messages on input.

Neither of these does anything MIME-specific, however.

> > And how are you saving them in the first place? Those would seem to be
> > the likely locations for a problem due to parsing the message content.
>
> I think you must be right - it is a Ruby script that parses the incoming
> mail that is intercepted by qmail-qfilter and, if it is not acceptable,
> written to a file, line by line - and the incoming SMTP message rejected
> with a return message (as opposed to other messages that I KNOW are no good
> and which are dropped silently).

No offense, but writing proper mail-handling software is difficult; because of
decades of historical baggage, many workarounds and undocumented tricks are
needed to deal with the variety of badly-written mail software out there.
Rather than rolling your own, you may wish to benefit from the experience of
the authors of well-known mail-handling software.

You could possibly use reformime or another tool to do your message parsing
and munging, or write one in a high-level language that has a good
MIME-handling library so that you're not just treating the message as a bunch
of lines of text. Python's email module is pretty good and easy to use for
this sort of thing.

> I don't like HTML mail and I have only ever wanted to see the plain text

Me too, and the same goes for many users of this list, but we are
unfortunately in the minority these days. I don't bother munging incoming
mail; I just have mutt configured to prefer to display the plaintext version
of any multipart/alternative message, and if an HTML-only message arrives
(that *isn't* spam ;) mutt just runs it through w3m to display it to me in
plaintext.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: Quarantining an HTML email, later injecting it - not displaying properly - BUG FIXED [ In reply to ]
Charles,


On 2014-04-11 11:56, Charles Cazabon wrote:
> Philip Rhoades <phil@pricom.com.au> wrote:
>> On 2014-04-11 04:04, Charles Cazabon wrote:
>> > qmail doesn't even parse the message during delivery, much less care about
>> > its MIME formatting.
>>
>> Right.
>>
>> > Messages can be parsed during message injection (though not if they're
>> > received by SMTP in a vanilla qmail setup) and broken MIME structure could
>> > cause problems there.
>>
>> OK, that seems like a possibility . .
>>
>> >How, exactly, are you re-injecting these quarantined messages?
>>
>> /var/qmail/bin/qmail-inject < filename
>
> That can be problematic. qmail-inject isn't designed to deal with
> arbitrarily-badly-written messages from random MUAs on the 'net, and it
> can
> fail to parse some badly misformatted messages, resulting in no queued
> mail.
> It does exit nonzero in this case, so if whatever is calling
> qmail-inject is
> well-written, it shouldn't lose mail.
>
> You may wish to use new-inject from the mess822 package instead. It is
> designed to cope with really badly malformatted messages on input.
>
> Neither of these does anything MIME-specific, however.


Right.


>> > And how are you saving them in the first place? Those would seem to be
>> > the likely locations for a problem due to parsing the message content.
>>
>> I think you must be right - it is a Ruby script that parses the
>> incoming
>> mail that is intercepted by qmail-qfilter and, if it is not
>> acceptable,
>> written to a file, line by line - and the incoming SMTP message
>> rejected
>> with a return message (as opposed to other messages that I KNOW are no
>> good
>> and which are dropped silently).
>
> No offense, but writing proper mail-handling software is difficult;
> because of
> decades of historical baggage, many workarounds and undocumented tricks
> are
> needed to deal with the variety of badly-written mail software out
> there.


No offense taken!


> Rather than rolling your own, you may wish to benefit from the
> experience of
> the authors of well-known mail-handling software.


When I tried stuff that was around at the time (2006) - nothing would do
exactly what I wanted - so I thought it was an opportunity to get
familiar with Ruby by using it for a real-world function and to learn a
bit about MTAs as well. Of course there have been occasional dramas and
occasional cranky people who were trying to mail me . .


> You could possibly use reformime or another tool to do your message
> parsing
> and munging, or write one in a high-level language that has a good
> MIME-handling library so that you're not just treating the message as a
> bunch
> of lines of text. Python's email module is pretty good and easy to use
> for
> this sort of thing.


After temporarily turning of the spam checker and getting the other
Google Doc contributor to add another comment - I eventually found out
what the problem was: my script was keeping all the lines of the body of
the mail OK but when it was writing it out to the file, it was missing
line one ie it was an array[0] problem. The surprising thing is that
that bug has been there for years and I have only ever had an issue with
rejected HTML mails - and until this particular case, I didn't really
care about keeping the HTML stuff anyway . .


>> I don't like HTML mail and I have only ever wanted to see the plain
>> text
>
> Me too, and the same goes for many users of this list, but we are
> unfortunately in the minority these days. I don't bother munging
> incoming
> mail; I just have mutt configured to prefer to display the plaintext
> version
> of any multipart/alternative message, and if an HTML-only message
> arrives
> (that *isn't* spam ;) mutt just runs it through w3m to display it to me
> in
> plaintext.


Right - I still do use mutt sometimes - mostly when I only have ssh
access or a slow connection - or when I am checking something.

I guess the other thing is that, like Qmail itself, my spam system
(which also includes greylight) has evolved and, generally, works very
well for my particular situation and so I don't really have any
incentive to change . . maybe if the licencing situation with Qmail
changed dramatically and there was a new wave of development, it would
be time to reconsider my situation . .

Thanks for the help!

Regards,

Phil.
--
Philip Rhoades

GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au
Re: Quarantining an HTML email, later injecting it - not displaying properly - BUG FIXED [ In reply to ]
Thus said Philip Rhoades on Sat, 12 Apr 2014 01:34:58 +1000:

> Of course there have been occasional dramas and occasional cranky
> people who were trying to mail me . .

Those are always fun, especially when their MTA drops the message from
the queue after only a few hours of queue time.

> . . maybe if the licencing situation with Qmail changed dramatically
> and there was a new wave of development, it would be time to
> reconsider my situation . .

I don't think licensing will be a problem. qmai was released to public
domain years ago.

Andy
--
TAI64 timestamp: 400000005348342a
Re: Quarantining an HTML email, later injecting it - not displaying properly - BUG FIXED [ In reply to ]
Andy,


On 2014-04-12 04:27, Andy Bradford wrote:
> Thus said Philip Rhoades on Sat, 12 Apr 2014 01:34:58 +1000:
>
>> Of course there have been occasional dramas and occasional cranky
>> people who were trying to mail me . .
>
> Those are always fun, especially when their MTA drops the message
> from
> the queue after only a few hours of queue time.
>
>> . . maybe if the licencing situation with Qmail changed dramatically
>> and there was a new wave of development, it would be time to
>> reconsider my situation . .
>
> I don't think licensing will be a problem. qmai was released to
> public
> domain years ago.


My understanding was that it was the licencing that was preventing
people from continuing to develop the base Qmail code ie instead of
having an old original source that has a whole stack of patches that
need to be added? I mean, it still works fine for me but if it has been
released to the public domain why are there not a lot of people
contributing code to the base?

Regards,

Phil.
--
Philip Rhoades

GPO Box 3411
Sydney NSW 2001
Australia
E-mail: phil@pricom.com.au
Re: Quarantining an HTML email, later injecting it - not displaying properly - BUG FIXED [ In reply to ]
Philip Rhoades <phil@pricom.com.au> wrote:
> After temporarily turning of the spam checker and getting the other
> Google Doc contributor to add another comment - I eventually found
> out what the problem was: my script was keeping all the lines of the
> body of the mail OK but when it was writing it out to the file, it
> was missing line one ie it was an array[0] problem.

Aha. I'm glad you figured it out.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: qmail in the public domain [ In reply to ]
Philip Rhoades <phil@pricom.com.au> wrote:
>
> My understanding was that it was the licencing that was preventing
> people from continuing to develop the base Qmail code ie instead of
> having an old original source that has a whole stack of patches that
> need to be added?

Nope. As Andy said, djb put qmail and most of his other software into the
public domain some years back.

There's nothing legally holding you (or I) back from developing qmail and
releasing it -- in fact, Russell Nelson organized a group of us to put
together netqmail in exactly that fashion. It still exists.

But I'm afraid it came too late for qmail to be saved. The world had
basically moved on from qmail by the time it became public domain. There is
very little reason to use qmail/netqmail in new installations today; if you
don't already have a significant time investment in systems running qmail,
you'd probably pick another package and be just as happy with it.

> I mean, it still works fine for me but if it has been released to the public
> domain why are there not a lot of people contributing code to the base?

Not many people these days contribute designs for performance-enhancing parts
for Model As, either.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: qmail in the public domain [ In reply to ]
Thus said Charles Cazabon on Fri, 11 Apr 2014 15:24:50 -0600:

> But I'm afraid it came too late for qmail to be saved. The world had
> basically moved on from qmail by the time it became public domain.
> There is very little reason to use qmail/netqmail in new installations
> today; if you don't already have a significant time investment in
> systems running qmail, you'd probably pick another package and be just
> as happy with it.

Why is there very little reason to use qmail/netqmail in new
installations today?

As for as an MTA is concerned, it really requires very low maintenance,
can be easily adapted to fit many situations, it's lightweight, secure,
etc... Does the world no longer need these things?

Perhaps it remains as-is because it's community prefers it what way?

Andy
--
TAI64 timestamp: 400000005349697f
Re: qmail in the public domain [ In reply to ]
Hi Charles & list,


--On 11. April 2014 15:24:50 -0600 Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:


> But I'm afraid it came too late for qmail to be saved. The world had
> basically moved on from qmail by the time it became public domain. There
> is very little reason to use qmail/netqmail in new installations today;
> if you don't already have a significant time investment in systems
> running qmail, you'd probably pick another package and be just as happy
> with it.

> Charles

So this is your coffin nail for qmail? Sad to hear.


While you guided over 15 years the community with your wisdom (in my book I
referenced you as 'answering machine' for qmail), it would certainly be
helpful to have some kind of retrospective view and to understand, why
other packages like Postfix and Exim are more popular these days.

There is, however, an important community still relying on qmail. In
addition, some kind of 'mono culture' is certainly not helpful.

As part of my restrospective:

Problem 1: How to spread the news. Unfortunately, qmail.org does neither
reflect the needs for a contempory mail admin, nor does it bundle the
developments. qmailrocks is a different story.

Problem 2: Unlike other email packages, qmail developers are isolated.
There is no forum, nobody to bundle their activities, and perhaps provide a
'product' out of it.

Problem 3: To many ways to go/fail. For qmail, there exist a lot of
specialized solutions (I contributed some). However - and unlike Postfix -
there is no major path. In fact: You can install Postfix today and it works
as contempory MTA solution. With qmail, you have to look for a lot of
(contradictory) patches -- and you find those at very many locations.


Anyway, I will continue to support qmail and develop it further --
hopefully in a direction DJB likes.

--

In case somebody is still interested in IM2000: I spent dome ideas on an
interesting concept which probably will work today, but simply was beyond
reacheability, when Dan decided to open up the discussion.

--

BTW: The concept of qmail-smtpd and qmail-pop3d to use ucspi-tls or
ucspi-ssl and to fork a new instance for every conection, prevents qmail
users/admins from some consequences of the Heartbleed bug in OpenSSL --
unlike other MTAs with a permanent daemon running (see my remarks above).

@John Levine (if listening): Do you know the discussion around RFC 6250 at
the IETF? This is a particular badly written RFC.

regards.
--eh.


--
Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id: 7E4034BE
Re: qmail in the public domain [ In reply to ]
Andy Bradford <amb-sendok-1399912028.fgkjabnabicmcfbfgied@bradfords.org> wrote:
> > There is very little reason to use qmail/netqmail in new installations
> > today; if you don't already have a significant time investment in
> > systems running qmail, you'd probably pick another package and be just
> > as happy with it.
>
> Why is there very little reason to use qmail/netqmail in new
> installations today?
>
> As for as an MTA is concerned, it really requires very low maintenance,
> can be easily adapted to fit many situations, it's lightweight, secure,
> etc... Does the world no longer need these things?

qmail's historical advantages -- particularly its low memory and CPU demands
vs. those of other MTAs under similar mail loads -- are no longer as important
as they once were. Even a cheap virtual server now has plenty enough "oomph"
to deliver millions of messages a day with less-efficient software like
Postfix.

So it's not that there's no reason to use qmail -- just that the reasons that
used to strongly point towards qmail are no longer as relevant. qmail's other
historical advantage, security, has not been much of a problem for its main
competition in recent years either -- though whether that's due to the other
MTAs being "secure enough" or just because no one bothers going after MTAs
because there are too many other more lucrative targets is difficult to say.

I still like qmail. I still run qmail. I still consult on qmail, and I still
follow this list, answering (nontrivial, non-FAQ) questions when they come up.
If a client comes to me with qmail infrastructure, I'm quite happy to help
them extend/customize/whatever it. But if someone comes to me and says "I
need to set up a brand new mail infrastructure that does X, Y, and Z", chances
are good that an honest evaluation of their needs is going to result in a
decision between qmail (which is not supplied by their OS of choice, and is
unsupported from their vendor) and some other MTA which has those particular
tickboxes checked -- and in that case, qmail is, likely as not, not going to
come out on top.

It can hardly be news to anyone that qmail is not exactly a hotbed of
development or even use these days. How many messages a month does this list
even get any more? It's not like there are thousands of newly installed qmail
mailhosts with newly minted postmasters in here begging for beginner's help
any more.

I didn't intend to ignite a debate over this. I simply admitted what seems to
be the facts. I don't do religion; my software zealotry days died with the
Amiga decades ago.

> Perhaps it remains as-is because it's community prefers it what way?

qmail is the way it is. That's fine, but I don't see a lot of point in
insisting, contrary to the evidence, that qmail still holds any truly
significant advantages over other (non-proprietary) MTAs these days.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: qmail in the public domain [ In reply to ]
Erwin Hoffmann <feh@fehcom.de> wrote:
>
> So this is your coffin nail for qmail?

Not at all. I'll still be running qmail for my own use, and consulting for
clients about it, in a decade's time. But see my other message.

> While you guided over 15 years the community with your wisdom (in my
> book I referenced you as 'answering machine' for qmail),

I truly lol'd at that ;)

> it would certainly be helpful to have some kind of retrospective view and to
> understand, why other packages like Postfix and Exim are more popular these
> days.

See my other message for some of my reasoning.

Yes, it would be nice to see qmail come into the 21st century. If someone
wanted to do the work, here a few points off the top of my head, in no
particular order, that they might want to address:

Working on qmail's code is difficult, or an interesting challenge if you
prefer a less negative term. djb's code is efficient and correct, but it
isn't exactly what you'd call obvious or commented. If you want to implement
a new MTA feature, and you look at what it would take to implement in qmail vs
in some other MTA, qmail loses out heavily here.

qmail's architecture, of mutually distrusting but cooperating processes, is
very good from a security point of view. It does make it more difficult to do
some things -- as you no doubt noticed in implementing recipient verification
during SMTP acceptance of messages. In general, the rigid
compartmentalization does exactly what it was designed to do - makes it
difficult to get information between the processes that wasn't designed into
the communications in the first place.

Code no longer has to be obtuse low-level C, hand-optimized for performance at
the expense of readability, to get "good enough" performance these days, due
to Moore's Law supplying more than ample CPU cycles and memory space. You
could get *much* more maintainable, extensible code simply by replacing some
of the qmail processes with equivalents written in a higher-level language,
and still have it more than fast enough to deliver all the mail you ever
wanted to deliver.

The internet mail infrastructure has moved on. If you were starting an MTA
from scratch these days, you would probably want to design in right from the
beginning many of the features which are more difficult to implement today
with qmail than with other MTAs: recipient verification, DKIM support, SPF
(bleh), TLS/SSL (specifically with STARTTLS) for all network protocols, SMTP
AUTH, ... Yes, you can add all of these things today, but if you just download
netqmail and "go", you're not going to get them.

Shipment with OSes at least as an optional replacement for other MTAs. This
might come automatically after addressing the other points, but it might not
either. And as long as you have to go hunting for it (rather than just
installing it from your OS's packaging system), qmail will remain a niche
product. Getting OSes to include it might require running qmail in a less
djb-esque fashion as well -- for example, I don't recall seeing anyone write
up how to run qmail under upstart or systemd, despite those systems having
vastly more traction than svscan ever will, even if svscan is technically
superior in many ways.

I'm sure there are other points that could be addressed.

> There is, however, an important community still relying on qmail.

Absolutely. As I said, if you already have time invested in setting up and
running qmail infrastructure, then chances are you're going to continue to use
qmail for your systems when you expand or evolve them. That's to be expected.

> In addition, some kind of 'mono culture' is certainly not helpful.

Agreed.

> Problem 1: How to spread the news. Unfortunately, qmail.org does
> neither reflect the needs for a contempory mail admin, nor does it
> bundle the developments. qmailrocks is a different story.

Ya... I've never cared for qmailrocks as it always seemed to me (whether this
was the intent or not) to be a "Here, run this script and with zero
understanding you'll have a qmail system up and running in minutes" kind of
thing.

> Problem 2: Unlike other email packages, qmail developers are
> isolated. There is no forum, nobody to bundle their activities, and
> perhaps provide a 'product' out of it.

Agreed. Maybe there's enough interest to have a public git repo and public
development mailing list, issue/feature request tracker, release notes,
roadmap... maybe there isn't.

> Problem 3: To many ways to go/fail. For qmail, there exist a lot of
> specialized solutions (I contributed some). However - and unlike
> Postfix - there is no major path. In fact: You can install Postfix
> today and it works as contempory MTA solution. With qmail, you have
> to look for a lot of (contradictory) patches -- and you find those
> at very many locations.

Yes, exactly. So if you didn't already have a historical reason to pick
qmail, it's unlikely an admin facing a choice between them would pick qmail,
no? Which was eactly my point...

> Anyway, I will continue to support qmail and develop it further --
> hopefully in a direction DJB likes.

I still use it and support it, and provide consulting and development to
clients. It was not my intent to declare qmail dead, like poor Yorick ;)

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: qmail in the public domain [ In reply to ]
For the public repo: I started to collect and combine patches at https://github.com/gurubert/qmail last year.
--
Robert Sander
Re: qmail in the public domain [ In reply to ]
Hi Charles,


--On 12. April 2014 15:13:41 -0600 Charles Cazabon
<search-web-for-address@pyropus.ca> wrote:

Thanks for your positive comments ;-)

> The internet mail infrastructure has moved on. If you were starting an
> MTA from scratch these days, you would probably want to design in right
> from the beginning many of the features which are more difficult to
> implement today with qmail than with other MTAs: recipient verification,
> DKIM support, SPF (bleh), TLS/SSL (specifically with STARTTLS) for all
> network protocols, SMTP AUTH, ... Yes, you can add all of these things
> today, but if you just download netqmail and "go", you're not going to
> get them.

I did some of those and they are working ;-)

>
> Shipment with OSes at least as an optional replacement for other MTAs.

At least for FreeBSD, there exist a port of my code; since I support clang
of the box ...
>From my point of view: The Linux community is driving to a different
direction.
See systemd and the discussion about it. qmail is targeting servers.


> This might come automatically after addressing the other points, but it
> might not either. And as long as you have to go hunting for it (rather
> than just installing it from your OS's packaging system), qmail will
> remain a niche product. Getting OSes to include it might require running
> qmail in a less djb-esque fashion as well -- for example,

> I don't recall
> seeing anyone write up how to run qmail under upstart or systemd, despite
> those systems having vastly more traction than svscan ever will, even if
> svscan is technically superior in many ways.

There *are* web-sites providing this information; you just haven't seen yet.
And this information has been neglected by Russ since.



>> Problem 2: Unlike other email packages, qmail developers are
>> isolated. There is no forum, nobody to bundle their activities, and
>> perhaps provide a 'product' out of it.
>
> Agreed. Maybe there's enough interest to have a public git repo and
> public development mailing list, issue/feature request tracker, release
> notes, roadmap... maybe there isn't.

This is still an unanswered Q. Even picking up the code of W. Baxter
(Superscript) and developing it to the direction I need, there is to litte
coherence.


>> Problem 3: To many ways to go/fail. For qmail, there exist a lot of
>> specialized solutions (I contributed some). However - and unlike
>> Postfix - there is no major path. In fact: You can install Postfix
>> today and it works as contempory MTA solution. With qmail, you have
>> to look for a lot of (contradictory) patches -- and you find those
>> at very many locations.
>
> Yes, exactly. So if you didn't already have a historical reason to pick
> qmail, it's unlikely an admin facing a choice between them would pick
> qmail, no? Which was eactly my point...

Well. I don't agree here.

1. You need to explain (and prove) why your product is superior (in some
respects).

2. You have to spread the news (DJB did; ages ago and he was successful).

3. You need to have a supporting infrastructure, thus a user (customer)
feels to be 'considered'.

Over the time, with qmail we failed at 3.), then 2.) and now 1.). Hm. This
strategy does not work out.


Just let me explain about my roadmap (you find it on my web page):

Done: SMTP Auth + TLS (together with ucspi-ssl). Next release Spamcontrol
2.7.31 (providing relaying by means of client certs).
Next: DNS-Integration (djbdnscurve6). This is a must for IPv6. rblsmtpd has
already been extended for IPv6 (anybody else here ?)
Next: IPv6 integration. Robert Sander did already most of it.

There are, of course, a lot of other extensions: SPF, DKIM, DMARC (see
Yahoo and their mess), policy based acceptance, Greylisting, a bucket of
TLS enhancements ... PAMs: Kerberos, NTLM, ....


In addition, I believe a good documentation is required.

To be honest, most of the items I am concerned with, are already solved by
(eg.) Postfix.
However, the solutions we con provide could be at least be as smart.
Nobody can refrain us to do so.

regards.
--eh.


--
Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id: 7E4034BE
Re: qmail in the public domain [ In reply to ]
Erwin Hoffmann <feh@fehcom.de> wrote:
> > Yes, you can add all of these things today, but if you just download
> > netqmail and "go", you're not going to get them.
>
> I did some of those and they are working ;-)

Indeed, and you deserve some public kudos for your contributions to the
community, such as remains of it. So kudos :)

> > I don't recall seeing anyone write up how to run qmail under upstart or
> > systemd [...]
>
> There *are* web-sites providing this information; you just haven't seen yet.

Entirely possible.

> And this information has been neglected by Russ since.

I suspect that Russ doesn't spend a lot of days on qmail community outreach
these days, but that's just a guess.

> There are, of course, a lot of other extensions: SPF, DKIM, DMARC (see Yahoo
> and their mess), policy based acceptance, Greylisting, a bucket of TLS
> enhancements ... PAMs: Kerberos, NTLM, ....
[...]
> To be honest, most of the items I am concerned with, are already solved by
> (eg.) Postfix. However, the solutions we con provide could be at least be
> as smart. Nobody can refrain us to do so.

No, nobody can stop you or anyone else. My point was more in a convesation
with a client that goes like this:

Client: So I need to set up a mailserver.
Me: Ok. Have you already picked an MTA?
Client: Well, I was thinking qmail or Postfix. The box runs CentOS.
I need it to do SPF and DKIM (signing and verifying).
Me: Ok. You could use Postfix, which is supplied by your OS and will
automatically get updates, is unlikely to be broken by system updates
of other packages, etc. It supports DKIM and SPF out of the box
or nearly so; you just enable these two plugins or features in the
config file and you're done.
Or you could use qmail. Which will take a few hours to set up
properly, and you'll have to hunt down patches or plugins for those
features. And it's difficult to get it to sign mail received from
the network correctly with doing Foo, Bar, and Baz. And system
updates might stomp all over some part of your mail setup because
your OS'es package management system won't know about the qmail stuff.

What client, in their right mind, would participate in that conversation and
go on to pick qmail?

While my particular example above is synthetic, the issue I'm trying to
illustrate is not. I have had this conversation with clients many, many
times. And any consultant with even a trace of engineers' ethics would find
it very difficult to justify trying to convince them to pick qmail absent some
other reason (like the client already having other qmail infrastructure).

I try to be as ethical in my dealings with clients as I can. And that means
that many times (and the frequency increases as the years go by), I do not end
up recommending a qmail solution to their particular needs -- despite my long
years of being a djb admirer and qmail enthusiast.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: qmail in the public domain [ In reply to ]
On Fri, Apr 11, 2014 at 2:24 PM, Charles Cazabon <
search-web-for-address@pyropus.ca> wrote:

> But I'm afraid it came too late for qmail to be saved. The world had
>
> basically moved on from qmail by the time it became public domain. There
> is
> very little reason to use qmail/netqmail in new installations today; if you
> don't already have a significant time investment in systems running qmail,
> you'd probably pick another package and be just as happy with it.
>
>
Do you have a current preferred MTA package?

Regards,

M.
Re: qmail in the public domain [ In reply to ]
> Erwin Hoffmann <feh@fehcom.de> ©ó 2014¦~4¤ë13¤é 6:06 ¼g¹D¡G
>
> Hi Charles,
>
>
> --On 12. April 2014 15:13:41 -0600 Charles Cazabon <search-web-for-address@pyropus.ca> wrote:
>
> Thanks for your positive comments ;-)
>
>> The internet mail infrastructure has moved on. If you were starting an
>> MTA from scratch these days, you would probably want to design in right
>> from the beginning many of the features which are more difficult to
>> implement today with qmail than with other MTAs: recipient verification,
>> DKIM support, SPF (bleh), TLS/SSL (specifically with STARTTLS) for all
>> network protocols, SMTP AUTH, ... Yes, you can add all of these things
>> today, but if you just download netqmail and "go", you're not going to
>> get them.
>
> I did some of those and they are working ;-)
>
>>
>> Shipment with OSes at least as an optional replacement for other MTAs.
>
> At least for FreeBSD, there exist a port of my code; since I support clang of the box ...
>> From my point of view: The Linux community is driving to a different
> direction.
> See systemd and the discussion about it. qmail is targeting servers.
>
>
>> This might come automatically after addressing the other points, but it
>> might not either. And as long as you have to go hunting for it (rather
>> than just installing it from your OS's packaging system), qmail will
>> remain a niche product. Getting OSes to include it might require running
>> qmail in a less djb-esque fashion as well -- for example,
>
>> I don't recall
>> seeing anyone write up how to run qmail under upstart or systemd, despite
>> those systems having vastly more traction than svscan ever will, even if
>> svscan is technically superior in many ways.
>
> There *are* web-sites providing this information; you just haven't seen yet.
> And this information has been neglected by Russ since.
>
>
>
>>> Problem 2: Unlike other email packages, qmail developers are
>>> isolated. There is no forum, nobody to bundle their activities, and
>>> perhaps provide a 'product' out of it.
>>
>> Agreed. Maybe there's enough interest to have a public git repo and
>> public development mailing list, issue/feature request tracker, release
>> notes, roadmap... maybe there isn't.
>
> This is still an unanswered Q. Even picking up the code of W. Baxter (Superscript) and developing it to the direction I need, there is to litte coherence.
>
>
>>> Problem 3: To many ways to go/fail. For qmail, there exist a lot of
>>> specialized solutions (I contributed some). However - and unlike
>>> Postfix - there is no major path. In fact: You can install Postfix
>>> today and it works as contempory MTA solution. With qmail, you have
>>> to look for a lot of (contradictory) patches -- and you find those
>>> at very many locations.
>>
>> Yes, exactly. So if you didn't already have a historical reason to pick
>> qmail, it's unlikely an admin facing a choice between them would pick
>> qmail, no? Which was eactly my point...
>
> Well. I don't agree here.
>
> 1. You need to explain (and prove) why your product is superior (in some respects).
>
> 2. You have to spread the news (DJB did; ages ago and he was successful).
>
> 3. You need to have a supporting infrastructure, thus a user (customer) feels to be 'considered'.
>
> Over the time, with qmail we failed at 3.), then 2.) and now 1.). Hm. This strategy does not work out.
>
>
> Just let me explain about my roadmap (you find it on my web page):
>
> Done: SMTP Auth + TLS (together with ucspi-ssl). Next release Spamcontrol 2.7.31 (providing relaying by means of client certs).
> Next: DNS-Integration (djbdnscurve6). This is a must for IPv6. rblsmtpd has already been extended for IPv6 (anybody else here ?)
> Next: IPv6 integration. Robert Sander did already most of it.
>
> There are, of course, a lot of other extensions: SPF, DKIM, DMARC (see Yahoo and their mess), policy based acceptance, Greylisting, a bucket of TLS enhancements ... PAMs: Kerberos, NTLM, ....
>
>
> In addition, I believe a good documentation is required.
>
> To be honest, most of the items I am concerned with, are already solved by (eg.) Postfix.
> However, the solutions we con provide could be at least be as smart.
> Nobody can refrain us to do so.
>
> regards.
> --eh.
>
>
> --
> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id: 7E4034BE

I was a die-hard qmail fan for 10 years and I have struggled to have DKIM, greylistung working. However, no implementation of DMARC verification and BATV are available.

I still believe that qmail is stronger by itself. It is a pity no active and up-to-date development of it just like what postfix is doing right now.

Kinglok, Fong
Re: qmail in the public domain [ In reply to ]
On Sun, 2014-04-13 at 00:06 +0200, Erwin Hoffmann wrote:

> 3. You need to have a supporting infrastructure, thus a user (customer)
> feels to be 'considered'.

It would be nice to have page check list of features that could be
optionally use, with explanations.

The prospective user should be able to choose the desired features
and get a customised built and easily installable package that may
be handled no differently from other packages.

This seems to be a long pending problem.

-C
Re: qmail in the public domain [ In reply to ]
Hi,


--On 13. April 2014 10:38:09 +0530 Chitta Mandal <chitta@iitkgp.ac.in>
wrote:

> On Sun, 2014-04-13 at 00:06 +0200, Erwin Hoffmann wrote:
>
>> 3. You need to have a supporting infrastructure, thus a user (customer)
>> feels to be 'considered'.
>
> It would be nice to have page check list of features that could be
> optionally use, with explanations.

I don't think a click&paste solution would do for a MTA, since the
dependencies are quite strong.

My concept is, to provide simplistic solutions which can be extended by
external modules, like eg. PAMs for authentitication. Remember, Postfix
uses SASL, a complete different approch.


> The prospective user should be able to choose the desired features
> and get a customised built and easily installable package that may
> be handled no differently from other packages.

To provide the individual modules as external programs -- but available in
the package by default -- seems for me to be the better way. Compilation of
source code is just a snap. However, considering the changing HW
architecture, you have to make sure, that everyhing is well aligned.

Example: On current Linuxes we have the glibc bug resulting in the errno
problem. Qbiff will not compile due to utmpx. On BSD we have Clang. The 64
bit environment is more difficult to resolve here. Even if you manage to
compile, qmail-send will fail.

Now consider the future: You have Debian with systemd (I call it
Debian/PE). Shall we accustom qmail to support it?

> This seems to be a long pending problem.

And there is a lot to do, too ....

regards.
--eh.


> -C
>



--
Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id: 7E4034BE
Re: qmail in the public domain [ In reply to ]
>> It's not like there are thousands of newly installed qmail mailhosts
with newly minted postmasters in here begging for beginner's help any more.

Charles,

May I say as one of those newbies of nearly a decade ago, how grateful I
was for your and others' support. It's been (and still is) a good ride.

Thanks again,

T.
Re: qmail in the public domain [ In reply to ]
On Sun, Apr 13, 2014 at 5:12 PM, Tom T <qmail@deckertelecom.net> wrote:

> >> It's not like there are thousands of newly installed qmail mailhosts
> with newly minted postmasters in here begging for beginner's help any more.
>
> Charles,
>
> May I say as one of those newbies of nearly a decade ago, how grateful I
> was for your and others' support. It's been (and still is) a good ride.
>
> Thanks again,
>
> T.
>

I would second that. I was a newbie struggling to replace and migrate
Critical Path's Isocor mail solution with an OSS solution based on qmail,
courier-imap way back in Dec 1999 for India's first private ISP known as
Sify. I know realize that I have asked tons of stupid questions and Charles
would answer them patiently. I kept on learning from people like Charles,
Dave Sills, Russel Nelson and later Dr Erwin Hoffman. I am a die-hard qmail
fan and now I have put up a distribution called indimail which has almost
all patches you can think off and which runs on almost all linux distros
with RPM and debian packages. I made by career using indimail and am proud
that it is being used by many corporates here in India. I am still
passionate about qmail and keep on updating it almost every month. For me,
qmail is never going to die. Thank you Charles.

--
Regards Manvendra - http://www.indimail.org
Re: qmail in the public domain [ In reply to ]
On Sun, Apr 13, 2014 at 8:15 AM, Manvendra Bhangui <mbhangui@gmail.com>wrote:

> On Sun, Apr 13, 2014 at 5:12 PM, Tom T <qmail@deckertelecom.net> wrote:
>


> I would second that. I was a newbie struggling to replace and migrate
> Critical Path's Isocor mail solution with an OSS solution based on qmail,
> courier-imap way back in Dec 1999 for India's first private ISP known as
> Sify. I know realize that I have asked tons of stupid questions and Charles
> would answer them patiently. I kept on learning from people like Charles,
> Dave Sills, Russel Nelson and later Dr Erwin Hoffman. I am a die-hard qmail
> fan and now I have put up a distribution called indimail which has almost
> all patches you can think off and which runs on almost all linux distros
> with RPM and debian packages. I made by career using indimail and am proud
> that it is being used by many corporates here in India. I am still
> passionate about qmail and keep on updating it almost every month. For me,
> qmail is never going to die. Thank you Charles.
>

+1

(it's been years and years since I posted here)
Re: qmail in the public domain [ In reply to ]
On 04/13/2014 08:15 AM, Manvendra Bhangui wrote:
> On Sun, Apr 13, 2014 at 5:12 PM, Tom T <qmail@deckertelecom.net> wrote:
>
>>>> It's not like there are thousands of newly installed qmail mailhosts
>> with newly minted postmasters in here begging for beginner's help any more.
>>
>> Charles,
>>
>> May I say as one of those newbies of nearly a decade ago, how grateful I
>> was for your and others' support. It's been (and still is) a good ride.
>>
>> Thanks again,
>>
>> T.
>>
> I would second that. I was a newbie struggling to replace and migrate
> Critical Path's Isocor mail solution with an OSS solution based on qmail,
> courier-imap way back in Dec 1999 for India's first private ISP known as
> Sify. I know realize that I have asked tons of stupid questions and Charles
> would answer them patiently. I kept on learning from people like Charles,
> Dave Sills, Russel Nelson and later Dr Erwin Hoffman. I am a die-hard qmail
> fan and now I have put up a distribution called indimail which has almost
> all patches you can think off and which runs on almost all linux distros
> with RPM and debian packages. I made by career using indimail and am proud
> that it is being used by many corporates here in India. I am still
> passionate about qmail and keep on updating it almost every month. For me,
> qmail is never going to die. Thank you Charles.
>

+1

From another newbie ten years ago, who adopted an existing qmail
installation at the State University of New York at Potsdam, and built
it out over the next ten years using the contributions and documentation
from all the folks you mention and more. Inspired by that, I made my
own little contribution to qmail documentation on the net, and sent it
to this list a year ago (2013-03-13).

It seems the timing is right to note here that it has a new home:

http://www.fritzhardy.com/projects/email

Having moved to a new job, this will actually be one of the last
messages I send anywhere from that particular qmail setup. Enjoyed the
ride, and thanks to all.

-Jeff
Re: qmail in the public domain [ In reply to ]
On Thu, Apr 24, 2014 at 3:03 PM, Jeff Hardy <hardyjm@potsdam.edu> wrote:

> http://www.fritzhardy.com/projects/email

Thanks for the clear writeup! In that spirit, here are a few (shorter)
articles about aspects of my (smaller) installations:

http://www.schmonz.com/2007/01/14/qmail-smtp-auth-ssl-tls-patches
http://www.schmonz.com/2007/01/15/qmail-badrcptto-patches
http://www.schmonz.com/2007/02/07/qmail-netbsd-nightly-maintenance
http://www.schmonz.com/2007/02/28/qmail-imap-before-smtp
http://www.schmonz.com/2007/03/07/qmail-spam-filtering

A decade ago I did a bunch of work on pkgsrc's qmail and qmail-run
packages in order to save my own future time and effort (and who
knows, maybe that of a few others). Thanks to Charles' input, it
appears my packages never caused much support headache here. Whether
that's because they're so great, or because hardly anyone uses them...
in any case, I'm glad to know that I didn't wind up making this list's
life harder in order to make my sysadmin life easier. Being able to
install a nicely configured qmail from pkgsrc on most any Unix system
has done that for me, and continues to.

- Amitai