Hi there,
The subject:
This is about scanning mail on a mailserver using clamd - specifically
about a milter for interfacing clamd to an MTA. If you've no interest
in such things, then this probably isn't for you.
Thanks:
ClamAV (specifically clamd, via clamav-milter) has been scanning small
volumes of mail on servers which I manage here for many years. Before
all else, please let me say thank you, to all who have contributed, in
whatever way. For the past couple of decades I've contributed in some
small ways. I hope this will become another contribution.
History:
For the past several years about 98 percent of attempts to send email
to us were not wanted, and without some form of filtering email would
no longer have offered a useful means of communication. This is just
a matter of the volumes; it excludes the quite separate issue of the
potential for mailicious email to pose a threat. Even though a Linux-
only shop like us will be immune from Windows malware and it will tend
to be less of a worry, we wouldn't want to help it to propagate so we
still scan for it; and third-party ClamAV databases have been valuable
in weeding out things like phishing and some other types of spam.
Impetus:
Until fairly recently I've found myself using seven or more milters to
protect against unwanted mail. Other milters address things which
ClamAV doesn't, such as greylisting; rejecting mail sent to spam-traps
and mail from unwanted sources such as those identified by geolocation
and various DNSBL tests; SPF, DKIM and DMARC processing; and regex
scanning of message parts in general (for whitelisting, blacklisting,
and other purposes). My reward has been the quite insignificant level
of unwanted mail breaking through - ClamAV hasn't been called upon to
reject a message here since last September - but on the other hand...
Issues:
the use of many different milters introduced near as many problems as
it solved. Differing (let's say) design philosophies, implementation
details and limitations (even no IPv6!) and their support requirements
- not to mention some *very* different takes on configuration files -
have sometimes found me expending unreasonable effort to track down
failures of one sort or another. This led me to begin developing one
single milter of my own, with multiple goals: replace all the seven
milters which I'd typically use; simplify configuration; eliminate a
few of the limitations and compromises (and their associated confusion
and frustration); whilst at the same time increase flexibility. That
work took almost three years, and is now substantially complete.
Development:
Although with the replacement of clamav-milter (the last milter which
I replaced) the work reached something of a milestone, much remains to
be done to assess e.g. the reliability and scalability of both the
milter and the Sendmail interface, especially at higher mail volumes.
That's where you, gentle reader, might come in.
The milter is pure Perl, and I can easily produce a "cut-down" version
of the script which only replaces clamav-milter. I do not mean in any
way to suggest that there is anything wrong with clamav-milter, but it
could be that there are some tradeoffs.
Tradeoffs - minus:
1. On the small-volume servers I manage I can't remember the last time
that a clamav-milter failed. The Perl milter is not as well exercised
as its 'C' counterpart, and it might break - although it's unusual for
that to happen now, except when I'm developing on production (which is
mostly how I do it:).
2. The Perl milter may be slower. I do not know how much that will be
an issue in higher volume settings than my own, but, given that clamd
typically takes at least tens of milliseconds to scan a short message,
I guess that it isn't going to be serious. I'd like to know; there's
still the option of using XS for some parts of the milter. It has to
be said that the way in which Sendmail presents data to milters isn't
exactly streamlined, but that's out of my hands for the foreseeable.
3. Sendmail's milter interface may perform some sanity tests which as
yet the Perl interface doesn't do. That's a work in progress. At the
moment it doesn't appear to present any problems but one needs to be
prepared for surprises.
4. It's a Perl milter. Obviously you'll need Perl on the system, and
it should be 5.16 or later (think UTF-8).
Tradeoffs - plus:
1. The Perl milter can easily be customized for specific purposes.
For example, things like adding headers, logging, whitelisting (also
other custom short-circuits), custom reply codes, talking to multiple
clamd daemons, tailored responses and similar can, even if you're not
a Perl guru, easily be configured using the Perl milter script as a
kind of template.
2. Control is more fine-grained. For example: (1) the milter can pass
the message headers and body to clamd separately - clamd's nifty cache
of md5sums allows that when there are messages with identical bodies,
the body need only be scanned by the engine once; (2) choice between
ACCEPT, REJECT, TEMPFAIL, DISCARD and QUARANTINE can be more flexible,
as can response codes etc returned to the client; (3) operating system
tools and facilities are available to the milter. If you want, say,
to reply with "5.7.26" to mail scanned under particular circumstances,
or even TARPIT the blighters, it's very easy to do that. Express the
circumstances in Perl code (how many lines of code it takes to do that
doesn't really matter), and then call a couple of functions.
3. The milter might enable you to respond more quickly for example to
attacks, or the odd issues which crop up in other parts of the system.
Writing a one-line statement with a Perl regex filter might be quicker
than e.g. waiting for a vendor to write, test and publish patch.
4. The Perl milter uses its own Sendmail interface, and this will talk
to all reasonably recent versions of Sendmail. You'd be crazy to run
a Sendmail that's so old that the milter can't talk to it. There's no
need to build Sendmail "libmilter", nor for Sendmail to be recompiled.
You don't even need a compiler on the system. You do need to be able
to configure the MTA to use the milter, but that's very easy; insert a
suitable 'X' line in sendmail.cf - either by use the m4 preprocessor
to rebuild your sendmail.cf file (like you're supposed to), or edit a
line in that file (that's what I usually do).
More on the Sendmail interface:
The Sendmail interface is a Perl module, which is published on CPAN.
It's called Sendmail::PMilter. It replaces Sendmail's "libmilter"
library which is normally used for milters which are written in C.
Working with the Perl interface is very much easier than working with
the C interface; you can actually concentrate on what you want to get
done, rather than how you're going to do it. The interface is rather
old, and when I found it the development had stalled with some rather
nasty outstanding issues so I took on its maintainership to fix them.
I've been using it for about three years. The latest *development*
version should be used which at present is 1.20_03. To fix bugs, and
to support the latest Milter specifications, I wrote and/or modified
much of the code. Currently I use the pre-fork mechanism to handle
simultaneous connections. Threaded versions of parts which provide
concurrency need exercise. Anyone willing to give that a try is both
welcome and encouraged to do so. The CPAN distribution includes some
old example milters. My plan, if (a) there's interest here, and (b)
the thing doesn't just crash and burn is to add 'clamav-perl-milter'
(if that's eventually what it gets called) as another example packaged
by the CPAN distribution. I would have no objection to it also being
included by the ClamAV distro. As I wrote the milter I have the right
to say that; being no lawyer I'm not sure what the position is with
the interface Perl module, but in any case it's freely available on
CPAN and can be installed easily from there. Installation involves
little more than extracting a tarball. The CPAN install mechanism (a
one-line shell command) can do that, then after installation run some
module tests. At the time of writing that has been done on at least
45 different configurations on over 80 systems.
Why a "cut-down" milter?
Bigger target audience. Up and running much more easily, no need for
a database for example. And less risk. I'll be publishing the full
monty later on, but I'm not ready for that yet. And I really want to
test the support module thoroughly before perching much atop it, and I
did promise a guy in New York that he can be the first guinea-pig.
Other MTAs:
The milter might also work with Postfix, although many assumptions may
need to be re-visited. I'd especially like to know about that too.
Over to you:
Is anyone interested enough to have a go?
Please reply on-list. Non-list mail to my list address is rejected.
Sorry for the length of all this. There was a lot more I wanted to
say but I had to draw the line somewhere.
If there's no objection I'll drop a link to this post over on the
ClamAV Users' List.
--
73,
Ged.
_______________________________________________
clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel
Please submit your patches to our Bugzilla: http://bugzilla.clamav.net
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml
The subject:
This is about scanning mail on a mailserver using clamd - specifically
about a milter for interfacing clamd to an MTA. If you've no interest
in such things, then this probably isn't for you.
Thanks:
ClamAV (specifically clamd, via clamav-milter) has been scanning small
volumes of mail on servers which I manage here for many years. Before
all else, please let me say thank you, to all who have contributed, in
whatever way. For the past couple of decades I've contributed in some
small ways. I hope this will become another contribution.
History:
For the past several years about 98 percent of attempts to send email
to us were not wanted, and without some form of filtering email would
no longer have offered a useful means of communication. This is just
a matter of the volumes; it excludes the quite separate issue of the
potential for mailicious email to pose a threat. Even though a Linux-
only shop like us will be immune from Windows malware and it will tend
to be less of a worry, we wouldn't want to help it to propagate so we
still scan for it; and third-party ClamAV databases have been valuable
in weeding out things like phishing and some other types of spam.
Impetus:
Until fairly recently I've found myself using seven or more milters to
protect against unwanted mail. Other milters address things which
ClamAV doesn't, such as greylisting; rejecting mail sent to spam-traps
and mail from unwanted sources such as those identified by geolocation
and various DNSBL tests; SPF, DKIM and DMARC processing; and regex
scanning of message parts in general (for whitelisting, blacklisting,
and other purposes). My reward has been the quite insignificant level
of unwanted mail breaking through - ClamAV hasn't been called upon to
reject a message here since last September - but on the other hand...
Issues:
the use of many different milters introduced near as many problems as
it solved. Differing (let's say) design philosophies, implementation
details and limitations (even no IPv6!) and their support requirements
- not to mention some *very* different takes on configuration files -
have sometimes found me expending unreasonable effort to track down
failures of one sort or another. This led me to begin developing one
single milter of my own, with multiple goals: replace all the seven
milters which I'd typically use; simplify configuration; eliminate a
few of the limitations and compromises (and their associated confusion
and frustration); whilst at the same time increase flexibility. That
work took almost three years, and is now substantially complete.
Development:
Although with the replacement of clamav-milter (the last milter which
I replaced) the work reached something of a milestone, much remains to
be done to assess e.g. the reliability and scalability of both the
milter and the Sendmail interface, especially at higher mail volumes.
That's where you, gentle reader, might come in.
The milter is pure Perl, and I can easily produce a "cut-down" version
of the script which only replaces clamav-milter. I do not mean in any
way to suggest that there is anything wrong with clamav-milter, but it
could be that there are some tradeoffs.
Tradeoffs - minus:
1. On the small-volume servers I manage I can't remember the last time
that a clamav-milter failed. The Perl milter is not as well exercised
as its 'C' counterpart, and it might break - although it's unusual for
that to happen now, except when I'm developing on production (which is
mostly how I do it:).
2. The Perl milter may be slower. I do not know how much that will be
an issue in higher volume settings than my own, but, given that clamd
typically takes at least tens of milliseconds to scan a short message,
I guess that it isn't going to be serious. I'd like to know; there's
still the option of using XS for some parts of the milter. It has to
be said that the way in which Sendmail presents data to milters isn't
exactly streamlined, but that's out of my hands for the foreseeable.
3. Sendmail's milter interface may perform some sanity tests which as
yet the Perl interface doesn't do. That's a work in progress. At the
moment it doesn't appear to present any problems but one needs to be
prepared for surprises.
4. It's a Perl milter. Obviously you'll need Perl on the system, and
it should be 5.16 or later (think UTF-8).
Tradeoffs - plus:
1. The Perl milter can easily be customized for specific purposes.
For example, things like adding headers, logging, whitelisting (also
other custom short-circuits), custom reply codes, talking to multiple
clamd daemons, tailored responses and similar can, even if you're not
a Perl guru, easily be configured using the Perl milter script as a
kind of template.
2. Control is more fine-grained. For example: (1) the milter can pass
the message headers and body to clamd separately - clamd's nifty cache
of md5sums allows that when there are messages with identical bodies,
the body need only be scanned by the engine once; (2) choice between
ACCEPT, REJECT, TEMPFAIL, DISCARD and QUARANTINE can be more flexible,
as can response codes etc returned to the client; (3) operating system
tools and facilities are available to the milter. If you want, say,
to reply with "5.7.26" to mail scanned under particular circumstances,
or even TARPIT the blighters, it's very easy to do that. Express the
circumstances in Perl code (how many lines of code it takes to do that
doesn't really matter), and then call a couple of functions.
3. The milter might enable you to respond more quickly for example to
attacks, or the odd issues which crop up in other parts of the system.
Writing a one-line statement with a Perl regex filter might be quicker
than e.g. waiting for a vendor to write, test and publish patch.
4. The Perl milter uses its own Sendmail interface, and this will talk
to all reasonably recent versions of Sendmail. You'd be crazy to run
a Sendmail that's so old that the milter can't talk to it. There's no
need to build Sendmail "libmilter", nor for Sendmail to be recompiled.
You don't even need a compiler on the system. You do need to be able
to configure the MTA to use the milter, but that's very easy; insert a
suitable 'X' line in sendmail.cf - either by use the m4 preprocessor
to rebuild your sendmail.cf file (like you're supposed to), or edit a
line in that file (that's what I usually do).
More on the Sendmail interface:
The Sendmail interface is a Perl module, which is published on CPAN.
It's called Sendmail::PMilter. It replaces Sendmail's "libmilter"
library which is normally used for milters which are written in C.
Working with the Perl interface is very much easier than working with
the C interface; you can actually concentrate on what you want to get
done, rather than how you're going to do it. The interface is rather
old, and when I found it the development had stalled with some rather
nasty outstanding issues so I took on its maintainership to fix them.
I've been using it for about three years. The latest *development*
version should be used which at present is 1.20_03. To fix bugs, and
to support the latest Milter specifications, I wrote and/or modified
much of the code. Currently I use the pre-fork mechanism to handle
simultaneous connections. Threaded versions of parts which provide
concurrency need exercise. Anyone willing to give that a try is both
welcome and encouraged to do so. The CPAN distribution includes some
old example milters. My plan, if (a) there's interest here, and (b)
the thing doesn't just crash and burn is to add 'clamav-perl-milter'
(if that's eventually what it gets called) as another example packaged
by the CPAN distribution. I would have no objection to it also being
included by the ClamAV distro. As I wrote the milter I have the right
to say that; being no lawyer I'm not sure what the position is with
the interface Perl module, but in any case it's freely available on
CPAN and can be installed easily from there. Installation involves
little more than extracting a tarball. The CPAN install mechanism (a
one-line shell command) can do that, then after installation run some
module tests. At the time of writing that has been done on at least
45 different configurations on over 80 systems.
Why a "cut-down" milter?
Bigger target audience. Up and running much more easily, no need for
a database for example. And less risk. I'll be publishing the full
monty later on, but I'm not ready for that yet. And I really want to
test the support module thoroughly before perching much atop it, and I
did promise a guy in New York that he can be the first guinea-pig.
Other MTAs:
The milter might also work with Postfix, although many assumptions may
need to be re-visited. I'd especially like to know about that too.
Over to you:
Is anyone interested enough to have a go?
Please reply on-list. Non-list mail to my list address is rejected.
Sorry for the length of all this. There was a lot more I wanted to
say but I had to draw the line somewhere.
If there's no objection I'll drop a link to this post over on the
ClamAV Users' List.
--
73,
Ged.
_______________________________________________
clamav-devel mailing list
clamav-devel@lists.clamav.net
https://lists.clamav.net/mailman/listinfo/clamav-devel
Please submit your patches to our Bugzilla: http://bugzilla.clamav.net
Help us build a comprehensive ClamAV guide:
https://github.com/vrtadmin/clamav-faq
http://www.clamav.net/contact.html#ml