Mailing List Archive

> Hi All,
>
> I need some insight on fighting against the problem of "stolen passwords".
>
> I run a "standard" netqmail server using the qmail-smtp-auth patch with
> vpopmail vchkpw which worked fairly well the last years. I get more and
> more trouble these days with stolen passwords, used to deliver tons of
> spam to my server to be relayed to external users.
>
> Today, I had a single client delivering about 2500 mails within a minute
> before my queue stats detected it. My best idea is to rate-limit a user
> to not send more than X mails within a period of time and perhaps even
> block the user after a certain number of mails. Any ideas/plugins on how
> to achieve that?
>
> Any further ideas are welcome.

I don't have any immediate suggestions other than to put an increasing
sleep() call between completion of receipt of an email and accepting the
next one, but that doesn't help for emails with multiple recipients.
Although an increasing sleep() call between recipients after a certain
number might work too, assuming it won't cause problems for legitimate
mails.

I too have this happen to me on occasion, usually when I'm unable to even
find out about it for a day or two :( If you come up with a solution I'd
be very interested to see it.

Thanks,
Josh

Joshua Megerman
SJGames MIB #5273 - OGRE AI Testing Division
You can't win; You can't break even; You can't even quit the game.
- Layman's translation of the Laws of Thermodynamics
qmail@honorablemenschen.com

Hi All,

I have the same problem. I ended up with 39,000 messages queued, which
caused Plesk a few problems!

I've only a few clients using my server and they don't send many e-mails
so I run the code below as a background task. I'd be grateful if anyone
has any better ideas or can point out any areas for improvement. (I
doubt I'd get the warning e-mails if my server was flooded, but at least
the mailer should stop.) Combining this with Josh's sleep() should be
quite effective.

#!/bin/bash
V_PID_FILE=/tmp/`basename $0`.pid
V_TODAY=`date +"%y%m%d%H%M%S"`
V_QUEUE_SIZE_FILE=/var/log/qmail_queue_size.log
V_QMAIL_BIN=/var/qmail/bin
V_MAIL_TO=<your e-mail address here>
V_SIZE_WARN=10
V_SIZE_LIMIT=20
V_RUN=true
# Param check
if [ $# -ne 0 ] ; then
echo -e "\nERROR - Too many parameters supplied\n"
echo -e "Usage:\n"
echo -e " `basename $0` \n"
exit 1;
fi;
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": `basename $0` started.\n"
echo $$ > $V_PID_FILE
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Pid $$ written to pid file
'$V_PID_FILE'\n"
trap "V_RUN=false" SIGHUP SIGINT SIGTERM
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Signal trap set for SIGHUP
SIGINT SIGTERM\n"
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Queue size stats file is
'$V_QUEUE_SIZE_FILE'\n"
while $V_RUN; do
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Checking mail queue size...\n"
V_QUEUE_ENTRIES=`$V_QMAIL_BIN/qmail-qstat | awk 'BEGIN {FS=" "}
/messages in queue:/ {print $4}'`
echo `date +"%d/%m/%Y %H:%M:%S"` $V_QUEUE_ENTRIES >>
$V_QUEUE_SIZE_FILE
# Check if queue size is above warning level
if [ $V_QUEUE_ENTRIES -gt $V_SIZE_WARN ] ; then
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Queue entry count >
queue size warning threshold\n"
touch /tmp/message.txt
echo -e `date +"%d/%m/%Y %H:%M:%S"` "WARNING - QMAIL
MESSAGE QUEUE REACHED $V_QUEUE_ENTRIES MESSAGES\n\n" > /tmp/message.txt
/usr/bin/mail -s "WARNING - QMAIL MESSAGE QUEUE EXCEEDED
$V_SIZE_WARN MESSAGES" "$V_MAIL_TO" < /tmp/message.txt
rm /tmp/message.txt
fi
if [ $V_QUEUE_ENTRIES -gt $V_SIZE_LIMIT ] ; then
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Queue entry count >
queue size limit threshold\n"
touch /tmp/message.txt
echo -e `date +"%d/%m/%Y %H:%M:%S"` "ERROR - QMAIL MESSAGE
QUEUE REACHED $V_QUEUE_ENTRIES MESSAGES - MAIL SERVER SHUT DOWN\n\n" >
/tmp/message.txt
/usr/bin/mail -s "ERROR - QMAIL MESSAGE QUEUE EXCEEDED
$V_SIZE_LIMIT MESSAGES - MAIL SERVER SHUT DOWN" "$V_MAIL_TO" <
/tmp/message.txt
rm /tmp/message.txt
sleep 2
/usr/sbin/service qmail stop
V_RUN=false
fi
sleep 15
done
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Monitoring finished.\n\n"


On 27/05/2014 20:09, Joshua Megerman wrote:
>> Hi All,
>>
>> I need some insight on fighting against the problem of "stolen passwords".
>>
>> I run a "standard" netqmail server using the qmail-smtp-auth patch with
>> vpopmail vchkpw which worked fairly well the last years. I get more and
>> more trouble these days with stolen passwords, used to deliver tons of
>> spam to my server to be relayed to external users.
>>
>> Today, I had a single client delivering about 2500 mails within a minute
>> before my queue stats detected it. My best idea is to rate-limit a user
>> to not send more than X mails within a period of time and perhaps even
>> block the user after a certain number of mails. Any ideas/plugins on how
>> to achieve that?
>>
>> Any further ideas are welcome.
> I don't have any immediate suggestions other than to put an increasing
> sleep() call between completion of receipt of an email and accepting the
> next one, but that doesn't help for emails with multiple recipients.
> Although an increasing sleep() call between recipients after a certain
> number might work too, assuming it won't cause problems for legitimate
> mails.
>
> I too have this happen to me on occasion, usually when I'm unable to even
> find out about it for a day or two :( If you come up with a solution I'd
> be very interested to see it.
>
> Thanks,
> Josh
>
> Joshua Megerman
> SJGames MIB #5273 - OGRE AI Testing Division
> You can't win; You can't break even; You can't even quit the game.
> - Layman's translation of the Laws of Thermodynamics
> qmail@honorablemenschen.com
>
>


--
------------------------------------------------------------------------
Verndale Systems Ltd Reg Office: 2a Grove Parade, Buxton, High Peak SK17
6AJ. Company No. 2376672 All Correspondence To: 71 High Street, Fareham
PO16 7BB.
This message contains confidential information and is intended only for
the individual named. If you are not the named addressee you should not
disseminate, distribute or copy this e-mail. Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and
delete this e-mail from your system. E-mail transmission cannot be
assured to be secure or correct as information could be intercepted,
corrupted, lost, destroyed, arrive late or incomplete, or contain
viruses. The sender therefore does not accept liability for any errors
or omissions in the contents of this message, which arise as a result of
e-mail transmission. If verification is required please request a
hard-copy version. © 2005 Verndale Systems Ltd. All rights reserved.

I do something like this to slow down our developers.
This is a send throttle based on the recipient (our devs love to send 10,000 emails to the same person for testing), but it would be trivial to change to the sender

rename /var/qmail/bin/qmail-remote to /var/qmail/bin/qmail-remote.orig and make this shell script /var/qmail/bin/qmail-remote
plus you'll need to create a '/var/qmail/sendthrottle' dir


#!/bin/bash
#
STAT="/usr/bin/stat -c%Z"
TOUCH="/bin/touch"
SENDTHROTTLE="/var/qmail/sendthrottle"
DKREMOTE="/var/qmail/bin/qmail-remote.orig"
THROTTLE=6
NOW=`/bin/date +%s`
ROUTETO=$1
SENDERDOMAIN=${2##*@}
SENDERLHS=${2%%@*}
RECIPDOMAIN=${3##*@}
RECIPLHS=${3%%@*}
shift
shift


#send throttle
LASTSEND=0
if [ -f $SENDTHROTTLE/$RECIPLHS@$RECIPDOMAIN ]; then
LASTSEND=`$STAT $SENDTHROTTLE/$RECIPLHS@$RECIPDOMAIN`
fi
$TOUCH $SENDTHROTTLE/$RECIPLHS@$RECIPDOMAIN
if [ $((LASTSEND+THROTTLE)) -gt $NOW ]; then
sleep $((RANDOM % (4*THROTTLE)))
echo -n -e "s\000ZOutbound Throttle $LASTSEND $NOW"
exit
fi

exec "$DKREMOTE" "$ROUTETO" "${SENDERLHS}@${SENDERDOMAIN}" "$@"

Scott Brynen
direct +1.604.638.9804 mobile +1.778.788.0543
web visioncritical.com<http://www.visioncritical.com/>

From: Mark Jackson [mailto:mark@verndalesystems.com]
Sent: Tuesday, 27 May, 2014 1:11 PM
To: qmail@list.cr.yp.to
Subject: Re: rate-limit relay clients

Hi All,

I have the same problem. I ended up with 39,000 messages queued, which caused Plesk a few problems!

I've only a few clients using my server and they don't send many e-mails so I run the code below as a background task. I'd be grateful if anyone has any better ideas or can point out any areas for improvement. (I doubt I'd get the warning e-mails if my server was flooded, but at least the mailer should stop.) Combining this with Josh's sleep() should be quite effective.

#!/bin/bash
V_PID_FILE=/tmp/`basename $0`.pid
V_TODAY=`date +"%y%m%d%H%M%S"`
V_QUEUE_SIZE_FILE=/var/log/qmail_queue_size.log
V_QMAIL_BIN=/var/qmail/bin
V_MAIL_TO=<your e-mail address here>
V_SIZE_WARN=10
V_SIZE_LIMIT=20
V_RUN=true
# Param check
if [ $# -ne 0 ] ; then
echo -e "\nERROR - Too many parameters supplied\n"
echo -e "Usage:\n"
echo -e " `basename $0` \n"
exit 1;
fi;
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": `basename $0` started.\n"
echo $$ > $V_PID_FILE
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Pid $$ written to pid file '$V_PID_FILE'\n"
trap "V_RUN=false" SIGHUP SIGINT SIGTERM
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Signal trap set for SIGHUP SIGINT SIGTERM\n"
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Queue size stats file is '$V_QUEUE_SIZE_FILE'\n"
while $V_RUN; do
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Checking mail queue size...\n"
V_QUEUE_ENTRIES=`$V_QMAIL_BIN/qmail-qstat | awk 'BEGIN {FS=" "} /messages in queue:/ {print $4}'`
echo `date +"%d/%m/%Y %H:%M:%S"` $V_QUEUE_ENTRIES >> $V_QUEUE_SIZE_FILE
# Check if queue size is above warning level
if [ $V_QUEUE_ENTRIES -gt $V_SIZE_WARN ] ; then
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Queue entry count > queue size warning threshold\n"
touch /tmp/message.txt
echo -e `date +"%d/%m/%Y %H:%M:%S"` "WARNING - QMAIL MESSAGE QUEUE REACHED $V_QUEUE_ENTRIES MESSAGES\n\n" > /tmp/message.txt
/usr/bin/mail -s "WARNING - QMAIL MESSAGE QUEUE EXCEEDED $V_SIZE_WARN MESSAGES" "$V_MAIL_TO" < /tmp/message.txt
rm /tmp/message.txt
fi
if [ $V_QUEUE_ENTRIES -gt $V_SIZE_LIMIT ] ; then
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Queue entry count > queue size limit threshold\n"
touch /tmp/message.txt
echo -e `date +"%d/%m/%Y %H:%M:%S"` "ERROR - QMAIL MESSAGE QUEUE REACHED $V_QUEUE_ENTRIES MESSAGES - MAIL SERVER SHUT DOWN\n\n" > /tmp/message.txt
/usr/bin/mail -s "ERROR - QMAIL MESSAGE QUEUE EXCEEDED $V_SIZE_LIMIT MESSAGES - MAIL SERVER SHUT DOWN" "$V_MAIL_TO" < /tmp/message.txt
rm /tmp/message.txt
sleep 2
/usr/sbin/service qmail stop
V_RUN=false
fi
sleep 15
done
echo -e `date +"%d/%m/%Y %H:%M:%S"` ": Monitoring finished.\n\n"


On 27/05/2014 20:09, Joshua Megerman wrote:



Hi All,



I need some insight on fighting against the problem of "stolen passwords".



I run a "standard" netqmail server using the qmail-smtp-auth patch with

vpopmail vchkpw which worked fairly well the last years. I get more and

more trouble these days with stolen passwords, used to deliver tons of

spam to my server to be relayed to external users.



Today, I had a single client delivering about 2500 mails within a minute

before my queue stats detected it. My best idea is to rate-limit a user

to not send more than X mails within a period of time and perhaps even

block the user after a certain number of mails. Any ideas/plugins on how

to achieve that?



Any further ideas are welcome.



I don't have any immediate suggestions other than to put an increasing

sleep() call between completion of receipt of an email and accepting the

next one, but that doesn't help for emails with multiple recipients.

Although an increasing sleep() call between recipients after a certain

number might work too, assuming it won't cause problems for legitimate

mails.



I too have this happen to me on occasion, usually when I'm unable to even

find out about it for a day or two :( If you come up with a solution I'd

be very interested to see it.



Thanks,

Josh



Joshua Megerman

SJGames MIB #5273 - OGRE AI Testing Division

You can't win; You can't break even; You can't even quit the game.

- Layman's translation of the Laws of Thermodynamics

qmail@honorablemenschen.com<mailto:qmail@honorablemenschen.com>





--
________________________________
Verndale Systems Ltd Reg Office: 2a Grove Parade, Buxton, High Peak SK17 6AJ. Company No. 2376672 All Correspondence To: 71 High Street, Fareham PO16 7BB.
This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. E-mail transmission cannot be assured to be secure or correct as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or contain viruses. The sender therefore does not accept liability for any errors or omissions in the contents of this message, which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. (c) 2005 Verndale Systems Ltd. All rights reserved.

Thus said Oliver Welter on Tue, 27 May 2014 20:09:51 +0200:

> I run a "standard" netqmail server using the qmail-smtp-auth patch
> with vpopmail vchkpw which worked fairly well the last years. I get
> more and more trouble these days with stolen passwords, used to
> deliver tons of spam to my server to be relayed to external users.

There used to be a tarpit patch. I don't recall if the tarpit was per
IP, or per recipient, or perhaps both. At any rate, this would certainly
slow down if not stop spam originating from authenticated users.

You could also use a firewall rule that blocks/drops the connection
after so many connections within a given period of time. Less effective
if they relay to thousands of recipients on a single connection.

Andy
--
TAI64 timestamp: 4000000053854fed

On Tuesday 27 May 2014 20:54:02 you wrote:
> Thus said Oliver Welter on Tue, 27 May 2014 20:09:51 +0200:
> > I run a "standard" netqmail server using the qmail-smtp-auth patch
> > with vpopmail vchkpw which worked fairly well the last years. I get
> > more and more trouble these days with stolen passwords, used to
> > deliver tons of spam to my server to be relayed to external users.
>
> There used to be a tarpit patch. I don't recall if the tarpit was per
> IP, or per recipient, or perhaps both. At any rate, this would certainly
> slow down if not stop spam originating from authenticated users.
>
> You could also use a firewall rule that blocks/drops the connection
> after so many connections within a given period of time. Less effective
> if they relay to thousands of recipients on a single connection.

To add to Andy's thoughts above,

The tarpit patch is still there, listed on qmail.org.

http://www.palomine.net/qmail/tarpit.html

a more complex approach is here,

http://spamthrottle.qmail.ca/

Early-talker / Greetdelay patches may be of interest, here's mine; links to
some other implementations also given here,

http://free.acrconsulting.co.uk/email/other.html#earlytalker

These may not address the heart of your problem though...

In your position I'd be considering splitting incoming ("MX") (port 25) email
from mail submission (typically port 587) if you've not already done so: Your
issue relates to the mail submission side. Then I'd explore how to rate-limit
the incoming mail submisison connections you're seeing. Part of this will be
to establish the type of abuse:

- Is each SMTP session a single transaction, or is the session kept
open for multiple messages?

- Are emails sent to single recipients, or is one message sent to
multiple recipients (RCPT TO)?

- Are multiple concurrent SMTP sessions used to pump in the emails?

- If multiple concurrent sessions are used, are they from the same IP
address or different IP addresses?

The answers to the above will determine which parts of your setup to
concentrate on re. rate-limiting, and whether the patches above will be useful
or not.

You assert that the trouble is with 'stolen passwords', however are you sure
about this - and how are they being stolen? Is it possible that passwords are
being cracked instead - esp. considering how poor some people's passwords can
be? Given that, I'd consider checking on the quality of your users' passwords.
Depending on the version of vpopmail, these may be stored unencrypted (that's
a bad thing in general, although it provides a shortcut here for you to
determine how good or bad your users' passwords are); alternatively try using
a tool to try testing for weak passwords, so that you find them before an
abuser does.

Plenty to think about there, I hope that's helpful.

Andrew.
--
====================================================================
* Custom email solutions * Systems Administration * Networking
http://www.acrconsulting.co.uk/email/qmail.html
====================================================================

Hi Oliver,

though most have been said about your issue, just let me comment the following:

a) AFAIK, there is no special patch to limit the number of RCPT TO per Auth-user/relay-client.
My Spamcontrol patch allows to use a MAXRECPIENT limit for any connection.

b) You won't get happy in this case, unless you log the SMTP session with some granularity and
identify/omit the problem at the source.

c) In case, the abuse of the mail account is realized by different 'Mail From:' return-paths, my MAV
patch (included as well in Spamcontrol) might be an useful tool.

--

* In any case, sending userids/passwords of the Internet requires encryption (even better with
CRAM-MD5/APOP).
* Ask your user to change their passwords and move to TLS/StartTTLS.
* You probably will get more happy using the Submission port.

--

Thus, your issue is not only a technical one but rather architectural one.

Have a look at my Spamcontrol patch for qmail, since it copes with all of those issues.

http://www.fehcom.de/qmail.html


regards.
--eh.
On Tue, 27 May 2014 20:09:51 +0200, Oliver Welter <mail@oliwel.de> wrote :

> Hi All,
>
> I need some insight on fighting against the problem of "stolen passwords".
>
> I run a "standard" netqmail server using the qmail-smtp-auth patch with
> vpopmail vchkpw which worked fairly well the last years. I get more and
> more trouble these days with stolen passwords, used to deliver tons of
> spam to my server to be relayed to external users.
>
> Today, I had a single client delivering about 2500 mails within a minute
> before my queue stats detected it. My best idea is to rate-limit a user
> to not send more than X mails within a period of time and perhaps even
> block the user after a certain number of mails. Any ideas/plugins on how
> to achieve that?
>
> Any further ideas are welcome.
>
> Oliver
>
> --
> Protect your environment - close windows and adopt a penguin!
>
>
>
>

--
Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/

Hi All,

thanks for all the answers - none of them really seems to help me out as
they address the problem at the wrong end or will have side effects on
legitimate use cases.

Am 28.05.2014 22:32, schrieb Erwin Hoffmann:

> a) AFAIK, there is no special patch to limit the number of RCPT TO per Auth-user/relay-client.
> My Spamcontrol patch allows to use a MAXRECPIENT limit for any connection.

This was exactly what I am looking for....

> b) You won't get happy in this case, unless you log the SMTP session with some granularity and
> identify/omit the problem at the source.

The problem IS identified - I have a split setup with seperate MX boxes
which have a full load of Anti-Spam tools on board and do a quite good
job. This box is the local mail storage system which I also use for my
own users to relay mail because the whole user/authentication stuff is
already there. As most of my users are cooperative I do not put any
blocks on that system - the current problems arise from the fact that
some evil people know the passwords of legitimate accounts. As soon as
my research went with some affected accounts, the users had either a
trojan on their home computer or used the same password in other places.

> c) In case, the abuse of the mail account is realized by different 'Mail From:' return-paths, my MAV
> patch (included as well in Spamcontrol) might be an useful tool.

As I need to refresh my system, I will consider embeding your patch and
try that out.

> * In any case, sending userids/passwords of the Internet requires encryption (even better with
> CRAM-MD5/APOP).
> * Ask your user to change their passwords and move to TLS/StartTTLS.

They are just to lazy and/or stupid and not aware of the problems.

> * You probably will get more happy using the Submission port.
see above - users are not willing to change their settings and in some
env they are even unable to use it due to firewalling issued (even
admins are sometimes ignorants).

Oli
--
Protect your environment - close windows and adopt a penguin!

It should also be possible to limit the number of new inbound port 25 connections from a specific IP

I do something similar with respect to UDP/53 queries against our authoritive DNS servers to prevent reflection attacks
(basically it says if you get 12 hits in 75 seconds drop the connection).

-A INPUT -p udp -m udp -m recent -i eth0 --dport 53 --update --seconds 75 --hitcount 12 --name DNSTHROTTLE --rsource -j DROP
-A INPUT -p udp -m udp -m recent -i eth0 --dport 53 -j ACCEPT --set --name DNSTHROTTLE --rsource



I'm guessing a couple of iptables like this would solve your problem; without killing casual users (who don't send more than 12/75 seconds)

iptables -A INPUT -p tcp -i eth0 --dport 25 -m tcp -m state --state NEW -m recent --update --seconds 75 --hitcount 12 --name MAILTHROTTLE --rsource -j DROP
iptables -A INPUT -p tcp -i eth0 --dport 25 -m tcp -m state -j ACCEPT -- set --name MAILTHROTTLE --rsource



Scott Brynen
direct +1.604.638.9804 mobile +1.778.788.0543
web visioncritical.com

-----Original Message-----
From: Oliver Welter [mailto:mail@oliwel.de]
Sent: Thursday, 29 May, 2014 12:51 AM
To: qmail@list.cr.yp.to
Subject: Re: rate-limit relay clients

Hi All,

thanks for all the answers - none of them really seems to help me out as
they address the problem at the wrong end or will have side effects on
legitimate use cases.

Am 28.05.2014 22:32, schrieb Erwin Hoffmann:

> a) AFAIK, there is no special patch to limit the number of RCPT TO per Auth-user/relay-client.
> My Spamcontrol patch allows to use a MAXRECPIENT limit for any connection.

This was exactly what I am looking for....

> b) You won't get happy in this case, unless you log the SMTP session with some granularity and
> identify/omit the problem at the source.

The problem IS identified - I have a split setup with seperate MX boxes
which have a full load of Anti-Spam tools on board and do a quite good
job. This box is the local mail storage system which I also use for my
own users to relay mail because the whole user/authentication stuff is
already there. As most of my users are cooperative I do not put any
blocks on that system - the current problems arise from the fact that
some evil people know the passwords of legitimate accounts. As soon as
my research went with some affected accounts, the users had either a
trojan on their home computer or used the same password in other places.

> c) In case, the abuse of the mail account is realized by different 'Mail From:' return-paths, my MAV
> patch (included as well in Spamcontrol) might be an useful tool.

As I need to refresh my system, I will consider embeding your patch and
try that out.

> * In any case, sending userids/passwords of the Internet requires encryption (even better with
> CRAM-MD5/APOP).
> * Ask your user to change their passwords and move to TLS/StartTTLS.

They are just to lazy and/or stupid and not aware of the problems.

> * You probably will get more happy using the Submission port.
see above - users are not willing to change their settings and in some
env they are even unable to use it due to firewalling issued (even
admins are sometimes ignorants).

Oli
--
Protect your environment - close windows and adopt a penguin!