Mailing List Archive: Re: LVS and another DOS defense strategy

Re: LVS and another DOS defense strategy

Nov 21, 2000, 10:27 AM

Post #1 of 4 (1793 views)

Hi,

Ok, I step in a little bit late and may suggest something
that has already been discuss, please bear with me. But I'm
interested in security related questions, so ...

Anush Elangovan wrote:
>
> True, but with tools like Trin00 and TFN, they spoof the IP addresses randomly
> and in such a situation, an ICMP would be some basis to prioritize and drop
> connections. Even though the connection could be valid we are just looking at
> some criteria to drop connections when there are too many connections to
> handle.

I tend to say you should use a well configured packetfilter. Anyway the
tendency is not to open icmp-ping/pong through a router or firewall. Your
suggestion would only work if you open the whole path to the director.

> >
> > > Once we reach near system capacity to service connections we drop
> > > (and reset with the real server) the lower priority connections.
> >
> > Yes, the current activation scheme can be used or may be another
> > one.
>
> But, I guess the current scheme stops the connection right at the lvs
> director rarther than sending them to the real server and then reseting it
> incase of it being suspected as a SYN flood. The advantage here is that you
> save per connection processing time as the connection is initially sent thru
> and then RST if it is a SYN flood.

How do you know it is a SYN flood? You have nothing but the SYN from an
IP. RFC1918 addresses won't come that far anyway. So how do you know it's
a SYN flood?

> >
> > So, the main question is: do you want to guard the real servers
> > or you prefer to keep the LVS director not busy, just like the current
> > LVS defense strategies?
> >
>
> I am interested in total system availablity to legitimate users, this means I
> should guard the real servers. Since I RST the connection on the real server,
> the corresponding entry in the Director is also removed, the director will
> also be available to service more legitimate connections.

What if you RST a SYN ACK [CONN_ESTABLISHED on RS] connection from the
director? The RS state table is fucked.

>
> Though I didnot have in mind any idea about defering the packet in the
> director, I guess it could be used in VS-NAT, and in VS-DR where all the real
> server and director are in the same switched ethernet segment (if we can have
> some code to sniff out all the outgoing packets). But this is not my aim here.
>
> I would like to forward the connection immediately to the real
> server, while the validation is done. The connection is allowed to continue
> normally (until we find that the whole cluster is going to reach full capacity
> this could also be accounted for in the number of connections. Say, each real
> server can support x number of connections and we have 3 real servers, so
> when we are reaching like 98% of 3x we could start resetting lower priority
> connections. My point here is that we donot need to track the replies of the
> real server in DR/TUN. The initial SYN packet is what we mostly use to
> prioritize the connection. We will also use a little history here, like how
> many concurrent connections from one IP etc. The following packets like ACK
> etc will be used to adjust the connection priority.

Priority scheduling of TCP states can IMHO only be done reliable with
connection tracking. You let pass the SYN and wait for the SYN ACK.
If it's not there: RST. Generally don't allow anything else then a SYN
for a new TCP conn. state table entry. If the connection template exists
only allow FIN ACK, FIN and some others. The SYN RST would be dropped.
Check the struct masq_tcp_states_t masq_tcp_states [] in ../ipv4/ip_masq.c
to get the idea.

> > changing the states. But we move the load of these checks from the
> > moment of the attack to the packet handling code (i.e. even when there
> > is no attack). But smarter spoofing programs can fool the director
> > in DR and TUN mode by sending ACK packet after the initial SYN packet.
>
> Yes it is possible to fool the system, but it will be of no use unless the
> attacker knows that this is a cluster, and then finds out it is an LVS
> cluster. But if he doesnt, our system becomes more useful. If he does, his
> attack is worth only half as he needs to send an double the packets i.e. the
> SYN and then the ACK, which makes it harder as he needs double the number of
> machines to launch an attack of similar impact.

Fair enough :)

> Say Company A has an LVS cluster and it wants its staregic partner company B
> to be given priority access to the cluster, we could tell the cluster
> directory to give say priorty 0 to connections coming from the subnet company
> B has. This could also be used to give them access to say a dedicated real
> server/ real servers for their purpose. For example yahoo, has something
> similar where, a company's employees can access that company's specfic
> internal information via my.yahoo.com.

Gosh, can't wait for 2.4.x and the advanced IProuting features to be
released.

Best regards,
Roberto Nibali, ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`

Re: LVS and another DOS defense strategy [ In reply to ]

eas at asu

Nov 21, 2000, 12:25 PM

Post #2 of 4 (1723 views)

Permalink

Hi,

> I tend to say you should use a well configured packetfilter. Anyway the
> tendency is not to open icmp-ping/pong through a router or firewall. Your
> suggestion would only work if you open the whole path to the director.

The ICMP requests are just one of the methods to locate spoofed IP
addresses. Let us say that the ICMP replies from the client is blocked, then
we lower the priority but, that doesnt mean we will RST the connection. Only
when we reach 98% system capacity will we drop those connections. So in a
system without this it would be dropping of connections randomly, but here it
is [above other things] going to be connections that did not respond to icmp,
which gives us hope that the connections dropped "may" be part of a SYN
flood. But as Julian had suggested, if we can accumulate a block of IP which
reply to icmp and use them it would be very destructive. The point is, with
script kiddies just using off the shelf tools (which randomly spoof IPs) we
stand a good chance.

> How do you know it is a SYN flood? You have nothing but the SYN from an
> IP. RFC1918 addresses won't come that far anyway. So how do you know it's
> a SYN flood?

Actually speaking, I dont care if it is a SYN flood, unless my system is going
to reach peak capacity. Then, I start dropping low priority connections.But
there, let us say all requests are valid, then the proposed system is of no
use since all the priorities would be the same, and we would have a DOS
attack.

> What if you RST a SYN ACK [CONN_ESTABLISHED on RS] connection from the
> director? The RS state table is fucked.

Any suggestions on how to generate this above mentioned RST packet. Remember
in DR and TUN as Julian pointed out we dont know what the real server replies.

> Priority scheduling of TCP states can IMHO only be done reliable with
> connection tracking. You let pass the SYN and wait for the SYN ACK.
> If it's not there: RST. Generally don't allow anything else then a SYN
> for a new TCP conn. state table entry. If the connection template exists
> only allow FIN ACK, FIN and some others. The SYN RST would be dropped.
> Check the struct masq_tcp_states_t masq_tcp_states [] in ../ipv4/ip_masq.c
> to get the idea.

This could also be added as an additional check apart from the described
mechanism.

On a different note, since it is hard to reset a connection via the TCP RST
since we cant guess the sequence number, I was thinking of a way where the
Director tells the RS to reset a particular connection, and the RS does
it. I would like to know how a RS can reset a particular TCP connection, say
from a particular IP address?

Thanks
Anush

Re: LVS and another DOS defense strategy [ In reply to ]

ratz at tac

Nov 21, 2000, 1:13 PM

Post #3 of 4 (1715 views)

Permalink

Hi,

Anush Elangovan wrote:
>
> Hi,
>
> > I tend to say you should use a well configured packetfilter. Anyway the
> > tendency is not to open icmp-ping/pong through a router or firewall. Your
> > suggestion would only work if you open the whole path to the director.
>
> The ICMP requests are just one of the methods to locate spoofed IP
> addresses. Let us say that the ICMP replies from the client is blocked, then
> we lower the priority but, that doesnt mean we will RST the connection. Only
> when we reach 98% system capacity will we drop those connections. So in a

Hmm, agreed but consider all ICMP traffic blocked. For every incoming connection
we would lower the priority and if we got 98% saturation we start dropping. If a
lot of those connections were ok, then the resulting wrongly dropped packets
will
not differ a lot from randomly dropped packets (it's just more systematic). If
we have a lot of forged packets this will also result in lot wrongly dropped
packets since you can't verify with your method if they are faked or not. So
IMHO you gain nothing but overhead trying to ICMP-request the origin without
success. I stand corrected.

> system without this it would be dropping of connections randomly, but here it
> is [above other things] going to be connections that did not respond to icmp,

With the above situation it will drop also innocent packets.

> which gives us hope that the connections dropped "may" be part of a SYN
> flood. But as Julian had suggested, if we can accumulate a block of IP which
> reply to icmp and use them it would be very destructive. The point is, with
> script kiddies just using off the shelf tools (which randomly spoof IPs) we
> stand a good chance.

Sorry if I disagree but I'd rather configure my router, packetfilter and
firewall
correctly. This proofes quite useful in real world. People putting their lvs
node directly into the internet must be happy people anyway. I still think
the current implemented defense strategies along with the already existing
ones such as rp_filter, tcp_syncookies, tcp_retries?, tcp_max_syn_backlog
(this one you could use for linux realserver protection) should be quite
good. I don't know why we should include another not 100% reliable protection.

> Actually speaking, I dont care if it is a SYN flood, unless my system is going
> to reach peak capacity. Then, I start dropping low priority connections.But
> there, let us say all requests are valid, then the proposed system is of no
> use since all the priorities would be the same, and we would have a DOS
> attack.

I wonder how you achieve to priorize different DoS attacks? See, if I do
security audits and there is a, let's say, network-based IDS, I first try
out the normal script kiddie attacks and move over to DDos attacks and if
it still resets my connections I write my own packet-forger with libnet.
If you think your method provides more reliable right positives than the
existing implementation I'd like to know. I'd appreciate if you could
prove my assumptions wrong.

> > What if you RST a SYN ACK [CONN_ESTABLISHED on RS] connection from the
> > director? The RS state table is fucked.
>
> Any suggestions on how to generate this above mentioned RST packet. Remember
> in DR and TUN as Julian pointed out we dont know what the real server replies.

Oups, sorry, I though you'd send a RST to the (forged) CIP and not to
the RIP. Ok, you're right.

> > Priority scheduling of TCP states can IMHO only be done reliable with
> > connection tracking. You let pass the SYN and wait for the SYN ACK.
> > If it's not there: RST. Generally don't allow anything else then a SYN
> > for a new TCP conn. state table entry. If the connection template exists
> > only allow FIN ACK, FIN and some others. The SYN RST would be dropped.
> > Check the struct masq_tcp_states_t masq_tcp_states [] in ../ipv4/ip_masq.c
> > to get the idea.
>
> This could also be added as an additional check apart from the described
> mechanism.

IMHO this is the only way of reliably marking/dropping bad packets.

> On a different note, since it is hard to reset a connection via the TCP RST
> since we cant guess the sequence number, I was thinking of a way where the
> Director tells the RS to reset a particular connection, and the RS does
> it. I would like to know how a RS can reset a particular TCP connection, say
> from a particular IP address?

You have to track to connection along with ISN, state table and to sychronize
it or you could tell the RS that the connection template x is fake and he
therefore has to close the socket, but this will most probably not work. No,
if we use triangulation mode (VS-DR) we have no information about the TCP states
of different RS and therefore have to know by the first packet arriving the
director if it's fake or not.

It's getting interesting. Best regards,
Roberto Nibali, ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`