Mailing List Archive: bug(s) in ipvs-0.2.4

bug(s) in ipvs-0.2.4

Feb 22, 2001, 4:18 AM

Post #1 of 7 (1642 views)

Hi,

While trying to implement a load-balancer based on ipvs-0.2.4 I found some
bugs in the code. The testing environment used is: kernel 2.4.2-pre[34] for
load balancer and NAT as load-balancing method

1. TCP state transition diagrams in ipvs/ip_vs_conn.c have initial states in
the wrong order (according to ipvs/ip_vs.h). The problem is that
IP_VS_S_SYNACK should be the 5th state in the table and it is the last. I have
observed this because a close request initiated by the client leaves the
connection open on the director. Here are the debug messages:

IPVS: Schedule fwd:M s:NONE c:172.18.1.62:3206 v:172.18.1.58:21 d:172.18.1.231:21 flg:140 cnt:2
IPVS: TCP input [S...] 172.18.1.231:21->172.18.1.62:3206 state: NONE->SYN_RECV cnt:2
IPVS: TCP input [..A.] 172.18.1.231:21->172.18.1.62:3206 state: SYN_RECV->ESTABLISHED cnt:2
IPVS: TCP input [.FA.] 172.18.1.231:21->172.18.1.62:3206 state: ESTABLISHED->CLOSE_WAIT cnt:2
IPVS: TCP output [..A.] 172.18.1.231:21->172.18.1.62:3206 state: CLOSE_WAIT->LAST_ACK cnt:2
IPVS: TCP output [.FA.] 172.18.1.231:21->172.18.1.62:3206 state: LAST_ACK->LISTEN cnt:2
IPVS: TCP input [..A.] 172.18.1.231:21->172.18.1.62:3206 state: LISTEN->ESTABLISHED cnt:2

You can see that the last 2 lines are not exactly a correct behaviour.

2. The ftp module does not work for passive mode (it doesn't work at all but
the active mode can be managed using standard masquerading). The FTP handling
code does not properly deal with the new connection (the director doesn't
handle it at all).

There is an attached patch that solves the two above-mentioned problems. I'm
not sure that the second problem is resolved the right way, but at least it
works.

Bye

Radu-Adrian Feurdean
mailto: raf@chez.com
-------------------------------------------------------------------
"If the night is silent enough you can hear a Windows NT rebooting"

Re: bug(s) in ipvs-0.2.4 [ In reply to ]

ja at ssi

Feb 22, 2001, 4:57 AM

Post #2 of 7 (1633 views)

Permalink

Hello,

On Thu, 22 Feb 2001, Radu-Adrian Feurdean wrote:

>
> Hi,
> While trying to implement a load-balancer based on ipvs-0.2.4 I found some
> bugs in the code. The testing environment used is: kernel 2.4.2-pre[34] for
> load balancer and NAT as load-balancing method
>
> 1. TCP state transition diagrams in ipvs/ip_vs_conn.c have initial states in
> the wrong order (according to ipvs/ip_vs.h). The problem is that
> IP_VS_S_SYNACK should be the 5th state in the table and it is the last. I have
> observed this because a close request initiated by the client leaves the
> connection open on the director. Here are the debug messages:
>
> IPVS: Schedule fwd:M s:NONE c:172.18.1.62:3206 v:172.18.1.58:21 d:172.18.1.231:21 flg:140 cnt:2
> IPVS: TCP input [S...] 172.18.1.231:21->172.18.1.62:3206 state: NONE->SYN_RECV cnt:2
> IPVS: TCP input [..A.] 172.18.1.231:21->172.18.1.62:3206 state: SYN_RECV->ESTABLISHED cnt:2
> IPVS: TCP input [.FA.] 172.18.1.231:21->172.18.1.62:3206 state: ESTABLISHED->CLOSE_WAIT cnt:2
> IPVS: TCP output [..A.] 172.18.1.231:21->172.18.1.62:3206 state: CLOSE_WAIT->LAST_ACK cnt:2
> IPVS: TCP output [.FA.] 172.18.1.231:21->172.18.1.62:3206 state: LAST_ACK->LISTEN cnt:2
> IPVS: TCP input [..A.] 172.18.1.231:21->172.18.1.62:3206 state: LISTEN->ESTABLISHED cnt:2
>
> You can see that the last 2 lines are not exactly a correct behaviour.
>
> 2. The ftp module does not work for passive mode (it doesn't work at all but
> the active mode can be managed using standard masquerading). The FTP handling
> code does not properly deal with the new connection (the director doesn't
> handle it at all).
>
> There is an attached patch that solves the two above-mentioned problems. I'm
> not sure that the second problem is resolved the right way, but at least it
> works.

Ignoring the fact your patch is broken the (1) is a _REAL_ bug.
We must move and insert the SA column before FW. It seems the
SA value is different from Linux 2.2 but the same transition table is
used :)

For (2) we have to analyze where is really the problem. Personally,
I didn't tried the ftp module from long time ago.

Thanks for this report.

> Bye
>
> Radu-Adrian Feurdean
> mailto: raf@chez.com
> -------------------------------------------------------------------
> "If the night is silent enough you can hear a Windows NT rebooting"

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: bug(s) in ipvs-0.2.4 [ In reply to ]

raf at chez

Feb 22, 2001, 5:31 AM

Post #3 of 7 (1622 views)

Permalink

On Thu, 22 Feb 2001, Julian Anastasov wrote:

> Ignoring the fact your patch is broken the (1) is a _REAL_ bug.
> We must move and insert the SA column before FW. It seems the

That's exactly what I have changed in the transition tables: inserting sSA
between sSR and sFW and commenting the previous occurences of sSA (last).
But, yes, all the patch is a little broken (needs -R to apply).

> For (2) we have to analyze where is really the problem. Personally,
> I didn't tried the ftp module from long time ago.

Hint: The client tries to establish the data conection, the director replies
with RST, the client closes the connection. If we filter outgoing RST it can
be observed that nothing happens; the packets are silently discarded because
the director doesn't NAT that connection.

> --
> Julian Anastasov <ja@ssi.bg>
>
> _______________________________________________
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://www.in-addr.de/mailman/listinfo/lvs-users

Radu-Adrian Feurdean
mailto: raf@chez.com
-------------------------------------------------------------------
"If the night is silent enough you can hear a Windows NT rebooting"

Re: bug(s) in ipvs-0.2.4 [ In reply to ]

ja at ssi

Feb 22, 2001, 5:55 AM

Post #4 of 7 (1625 views)

Permalink

Hello,

On Thu, 22 Feb 2001, Radu-Adrian Feurdean wrote:

>
> On Thu, 22 Feb 2001, Julian Anastasov wrote:
>
> > Ignoring the fact your patch is broken the (1) is a _REAL_ bug.
> > We must move and insert the SA column before FW. It seems the
>
> That's exactly what I have changed in the transition tables: inserting sSA
> between sSR and sFW and commenting the previous occurences of sSA (last).
> But, yes, all the patch is a little broken (needs -R to apply).

Oh, yes. But commenting sSA is not correct. It is valid state
that can be entered while the defense strategy is ON. Sometimes it
switches to OFF while the connection is still in sSA, so this is valid
state. Of course, in your case you never enter sSA :)

> > For (2) we have to analyze where is really the problem. Personally,
> > I didn't tried the ftp module from long time ago.
>
> Hint: The client tries to establish the data conection, the director replies
> with RST, the client closes the connection. If we filter outgoing RST it can
> be observed that nothing happens; the packets are silently discarded because
> the director doesn't NAT that connection.

I'll test it soon. May be the problem is that the data is not
detected in all cases, not sure. May be the problem is in the "227 ..."
text. I remember such fixes to go into netfilter. What "227 ..." string
reports your ftp server for passive mode?

> Radu-Adrian Feurdean
> mailto: raf@chez.com
> -------------------------------------------------------------------
> "If the night is silent enough you can hear a Windows NT rebooting"

Regards

--
Julian Anastasov <ja@ssi.bg>

Re: bug(s) in ipvs-0.2.4 [ In reply to ]

raf at chez

Feb 22, 2001, 6:26 AM

Post #5 of 7 (1624 views)

Permalink

On Thu, 22 Feb 2001, Julian Anastasov wrote:

> Oh, yes. But commenting sSA is not correct. It is valid state
> that can be entered while the defense strategy is ON. Sometimes it
> switches to OFF while the connection is still in sSA, so this is valid
> state. Of course, in your case you never enter sSA :)

So, you mean that it is necessary another sSA at the end (one on the 5th
position and one at the end) ? In the patch it IS present in the 5th position
in the table (moved from last pos to 5th pos).
Please explain a litle more if I'm wrong.

> I'll test it soon. May be the problem is that the data is not
> detected in all cases, not sure. May be the problem is in the "227 ..."
> text. I remember such fixes to go into netfilter. What "227 ..." string
> reports your ftp server for passive mode?

227 Entering Passive Mode (172,18,1,58,8,174).

Debug logs indicate that the ftp module intercepts and processes it, but the
insertion in the NAT rules is somehow wrong. The /proc/net/ip_vs_conn show
that connection in its correct state (NONE or SYN_RCVD), but the connection
isn't forwarded and NATed. Just look at the patch (where ^- is ^+ :)
Here's some debugging from a session that didn't work with passive mode (the
data connection was simply refused)

IPVS: Ftp: loaded support on port[0] = 21
IPVS: got PASV at 12 of 18
IPVS: PASV response (172.18.1.233:2737) -> 172.18.1.62:0 detected
IPVS: new PORT 172.18.1.58:2737
IPVS: warning: packet_xmit is null

Ah, btw, the ftp module in 0.1.2 (ready-for-production version) systematically
causes the director machine to crash when a PORT or PASV response is
encountered.

> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>

Radu-Adrian Feurdean
mailto: raf@zehc.net
-------------------------------------------------------------------
"If the night is silent enough you can hear a Windows NT rebooting"

Re: bug(s) in ipvs-0.2.4 [ In reply to ]

wensong at gnuchina

Feb 22, 2001, 7:33 AM

Post #6 of 7 (1628 views)

Permalink

Hello,

Thanks a lot for the report.

Just found another terrible bug introduced since ipvs-0.2.3, I forgot
unregistering the forward_icmp hook in the module exit.
nf_unregister_hook(&ip_vs_forward_icmp_ops);

Will trace the bug now, and hope that we will release a new version
soon.

Thanks,

Wensong

On Thu, 22 Feb 2001, Julian Anastasov wrote:

>
> Hello,
>
> On Thu, 22 Feb 2001, Radu-Adrian Feurdean wrote:
>
> >
> > Hi,
> > While trying to implement a load-balancer based on ipvs-0.2.4 I found some
> > bugs in the code. The testing environment used is: kernel 2.4.2-pre[34] for
> > load balancer and NAT as load-balancing method
> >
> > 1. TCP state transition diagrams in ipvs/ip_vs_conn.c have initial states in
> > the wrong order (according to ipvs/ip_vs.h). The problem is that
> > IP_VS_S_SYNACK should be the 5th state in the table and it is the last. I have
> > observed this because a close request initiated by the client leaves the
> > connection open on the director. Here are the debug messages:
> >
> > IPVS: Schedule fwd:M s:NONE c:172.18.1.62:3206 v:172.18.1.58:21 d:172.18.1.231:21 flg:140 cnt:2
> > IPVS: TCP input [S...] 172.18.1.231:21->172.18.1.62:3206 state: NONE->SYN_RECV cnt:2
> > IPVS: TCP input [..A.] 172.18.1.231:21->172.18.1.62:3206 state: SYN_RECV->ESTABLISHED cnt:2
> > IPVS: TCP input [.FA.] 172.18.1.231:21->172.18.1.62:3206 state: ESTABLISHED->CLOSE_WAIT cnt:2
> > IPVS: TCP output [..A.] 172.18.1.231:21->172.18.1.62:3206 state: CLOSE_WAIT->LAST_ACK cnt:2
> > IPVS: TCP output [.FA.] 172.18.1.231:21->172.18.1.62:3206 state: LAST_ACK->LISTEN cnt:2
> > IPVS: TCP input [..A.] 172.18.1.231:21->172.18.1.62:3206 state: LISTEN->ESTABLISHED cnt:2
> >
> > You can see that the last 2 lines are not exactly a correct behaviour.
> >
> > 2. The ftp module does not work for passive mode (it doesn't work at all but
> > the active mode can be managed using standard masquerading). The FTP handling
> > code does not properly deal with the new connection (the director doesn't
> > handle it at all).
> >
> > There is an attached patch that solves the two above-mentioned problems. I'm
> > not sure that the second problem is resolved the right way, but at least it
> > works.
>
> Ignoring the fact your patch is broken the (1) is a _REAL_ bug.
> We must move and insert the SA column before FW. It seems the
> SA value is different from Linux 2.2 but the same transition table is
> used :)
>
> For (2) we have to analyze where is really the problem. Personally,
> I didn't tried the ftp module from long time ago.
>
> Thanks for this report.
>
> > Bye
> >
> > Radu-Adrian Feurdean
> > mailto: raf@chez.com
> > -------------------------------------------------------------------
> > "If the night is silent enough you can hear a Windows NT rebooting"
>
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>

Re: bug(s) in ipvs-0.2.4 [ In reply to ]

raf at chez

Feb 22, 2001, 8:47 AM

Post #7 of 7 (1625 views)

Permalink

On Thu, 22 Feb 2001, Wensong Zhang wrote:

>
> Hello,
>
> Thanks a lot for the report.
>
> Just found another terrible bug introduced since ipvs-0.2.3, I forgot
> unregistering the forward_icmp hook in the module exit.
> nf_unregister_hook(&ip_vs_forward_icmp_ops);

That's why "rmmod ip_vs" crashes the machine ? I almos got used with it :)

> Will trace the bug now, and hope that we will release a new version
> soon.

Till then I collect patches :) You cand find them in one file at
http://www.chez.com/raf/ipvs.diff

Radu-Adrian Feurdean
mailto: raf@chez.com
-------------------------------------------------------------------
"If the night is silent enough you can hear a Windows NT rebooting"

Mailing List Archive

Mailing List Archive

Attached Files: