Mailing List Archive: [lvs-users] passive ftp configuration issue

[lvs-users] passive ftp configuration issue

Mar 24, 2014, 9:40 PM

Post #1 of 4 (2923 views)

Hi,

I am following http://www.centos.org/docs/4/html/rh-cs-en-4/s1-lvs-ftp.html
to set ftp virtual server, passive ftp not working , the active ftp
works fine:

-----LVS config

-A -t 192.168.1.169:21 -s rr
-a -t 192.168.1.169:21 -r 192.168.3.2:21 -m
-a -t 192.168.1.169:21 -r 192.168.3.3:21 -m

# ipvsadm -L
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 192.168.1.169:ftp rr
-> 192.168.3.2:ftp Masq 1 0 0
-> 192.168.3.3:ftp Masq 1 0 0

iptables rules

/sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
--dport 21 -j MARK --set-mark 21
/sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
--dport 10000:20000 -j MARK --set-mark 21

-----real server vsftpd related config

pasv_address=192.168.1.169
pasv_min_port=10000
pasv_max_port=20000

tcpdump for passive ftp data connection shows :

21 0.420062 192.168.1.9 192.168.1.169 TCP 74 50641 > 11196 [SYN] Seq=0
Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1491521 TSecr=0 WS=32

25 3.425842 192.168.1.169 192.168.1.9 ICMP 102 Destination unreachable
(Host unreachable)

Transmission Control Protocol, Src Port: 50641 (50641), Dst Port:
11196 (11196), Seq: 4055858902

I extracted out the tcp part of the icmp host unreachable message
above, as you can see, it indicates that LVS seems has no
listener/virtual for the client ftp data connection '50641 > 11196
[SYN]', I thought the iptable mangle rules should handle the passive
ftp data connection, it appears not. am I miss anything?

I am running the latest Linux 3.14.0-rc7+, all the related
netfilter/ipvs ftp modules are loaded properly, netfilter/ipvs
connection tracking are enabled too.

Regards,

Vincent

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] passive ftp configuration issue [ In reply to ]

malcolm at loadbalancer

Mar 24, 2014, 11:52 PM

Post #2 of 4 (2826 views)

Permalink

Vincent,

In NAT mode, the
ip_vs_ftp module is used to ensure that the client connects back via
the load balancer rather than attempting
to connect directly to the Real Server. Have you got the module loaded?

You only need the firewall rules if you are in DR/gate mode.

On 25 March 2014 04:40, Vincent Li <vincent.mc.li@gmail.com> wrote:
> Hi,
>
> I am following http://www.centos.org/docs/4/html/rh-cs-en-4/s1-lvs-ftp.html
> to set ftp virtual server, passive ftp not working , the active ftp
> works fine:
>
> -----LVS config
>
>
> -A -t 192.168.1.169:21 -s rr
> -a -t 192.168.1.169:21 -r 192.168.3.2:21 -m
> -a -t 192.168.1.169:21 -r 192.168.3.3:21 -m
>
> # ipvsadm -L
> IP Virtual Server version 1.2.1 (size=4096)
> Prot LocalAddress:Port Scheduler Flags
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> TCP 192.168.1.169:ftp rr
> -> 192.168.3.2:ftp Masq 1 0 0
> -> 192.168.3.3:ftp Masq 1 0 0
>
>
> iptables rules
>
> /sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
> --dport 21 -j MARK --set-mark 21
> /sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
> --dport 10000:20000 -j MARK --set-mark 21
>
> -----real server vsftpd related config
>
> pasv_address=192.168.1.169
> pasv_min_port=10000
> pasv_max_port=20000
>
>
> tcpdump for passive ftp data connection shows :
>
> 21 0.420062 192.168.1.9 192.168.1.169 TCP 74 50641 > 11196 [SYN] Seq=0
> Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1491521 TSecr=0 WS=32
>
> 25 3.425842 192.168.1.169 192.168.1.9 ICMP 102 Destination unreachable
> (Host unreachable)
>
> Transmission Control Protocol, Src Port: 50641 (50641), Dst Port:
> 11196 (11196), Seq: 4055858902
>
> I extracted out the tcp part of the icmp host unreachable message
> above, as you can see, it indicates that LVS seems has no
> listener/virtual for the client ftp data connection '50641 > 11196
> [SYN]', I thought the iptable mangle rules should handle the passive
> ftp data connection, it appears not. am I miss anything?
>
> I am running the latest Linux 3.14.0-rc7+, all the related
> netfilter/ipvs ftp modules are loaded properly, netfilter/ipvs
> connection tracking are enabled too.
>
> Regards,
>
> Vincent
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] passive ftp configuration issue [ In reply to ]

vincent.mc.li at gmail

Mar 25, 2014, 12:28 PM

Post #3 of 4 (2829 views)

Permalink

Yes, ip_vs_ftp module is loaded properly

# lsmod | grep nf
nf_conntrack_ipv4 16212 2
nf_defrag_ipv4 1710 1 nf_conntrack_ipv4
nf_nat 16229 1 ip_vs_ftp
nf_conntrack_ipv6 10295 2
nf_defrag_ipv6 27851 1 nf_conntrack_ipv6
nf_conntrack 93079 5
nf_conntrack_ipv4,nf_nat,ip_vs,nf_conntrack_ipv6,xt_state
ipv6 359467 53
bridge,ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6,[permanent]

# lsmod | grep ftp
ip_vs_ftp 6731 0
nf_nat 16229 1 ip_vs_ftp
ip_vs 157311 6 ip_vs_ftp,ip_vs_rr

it looks to me when client start the ftp data connection with SYN with
ephemeral port, the director has no idea how to handle this ftp data
connection, there is no listener 192.168.1.169:11196 on director to
handle the SYN packet, my understanding is that the iptables mangle
rule will set a firewall mark on 192.168.1.169:11196 so director knows
it should be some connection relate to vip 192.168.1.169:21 and LVS
should handle it, or I am missing something?

On Mon, Mar 24, 2014 at 11:52 PM, Malcolm Turnbull
<malcolm@loadbalancer.org> wrote:
> Vincent,
>
> In NAT mode, the
> ip_vs_ftp module is used to ensure that the client connects back via
> the load balancer rather than attempting
> to connect directly to the Real Server. Have you got the module loaded?
>
> You only need the firewall rules if you are in DR/gate mode.
>
>
>
>
>
> On 25 March 2014 04:40, Vincent Li <vincent.mc.li@gmail.com> wrote:
>> Hi,
>>
>> I am following http://www.centos.org/docs/4/html/rh-cs-en-4/s1-lvs-ftp.html
>> to set ftp virtual server, passive ftp not working , the active ftp
>> works fine:
>>
>> -----LVS config
>>
>>
>> -A -t 192.168.1.169:21 -s rr
>> -a -t 192.168.1.169:21 -r 192.168.3.2:21 -m
>> -a -t 192.168.1.169:21 -r 192.168.3.3:21 -m
>>
>> # ipvsadm -L
>> IP Virtual Server version 1.2.1 (size=4096)
>> Prot LocalAddress:Port Scheduler Flags
>> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>> TCP 192.168.1.169:ftp rr
>> -> 192.168.3.2:ftp Masq 1 0 0
>> -> 192.168.3.3:ftp Masq 1 0 0
>>
>>
>> iptables rules
>>
>> /sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
>> --dport 21 -j MARK --set-mark 21
>> /sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
>> --dport 10000:20000 -j MARK --set-mark 21
>>
>> -----real server vsftpd related config
>>
>> pasv_address=192.168.1.169
>> pasv_min_port=10000
>> pasv_max_port=20000
>>
>>
>> tcpdump for passive ftp data connection shows :
>>
>> 21 0.420062 192.168.1.9 192.168.1.169 TCP 74 50641 > 11196 [SYN] Seq=0
>> Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1491521 TSecr=0 WS=32
>>
>> 25 3.425842 192.168.1.169 192.168.1.9 ICMP 102 Destination unreachable
>> (Host unreachable)
>>
>> Transmission Control Protocol, Src Port: 50641 (50641), Dst Port:
>> 11196 (11196), Seq: 4055858902
>>
>> I extracted out the tcp part of the icmp host unreachable message
>> above, as you can see, it indicates that LVS seems has no
>> listener/virtual for the client ftp data connection '50641 > 11196
>> [SYN]', I thought the iptable mangle rules should handle the passive
>> ftp data connection, it appears not. am I miss anything?
>>
>> I am running the latest Linux 3.14.0-rc7+, all the related
>> netfilter/ipvs ftp modules are loaded properly, netfilter/ipvs
>> connection tracking are enabled too.
>>
>> Regards,
>>
>> Vincent
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)870 443 8779
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

Re: [lvs-users] passive ftp configuration issue [ In reply to ]

vincent.mc.li at gmail

Mar 25, 2014, 9:43 PM

Post #4 of 4 (2852 views)

Permalink

did further testing, indeed the the NAT mode has nothing to do with
the iptable firewall mark setting, I made it working by comment out
the vsftpd.conf configuration

#pasv_address=192.168.1.169

originally I had the pasv_address pointed to VIP 192.168.1.169,
commented that line made the ftp virtual server work for passive ftp,
reading the code in ip_vs_ftp.c, it will replace original "227
Entering Passive Mode (xxx,xxx,xxx,xxx,ppp,ppp) " from real server
with "227 Entering Passive Mode (yyy,yyy,yyy,yyy,ppp,ppp)", which xxx
is octet of RIP and yyy is octet of VIP, so "pasv_address =
192.168.1.169" is not nessary, but still I don't understand why
configuring the unneeded pasv_address config would break virtual ftp
server passive mode.

Vincent

On Tue, Mar 25, 2014 at 12:28 PM, Vincent Li <vincent.mc.li@gmail.com> wrote:
> Yes, ip_vs_ftp module is loaded properly
>
> # lsmod | grep nf
> nf_conntrack_ipv4 16212 2
> nf_defrag_ipv4 1710 1 nf_conntrack_ipv4
> nf_nat 16229 1 ip_vs_ftp
> nf_conntrack_ipv6 10295 2
> nf_defrag_ipv6 27851 1 nf_conntrack_ipv6
> nf_conntrack 93079 5
> nf_conntrack_ipv4,nf_nat,ip_vs,nf_conntrack_ipv6,xt_state
> ipv6 359467 53
> bridge,ip_vs,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6,[permanent]
>
> # lsmod | grep ftp
> ip_vs_ftp 6731 0
> nf_nat 16229 1 ip_vs_ftp
> ip_vs 157311 6 ip_vs_ftp,ip_vs_rr
>
> it looks to me when client start the ftp data connection with SYN with
> ephemeral port, the director has no idea how to handle this ftp data
> connection, there is no listener 192.168.1.169:11196 on director to
> handle the SYN packet, my understanding is that the iptables mangle
> rule will set a firewall mark on 192.168.1.169:11196 so director knows
> it should be some connection relate to vip 192.168.1.169:21 and LVS
> should handle it, or I am missing something?
>
>
>
> On Mon, Mar 24, 2014 at 11:52 PM, Malcolm Turnbull
> <malcolm@loadbalancer.org> wrote:
>> Vincent,
>>
>> In NAT mode, the
>> ip_vs_ftp module is used to ensure that the client connects back via
>> the load balancer rather than attempting
>> to connect directly to the Real Server. Have you got the module loaded?
>>
>> You only need the firewall rules if you are in DR/gate mode.
>>
>>
>>
>>
>>
>> On 25 March 2014 04:40, Vincent Li <vincent.mc.li@gmail.com> wrote:
>>> Hi,
>>>
>>> I am following http://www.centos.org/docs/4/html/rh-cs-en-4/s1-lvs-ftp.html
>>> to set ftp virtual server, passive ftp not working , the active ftp
>>> works fine:
>>>
>>> -----LVS config
>>>
>>>
>>> -A -t 192.168.1.169:21 -s rr
>>> -a -t 192.168.1.169:21 -r 192.168.3.2:21 -m
>>> -a -t 192.168.1.169:21 -r 192.168.3.3:21 -m
>>>
>>> # ipvsadm -L
>>> IP Virtual Server version 1.2.1 (size=4096)
>>> Prot LocalAddress:Port Scheduler Flags
>>> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>>> TCP 192.168.1.169:ftp rr
>>> -> 192.168.3.2:ftp Masq 1 0 0
>>> -> 192.168.3.3:ftp Masq 1 0 0
>>>
>>>
>>> iptables rules
>>>
>>> /sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
>>> --dport 21 -j MARK --set-mark 21
>>> /sbin/iptables -t mangle -A PREROUTING -p tcp -d 192.168.1.169/32
>>> --dport 10000:20000 -j MARK --set-mark 21
>>>
>>> -----real server vsftpd related config
>>>
>>> pasv_address=192.168.1.169
>>> pasv_min_port=10000
>>> pasv_max_port=20000
>>>
>>>
>>> tcpdump for passive ftp data connection shows :
>>>
>>> 21 0.420062 192.168.1.9 192.168.1.169 TCP 74 50641 > 11196 [SYN] Seq=0
>>> Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1491521 TSecr=0 WS=32
>>>
>>> 25 3.425842 192.168.1.169 192.168.1.9 ICMP 102 Destination unreachable
>>> (Host unreachable)
>>>
>>> Transmission Control Protocol, Src Port: 50641 (50641), Dst Port:
>>> 11196 (11196), Seq: 4055858902
>>>
>>> I extracted out the tcp part of the icmp host unreachable message
>>> above, as you can see, it indicates that LVS seems has no
>>> listener/virtual for the client ftp data connection '50641 > 11196
>>> [SYN]', I thought the iptable mangle rules should handle the passive
>>> ftp data connection, it appears not. am I miss anything?
>>>
>>> I am running the latest Linux 3.14.0-rc7+, all the related
>>> netfilter/ipvs ftp modules are loaded properly, netfilter/ipvs
>>> connection tracking are enabled too.
>>>
>>> Regards,
>>>
>>> Vincent
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>>> Send requests to lvs-users-request@LinuxVirtualServer.org
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>>
>>
>> --
>> Regards,
>>
>> Malcolm Turnbull.
>>
>> Loadbalancer.org Ltd.
>> Phone: +44 (0)870 443 8779
>> http://www.loadbalancer.org/
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users