Mailing List Archive

[lvs-users] DR-TUN Issues: Packets become martians on real server
I am building a DR-TUN configuration to load balance DNS traffic. My issue
is with the real servers: the ipip packet get unwrapped and appears on the
tunnel interface and becomes a martian. Also, I have not been able to
determine how to bind to the tunnel interface to capture these packets. I
have verified details with tcpdump/dmesg. All my configuration is scripted
to ensure reproducibility. tcpdumps below do not show real DNS traffic;
using netcat to send text.

Much more detail can be supplied, please request it.

Configuration:

==============

- All machines are currently Debian 9 (4.9.30-2+deb9u2)

- Client: 192.168.200.10

- Director: 192.168.200.11

- Real Server: 192.168.200.12

- No VIP: I am using the IP address for my director interface -
192.168.200.11


Director Configuration:

==================

vagrant@debian-9-lb:~$ sudo ipvsadm -Ln

IP Virtual Server version 1.2.1 (size=4096)

Prot LocalAddress:Port Scheduler Flags

-> RemoteAddress:Port Forward Weight ActiveConn InActConn

UDP 192.168.200.11:53 rr

-> 192.168.200.12:53 Tunnel 1 0 0



Real Server:

============

# modprobe ipip

# echo 1 > /proc/sys/net/ipv4/ip_forward

# ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
192.168.200.11

# ip link set tunl1 up arp off

# echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter

# echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter

# echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter

# echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter



vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv

tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size 262144
bytes

10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags [DF], proto
UDP (17), length 40)

192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226 updateMA+
[b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]



If I add an IP address to the interface the packet never reaches tunl1

# ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1



Notes:

======



LVS-DR was not an option because the real servers will be in different
subnets (I was able to get LVS-DR working though).



I am looking for advice on how to better understand or resolve this issue.
Or, who would be better to answer this question.



I have groked as many articles on how to setup this configuration as I can.
And, I have read a good deal of the archives of this mailing list.



Linux distributions I have tried are Debian, Ubuntu and Amazon linux. This
is been tested on both AWS and VMware fusion.



------



I have a feeling I am missing something simple...



Thanks!

-Timothy
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Hi Timothy,

I've been watching this post to see if you get any replies, I too
recently had issues trying LVS-TUN in Amazon AWS... I was using an
IPsec VPN from home to my VPC attempting to load balance servers
across this link. Later, I tried as a sanity check against my
own(Proxmox) hosted server(In a local DC) using an OpenVPN link which
I also couldn't get to work... Although it works in the LAN with Linux
boxes as my routers when I try to simulate the network layout without
VPN's...

Other pressures eventually pulled me away from this task but I had
planned to go back to it at a later date so on seeing your post I've
been eagerly awaiting any replies you might get.

I have nothing to really offer at this stage as a resolution but
wanted to give the topic a bump at least, however, if I learn anything
on my next round of testing I'll share it.

>From my troubleshooting I think I proved that the traffic got as far
as the tunnel adapter on the real server but seemed to get lost on the
return path, I believe it's only the part between director and real
server that uses a tunnel with replies going directly back to the
client like DR mode so I was assuming(Maybe incorrectly) that
something was dropping them on the path back.

Aaron West

Loadbalancer.org

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
aaron@loadbalancer.org

LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG


On 14 July 2017 at 16:13, Timothy R. Weiand <timothy.weiand@gmail.com> wrote:
> I am building a DR-TUN configuration to load balance DNS traffic. My issue
> is with the real servers: the ipip packet get unwrapped and appears on the
> tunnel interface and becomes a martian. Also, I have not been able to
> determine how to bind to the tunnel interface to capture these packets. I
> have verified details with tcpdump/dmesg. All my configuration is scripted
> to ensure reproducibility. tcpdumps below do not show real DNS traffic;
> using netcat to send text.
>
> Much more detail can be supplied, please request it.
>
> Configuration:
>
> ==============
>
> - All machines are currently Debian 9 (4.9.30-2+deb9u2)
>
> - Client: 192.168.200.10
>
> - Director: 192.168.200.11
>
> - Real Server: 192.168.200.12
>
> - No VIP: I am using the IP address for my director interface -
> 192.168.200.11
>
>
> Director Configuration:
>
> ==================
>
> vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
>
> IP Virtual Server version 1.2.1 (size=4096)
>
> Prot LocalAddress:Port Scheduler Flags
>
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>
> UDP 192.168.200.11:53 rr
>
> -> 192.168.200.12:53 Tunnel 1 0 0
>
>
>
> Real Server:
>
> ============
>
> # modprobe ipip
>
> # echo 1 > /proc/sys/net/ipv4/ip_forward
>
> # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
> 192.168.200.11
>
> # ip link set tunl1 up arp off
>
> # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
>
> # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
>
> # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
>
> # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
>
>
>
> vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
>
> tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size 262144
> bytes
>
> 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags [DF], proto
> UDP (17), length 40)
>
> 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226 updateMA+
> [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
>
>
>
> If I add an IP address to the interface the packet never reaches tunl1
>
> # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
>
>
>
> Notes:
>
> ======
>
>
>
> LVS-DR was not an option because the real servers will be in different
> subnets (I was able to get LVS-DR working though).
>
>
>
> I am looking for advice on how to better understand or resolve this issue.
> Or, who would be better to answer this question.
>
>
>
> I have groked as many articles on how to setup this configuration as I can.
> And, I have read a good deal of the archives of this mailing list.
>
>
>
> Linux distributions I have tried are Debian, Ubuntu and Amazon linux. This
> is been tested on both AWS and VMware fusion.
>
>
>
> ------
>
>
>
> I have a feeling I am missing something simple...
>
>
>
> Thanks!
>
> -Timothy
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Hello,

On Fri, 14 Jul 2017, Timothy R. Weiand wrote:

> I am building a DR-TUN configuration to load balance DNS traffic. My issue
> is with the real servers: the ipip packet get unwrapped and appears on the
> tunnel interface and becomes a martian. Also, I have not been able to
> determine how to bind to the tunnel interface to capture these packets. I
> have verified details with tcpdump/dmesg. All my configuration is scripted
> to ensure reproducibility. tcpdumps below do not show real DNS traffic;
> using netcat to send text.
>
> Much more detail can be supplied, please request it.
>
> Configuration:
>
> ==============
>
> - All machines are currently Debian 9 (4.9.30-2+deb9u2)
>
> - Client: 192.168.200.10
>
> - Director: 192.168.200.11
>
> - Real Server: 192.168.200.12
>
> - No VIP: I am using the IP address for my director interface -
> 192.168.200.11
>
>
> Director Configuration:
>
> ==================
>
> vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
>
> IP Virtual Server version 1.2.1 (size=4096)
>
> Prot LocalAddress:Port Scheduler Flags
>
> -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>
> UDP 192.168.200.11:53 rr
>
> -> 192.168.200.12:53 Tunnel 1 0 0
>
>
>
> Real Server:
>
> ============
>
> # modprobe ipip
>
> # echo 1 > /proc/sys/net/ipv4/ip_forward
>
> # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
> 192.168.200.11
>
> # ip link set tunl1 up arp off
>
> # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
>
> # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
>
> # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
>
> # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter

tunl1/rp_filter=2 is ignored if there are no
IPv4 addresses configured on tunl1, you can see
__fib_validate_source() for reference:

if (no_addr)
goto last_resort;

As result, packets are dropped as martians due to rpf=2.

To check if packets are dropped by routing try:

ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1

Here is small howto for TUN (outdated):

http://ja.ssi.bg/TUN-HOWTO.txt

> vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
>
> tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size 262144
> bytes
>
> 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags [DF], proto
> UDP (17), length 40)
>
> 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226 updateMA+
> [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
>
>
>
> If I add an IP address to the interface the packet never reaches tunl1
>
> # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1

Try with (note mask /32):

ip addr add 192.168.200.11/32 dev tunl1

as shown in LVS HOWTO (8.3):

http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html

> Notes:
>
> ======
>
>
>
> LVS-DR was not an option because the real servers will be in different
> subnets (I was able to get LVS-DR working though).
>
>
>
> I am looking for advice on how to better understand or resolve this issue.
> Or, who would be better to answer this question.
>
>
>
> I have groked as many articles on how to setup this configuration as I can.
> And, I have read a good deal of the archives of this mailing list.
>
>
>
> Linux distributions I have tried are Debian, Ubuntu and Amazon linux. This
> is been tested on both AWS and VMware fusion.
>
>
>
> ------
>
>
>
> I have a feeling I am missing something simple...
>
>
>
> Thanks!
>
> -Timothy

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Julian:

Good news and bad news: this guidance helped me get LVS-TUN working with a
3.10.0 kernel. But, I am not able to use the same configuration to get
4.4.x,4.8.x or 4.12.x working. We are working on doing further analysis to
ensure it is kernel dependent and not distribution dependent. I hope to
find the kernel version that this configuration stops working.

Note on 4.x machines: If we do not have the correct ip assigned to the
tunnel interface we can see traffic on that interface. But, if we use the
correct VIP then we no longer see the traffic.

I will keep you updated as I find information out.

Thanks!
-Timothy

On Tue, Jul 25, 2017 at 4:25 PM Julian Anastasov <ja@ssi.bg> wrote:

>
> Hello,
>
> On Fri, 14 Jul 2017, Timothy R. Weiand wrote:
>
> > I am building a DR-TUN configuration to load balance DNS traffic. My
> issue
> > is with the real servers: the ipip packet get unwrapped and appears on
> the
> > tunnel interface and becomes a martian. Also, I have not been able to
> > determine how to bind to the tunnel interface to capture these packets. I
> > have verified details with tcpdump/dmesg. All my configuration is
> scripted
> > to ensure reproducibility. tcpdumps below do not show real DNS traffic;
> > using netcat to send text.
> >
> > Much more detail can be supplied, please request it.
> >
> > Configuration:
> >
> > ==============
> >
> > - All machines are currently Debian 9 (4.9.30-2+deb9u2)
> >
> > - Client: 192.168.200.10
> >
> > - Director: 192.168.200.11
> >
> > - Real Server: 192.168.200.12
> >
> > - No VIP: I am using the IP address for my director interface -
> > 192.168.200.11
> >
> >
> > Director Configuration:
> >
> > ==================
> >
> > vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
> >
> > IP Virtual Server version 1.2.1 (size=4096)
> >
> > Prot LocalAddress:Port Scheduler Flags
> >
> > -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> >
> > UDP 192.168.200.11:53 rr
> >
> > -> 192.168.200.12:53 Tunnel 1 0 0
> >
> >
> >
> > Real Server:
> >
> > ============
> >
> > # modprobe ipip
> >
> > # echo 1 > /proc/sys/net/ipv4/ip_forward
> >
> > # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
> > 192.168.200.11
> >
> > # ip link set tunl1 up arp off
> >
> > # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
> >
> > # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
> >
> > # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
> >
> > # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
>
> tunl1/rp_filter=2 is ignored if there are no
> IPv4 addresses configured on tunl1, you can see
> __fib_validate_source() for reference:
>
> if (no_addr)
> goto last_resort;
>
> As result, packets are dropped as martians due to rpf=2.
>
> To check if packets are dropped by routing try:
>
> ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1
>
> Here is small howto for TUN (outdated):
>
> http://ja.ssi.bg/TUN-HOWTO.txt
>
> > vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
> >
> > tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size 262144
> > bytes
> >
> > 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags [DF],
> proto
> > UDP (17), length 40)
> >
> > 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226
> updateMA+
> > [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
> >
> >
> >
> > If I add an IP address to the interface the packet never reaches tunl1
> >
> > # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
>
> Try with (note mask /32):
>
> ip addr add 192.168.200.11/32 dev tunl1
>
> as shown in LVS HOWTO (8.3):
>
> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
>
> > Notes:
> >
> > ======
> >
> >
> >
> > LVS-DR was not an option because the real servers will be in different
> > subnets (I was able to get LVS-DR working though).
> >
> >
> >
> > I am looking for advice on how to better understand or resolve this
> issue.
> > Or, who would be better to answer this question.
> >
> >
> >
> > I have groked as many articles on how to setup this configuration as I
> can.
> > And, I have read a good deal of the archives of this mailing list.
> >
> >
> >
> > Linux distributions I have tried are Debian, Ubuntu and Amazon linux.
> This
> > is been tested on both AWS and VMware fusion.
> >
> >
> >
> > ------
> >
> >
> >
> > I have a feeling I am missing something simple...
> >
> >
> >
> > Thanks!
> >
> > -Timothy
>
> Regards
>
> --
> Julian Anastasov <ja@ssi.bg>
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Hi Timothy,

What you had working was it in AWS or VMWare?

I've pretty much given up trying to get the spoofed reply traffic back
out of AWS it gets dropped always with either DR mode or TUN unless I
send it back via the director and that case I may as well do NAT mode
instead...

My understanding is that when egress traffic hits their edge they
check the source matches the instance it came from and if it doesn't
the packet gets dropped. However, all works fine if the client is in
the same VPC so it doesn't need to go out past the edge of the VPC.

Just interested if you found the same in your testing?
Aaron West

Loadbalancer.org

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
aaron@loadbalancer.org

LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG


On 10 August 2017 at 18:08, Timothy R. Weiand <timothy.weiand@gmail.com> wrote:
> Julian:
>
> Good news and bad news: this guidance helped me get LVS-TUN working with a
> 3.10.0 kernel. But, I am not able to use the same configuration to get
> 4.4.x,4.8.x or 4.12.x working. We are working on doing further analysis to
> ensure it is kernel dependent and not distribution dependent. I hope to
> find the kernel version that this configuration stops working.
>
> Note on 4.x machines: If we do not have the correct ip assigned to the
> tunnel interface we can see traffic on that interface. But, if we use the
> correct VIP then we no longer see the traffic.
>
> I will keep you updated as I find information out.
>
> Thanks!
> -Timothy
>
> On Tue, Jul 25, 2017 at 4:25 PM Julian Anastasov <ja@ssi.bg> wrote:
>
>>
>> Hello,
>>
>> On Fri, 14 Jul 2017, Timothy R. Weiand wrote:
>>
>> > I am building a DR-TUN configuration to load balance DNS traffic. My
>> issue
>> > is with the real servers: the ipip packet get unwrapped and appears on
>> the
>> > tunnel interface and becomes a martian. Also, I have not been able to
>> > determine how to bind to the tunnel interface to capture these packets. I
>> > have verified details with tcpdump/dmesg. All my configuration is
>> scripted
>> > to ensure reproducibility. tcpdumps below do not show real DNS traffic;
>> > using netcat to send text.
>> >
>> > Much more detail can be supplied, please request it.
>> >
>> > Configuration:
>> >
>> > ==============
>> >
>> > - All machines are currently Debian 9 (4.9.30-2+deb9u2)
>> >
>> > - Client: 192.168.200.10
>> >
>> > - Director: 192.168.200.11
>> >
>> > - Real Server: 192.168.200.12
>> >
>> > - No VIP: I am using the IP address for my director interface -
>> > 192.168.200.11
>> >
>> >
>> > Director Configuration:
>> >
>> > ==================
>> >
>> > vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
>> >
>> > IP Virtual Server version 1.2.1 (size=4096)
>> >
>> > Prot LocalAddress:Port Scheduler Flags
>> >
>> > -> RemoteAddress:Port Forward Weight ActiveConn InActConn
>> >
>> > UDP 192.168.200.11:53 rr
>> >
>> > -> 192.168.200.12:53 Tunnel 1 0 0
>> >
>> >
>> >
>> > Real Server:
>> >
>> > ============
>> >
>> > # modprobe ipip
>> >
>> > # echo 1 > /proc/sys/net/ipv4/ip_forward
>> >
>> > # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
>> > 192.168.200.11
>> >
>> > # ip link set tunl1 up arp off
>> >
>> > # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
>> >
>> > # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
>> >
>> > # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
>> >
>> > # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
>>
>> tunl1/rp_filter=2 is ignored if there are no
>> IPv4 addresses configured on tunl1, you can see
>> __fib_validate_source() for reference:
>>
>> if (no_addr)
>> goto last_resort;
>>
>> As result, packets are dropped as martians due to rpf=2.
>>
>> To check if packets are dropped by routing try:
>>
>> ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1
>>
>> Here is small howto for TUN (outdated):
>>
>> http://ja.ssi.bg/TUN-HOWTO.txt
>>
>> > vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
>> >
>> > tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size 262144
>> > bytes
>> >
>> > 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags [DF],
>> proto
>> > UDP (17), length 40)
>> >
>> > 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226
>> updateMA+
>> > [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
>> >
>> >
>> >
>> > If I add an IP address to the interface the packet never reaches tunl1
>> >
>> > # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
>>
>> Try with (note mask /32):
>>
>> ip addr add 192.168.200.11/32 dev tunl1
>>
>> as shown in LVS HOWTO (8.3):
>>
>> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
>>
>> > Notes:
>> >
>> > ======
>> >
>> >
>> >
>> > LVS-DR was not an option because the real servers will be in different
>> > subnets (I was able to get LVS-DR working though).
>> >
>> >
>> >
>> > I am looking for advice on how to better understand or resolve this
>> issue.
>> > Or, who would be better to answer this question.
>> >
>> >
>> >
>> > I have groked as many articles on how to setup this configuration as I
>> can.
>> > And, I have read a good deal of the archives of this mailing list.
>> >
>> >
>> >
>> > Linux distributions I have tried are Debian, Ubuntu and Amazon linux.
>> This
>> > is been tested on both AWS and VMware fusion.
>> >
>> >
>> >
>> > ------
>> >
>> >
>> >
>> > I have a feeling I am missing something simple...
>> >
>> >
>> >
>> > Thanks!
>> >
>> > -Timothy
>>
>> Regards
>>
>> --
>> Julian Anastasov <ja@ssi.bg>
>>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Aaron:

I was able to get LVS-TUN working in between subnets in AWS if the real
server is a 3.x kernel (RHEL to be specific). I had to set the destination
side of the tunnel and the address on that interface to the VIP. Then, I
configured unbound to listen for the VIP address.

We are working with RedHat to get the 4.x kernels fixed. But, no timeframe
on that fix.

FYI - We are going to work towards open sourcing our work so that everyone
can see exactly what we have done.

-Timothy

On Thu, Aug 17, 2017 at 10:56 AM Aaron West <aaron@loadbalancer.org> wrote:

> Hi Timothy,
>
> What you had working was it in AWS or VMWare?
>
> I've pretty much given up trying to get the spoofed reply traffic back
> out of AWS it gets dropped always with either DR mode or TUN unless I
> send it back via the director and that case I may as well do NAT mode
> instead...
>
> My understanding is that when egress traffic hits their edge they
> check the source matches the instance it came from and if it doesn't
> the packet gets dropped. However, all works fine if the client is in
> the same VPC so it doesn't need to go out past the edge of the VPC.
>
> Just interested if you found the same in your testing?
> Aaron West
>
> Loadbalancer.org
>
> www.loadbalancer.org
> +1 888 867 9504 / +44 (0)330 380 1064
> aaron@loadbalancer.org
>
> LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG
>
>
> On 10 August 2017 at 18:08, Timothy R. Weiand <timothy.weiand@gmail.com>
> wrote:
> > Julian:
> >
> > Good news and bad news: this guidance helped me get LVS-TUN working with
> a
> > 3.10.0 kernel. But, I am not able to use the same configuration to get
> > 4.4.x,4.8.x or 4.12.x working. We are working on doing further analysis
> to
> > ensure it is kernel dependent and not distribution dependent. I hope to
> > find the kernel version that this configuration stops working.
> >
> > Note on 4.x machines: If we do not have the correct ip assigned to the
> > tunnel interface we can see traffic on that interface. But, if we use the
> > correct VIP then we no longer see the traffic.
> >
> > I will keep you updated as I find information out.
> >
> > Thanks!
> > -Timothy
> >
> > On Tue, Jul 25, 2017 at 4:25 PM Julian Anastasov <ja@ssi.bg> wrote:
> >
> >>
> >> Hello,
> >>
> >> On Fri, 14 Jul 2017, Timothy R. Weiand wrote:
> >>
> >> > I am building a DR-TUN configuration to load balance DNS traffic. My
> >> issue
> >> > is with the real servers: the ipip packet get unwrapped and appears on
> >> the
> >> > tunnel interface and becomes a martian. Also, I have not been able to
> >> > determine how to bind to the tunnel interface to capture these
> packets. I
> >> > have verified details with tcpdump/dmesg. All my configuration is
> >> scripted
> >> > to ensure reproducibility. tcpdumps below do not show real DNS
> traffic;
> >> > using netcat to send text.
> >> >
> >> > Much more detail can be supplied, please request it.
> >> >
> >> > Configuration:
> >> >
> >> > ==============
> >> >
> >> > - All machines are currently Debian 9 (4.9.30-2+deb9u2)
> >> >
> >> > - Client: 192.168.200.10
> >> >
> >> > - Director: 192.168.200.11
> >> >
> >> > - Real Server: 192.168.200.12
> >> >
> >> > - No VIP: I am using the IP address for my director interface -
> >> > 192.168.200.11
> >> >
> >> >
> >> > Director Configuration:
> >> >
> >> > ==================
> >> >
> >> > vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
> >> >
> >> > IP Virtual Server version 1.2.1 (size=4096)
> >> >
> >> > Prot LocalAddress:Port Scheduler Flags
> >> >
> >> > -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> >> >
> >> > UDP 192.168.200.11:53 rr
> >> >
> >> > -> 192.168.200.12:53 Tunnel 1 0 0
> >> >
> >> >
> >> >
> >> > Real Server:
> >> >
> >> > ============
> >> >
> >> > # modprobe ipip
> >> >
> >> > # echo 1 > /proc/sys/net/ipv4/ip_forward
> >> >
> >> > # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
> >> > 192.168.200.11
> >> >
> >> > # ip link set tunl1 up arp off
> >> >
> >> > # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
> >> >
> >> > # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
> >> >
> >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
> >> >
> >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
> >>
> >> tunl1/rp_filter=2 is ignored if there are no
> >> IPv4 addresses configured on tunl1, you can see
> >> __fib_validate_source() for reference:
> >>
> >> if (no_addr)
> >> goto last_resort;
> >>
> >> As result, packets are dropped as martians due to rpf=2.
> >>
> >> To check if packets are dropped by routing try:
> >>
> >> ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1
> >>
> >> Here is small howto for TUN (outdated):
> >>
> >> http://ja.ssi.bg/TUN-HOWTO.txt
> >>
> >> > vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
> >> >
> >> > tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size
> 262144
> >> > bytes
> >> >
> >> > 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags [DF],
> >> proto
> >> > UDP (17), length 40)
> >> >
> >> > 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226
> >> updateMA+
> >> > [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
> >> >
> >> >
> >> >
> >> > If I add an IP address to the interface the packet never reaches tunl1
> >> >
> >> > # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
> >>
> >> Try with (note mask /32):
> >>
> >> ip addr add 192.168.200.11/32 dev tunl1
> >>
> >> as shown in LVS HOWTO (8.3):
> >>
> >> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
> >>
> >> > Notes:
> >> >
> >> > ======
> >> >
> >> >
> >> >
> >> > LVS-DR was not an option because the real servers will be in different
> >> > subnets (I was able to get LVS-DR working though).
> >> >
> >> >
> >> >
> >> > I am looking for advice on how to better understand or resolve this
> >> issue.
> >> > Or, who would be better to answer this question.
> >> >
> >> >
> >> >
> >> > I have groked as many articles on how to setup this configuration as I
> >> can.
> >> > And, I have read a good deal of the archives of this mailing list.
> >> >
> >> >
> >> >
> >> > Linux distributions I have tried are Debian, Ubuntu and Amazon linux.
> >> This
> >> > is been tested on both AWS and VMware fusion.
> >> >
> >> >
> >> >
> >> > ------
> >> >
> >> >
> >> >
> >> > I have a feeling I am missing something simple...
> >> >
> >> >
> >> >
> >> > Thanks!
> >> >
> >> > -Timothy
> >>
> >> Regards
> >>
> >> --
> >> Julian Anastasov <ja@ssi.bg>
> >>
> > _______________________________________________
> > Please read the documentation before posting - it's available at:
> > http://www.linuxvirtualserver.org/
> >
> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > Send requests to lvs-users-request@LinuxVirtualServer.org
> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Hi Timothy,

I've been testing ipip (Tun mode) and I have it working,

I note that the Weak ES Model needs to be accounted for and the tunl0
should be a /32 not a /24 as you have it on the real server.

ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1

becomes

ip addr add dev tunl1 192.168.200.11/32 brd 192.168.200.11 dev tunl1


I also have this in my /etc/sysctl.conf

net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.eth0.arp_ignore=1
net.ipv4.conf.tunl1.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
net.ipv4.conf.eth0.arp_announce=2
net.ipv4.conf.tunl1.arp_announce=2

net.ipv4.conf.all.log_martians=1
net.ipv4.conf.default.log_martians=1

And I ran this script portion to enable forwarding and disable the rp_filter

if [ $1 = "tunl1" ]; then
echo 1 > /proc/sys/net/ipv4/conf/tunl0/forwarding
echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
fi


This is with a 4.4.49 Kernel

I hope this help you


Andruw Smalley

Loadbalancer.org Ltd.

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
asmalley@loadbalancer.org

Leave a Review | Deployment Guides | Blog


On 22 August 2017 at 12:43, Timothy R. Weiand <timothy.weiand@gmail.com> wrote:
>
> Aaron:
>
> I was able to get LVS-TUN working in between subnets in AWS if the real
> server is a 3.x kernel (RHEL to be specific). I had to set the destination
> side of the tunnel and the address on that interface to the VIP. Then, I
> configured unbound to listen for the VIP address.
>
> We are working with RedHat to get the 4.x kernels fixed. But, no timeframe
> on that fix.
>
> FYI - We are going to work towards open sourcing our work so that everyone
> can see exactly what we have done.
>
> -Timothy
>
> On Thu, Aug 17, 2017 at 10:56 AM Aaron West <aaron@loadbalancer.org> wrote:
>
> > Hi Timothy,
> >
> > What you had working was it in AWS or VMWare?
> >
> > I've pretty much given up trying to get the spoofed reply traffic back
> > out of AWS it gets dropped always with either DR mode or TUN unless I
> > send it back via the director and that case I may as well do NAT mode
> > instead...
> >
> > My understanding is that when egress traffic hits their edge they
> > check the source matches the instance it came from and if it doesn't
> > the packet gets dropped. However, all works fine if the client is in
> > the same VPC so it doesn't need to go out past the edge of the VPC.
> >
> > Just interested if you found the same in your testing?
> > Aaron West
> >
> > Loadbalancer.org
> >
> > www.loadbalancer.org
> > +1 888 867 9504 / +44 (0)330 380 1064
> > aaron@loadbalancer.org
> >
> > LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG
> >
> >
> > On 10 August 2017 at 18:08, Timothy R. Weiand <timothy.weiand@gmail.com>
> > wrote:
> > > Julian:
> > >
> > > Good news and bad news: this guidance helped me get LVS-TUN working with
> > a
> > > 3.10.0 kernel. But, I am not able to use the same configuration to get
> > > 4.4.x,4.8.x or 4.12.x working. We are working on doing further analysis
> > to
> > > ensure it is kernel dependent and not distribution dependent. I hope to
> > > find the kernel version that this configuration stops working.
> > >
> > > Note on 4.x machines: If we do not have the correct ip assigned to the
> > > tunnel interface we can see traffic on that interface. But, if we use the
> > > correct VIP then we no longer see the traffic.
> > >
> > > I will keep you updated as I find information out.
> > >
> > > Thanks!
> > > -Timothy
> > >
> > > On Tue, Jul 25, 2017 at 4:25 PM Julian Anastasov <ja@ssi.bg> wrote:
> > >
> > >>
> > >> Hello,
> > >>
> > >> On Fri, 14 Jul 2017, Timothy R. Weiand wrote:
> > >>
> > >> > I am building a DR-TUN configuration to load balance DNS traffic. My
> > >> issue
> > >> > is with the real servers: the ipip packet get unwrapped and appears on
> > >> the
> > >> > tunnel interface and becomes a martian. Also, I have not been able to
> > >> > determine how to bind to the tunnel interface to capture these
> > packets. I
> > >> > have verified details with tcpdump/dmesg. All my configuration is
> > >> scripted
> > >> > to ensure reproducibility. tcpdumps below do not show real DNS
> > traffic;
> > >> > using netcat to send text.
> > >> >
> > >> > Much more detail can be supplied, please request it.
> > >> >
> > >> > Configuration:
> > >> >
> > >> > ==============
> > >> >
> > >> > - All machines are currently Debian 9 (4.9.30-2+deb9u2)
> > >> >
> > >> > - Client: 192.168.200.10
> > >> >
> > >> > - Director: 192.168.200.11
> > >> >
> > >> > - Real Server: 192.168.200.12
> > >> >
> > >> > - No VIP: I am using the IP address for my director interface -
> > >> > 192.168.200.11
> > >> >
> > >> >
> > >> > Director Configuration:
> > >> >
> > >> > ==================
> > >> >
> > >> > vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
> > >> >
> > >> > IP Virtual Server version 1.2.1 (size=4096)
> > >> >
> > >> > Prot LocalAddress:Port Scheduler Flags
> > >> >
> > >> > -> RemoteAddress:Port Forward Weight ActiveConn InActConn
> > >> >
> > >> > UDP 192.168.200.11:53 rr
> > >> >
> > >> > -> 192.168.200.12:53 Tunnel 1 0 0
> > >> >
> > >> >
> > >> >
> > >> > Real Server:
> > >> >
> > >> > ============
> > >> >
> > >> > # modprobe ipip
> > >> >
> > >> > # echo 1 > /proc/sys/net/ipv4/ip_forward
> > >> >
> > >> > # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
> > >> > 192.168.200.11
> > >> >
> > >> > # ip link set tunl1 up arp off
> > >> >
> > >> > # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
> > >> >
> > >> > # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
> > >> >
> > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
> > >> >
> > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
> > >>
> > >> tunl1/rp_filter=2 is ignored if there are no
> > >> IPv4 addresses configured on tunl1, you can see
> > >> __fib_validate_source() for reference:
> > >>
> > >> if (no_addr)
> > >> goto last_resort;
> > >>
> > >> As result, packets are dropped as martians due to rpf=2.
> > >>
> > >> To check if packets are dropped by routing try:
> > >>
> > >> ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1
> > >>
> > >> Here is small howto for TUN (outdated):
> > >>
> > >> http://ja.ssi.bg/TUN-HOWTO.txt
> > >>
> > >> > vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
> > >> >
> > >> > tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size
> > 262144
> > >> > bytes
> > >> >
> > >> > 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags [DF],
> > >> proto
> > >> > UDP (17), length 40)
> > >> >
> > >> > 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226
> > >> updateMA+
> > >> > [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
> > >> >
> > >> >
> > >> >
> > >> > If I add an IP address to the interface the packet never reaches tunl1
> > >> >
> > >> > # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
> > >>
> > >> Try with (note mask /32):
> > >>
> > >> ip addr add 192.168.200.11/32 dev tunl1
> > >>
> > >> as shown in LVS HOWTO (8.3):
> > >>
> > >> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
> > >>
> > >> > Notes:
> > >> >
> > >> > ======
> > >> >
> > >> >
> > >> >
> > >> > LVS-DR was not an option because the real servers will be in different
> > >> > subnets (I was able to get LVS-DR working though).
> > >> >
> > >> >
> > >> >
> > >> > I am looking for advice on how to better understand or resolve this
> > >> issue.
> > >> > Or, who would be better to answer this question.
> > >> >
> > >> >
> > >> >
> > >> > I have groked as many articles on how to setup this configuration as I
> > >> can.
> > >> > And, I have read a good deal of the archives of this mailing list.
> > >> >
> > >> >
> > >> >
> > >> > Linux distributions I have tried are Debian, Ubuntu and Amazon linux.
> > >> This
> > >> > is been tested on both AWS and VMware fusion.
> > >> >
> > >> >
> > >> >
> > >> > ------
> > >> >
> > >> >
> > >> >
> > >> > I have a feeling I am missing something simple...
> > >> >
> > >> >
> > >> >
> > >> > Thanks!
> > >> >
> > >> > -Timothy
> > >>
> > >> Regards
> > >>
> > >> --
> > >> Julian Anastasov <ja@ssi.bg>
> > >>
> > > _______________________________________________
> > > Please read the documentation before posting - it's available at:
> > > http://www.linuxvirtualserver.org/
> > >
> > > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > > Send requests to lvs-users-request@LinuxVirtualServer.org
> > > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> >
> > _______________________________________________
> > Please read the documentation before posting - it's available at:
> > http://www.linuxvirtualserver.org/
> >
> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > Send requests to lvs-users-request@LinuxVirtualServer.org
> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> >
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Ooohhhh!! Let me try this.

Thanks for the directions,
-Timothy
On Mon, Nov 6, 2017 at 1:55 PM Andrew Smalley <asmalley@loadbalancer.org>
wrote:

> Hi Timothy,
>
> I've been testing ipip (Tun mode) and I have it working,
>
> I note that the Weak ES Model needs to be accounted for and the tunl0
> should be a /32 not a /24 as you have it on the real server.
>
> ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
>
> becomes
>
> ip addr add dev tunl1 192.168.200.11/32 brd 192.168.200.11 dev tunl1
>
>
> I also have this in my /etc/sysctl.conf
>
> net.ipv4.conf.all.arp_ignore=1
> net.ipv4.conf.eth0.arp_ignore=1
> net.ipv4.conf.tunl1.arp_ignore=1
> net.ipv4.conf.all.arp_announce=2
> net.ipv4.conf.eth0.arp_announce=2
> net.ipv4.conf.tunl1.arp_announce=2
>
> net.ipv4.conf.all.log_martians=1
> net.ipv4.conf.default.log_martians=1
>
> And I ran this script portion to enable forwarding and disable the
> rp_filter
>
> if [ $1 = "tunl1" ]; then
> echo 1 > /proc/sys/net/ipv4/conf/tunl0/forwarding
> echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
> fi
>
>
> This is with a 4.4.49 Kernel
>
> I hope this help you
>
>
> Andruw Smalley
>
> Loadbalancer.org Ltd.
>
> www.loadbalancer.org
> +1 888 867 9504 / +44 (0)330 380 1064
> asmalley@loadbalancer.org
>
> Leave a Review | Deployment Guides | Blog
>
>
> On 22 August 2017 at 12:43, Timothy R. Weiand <timothy.weiand@gmail.com>
> wrote:
> >
> > Aaron:
> >
> > I was able to get LVS-TUN working in between subnets in AWS if the real
> > server is a 3.x kernel (RHEL to be specific). I had to set the
> destination
> > side of the tunnel and the address on that interface to the VIP. Then, I
> > configured unbound to listen for the VIP address.
> >
> > We are working with RedHat to get the 4.x kernels fixed. But, no
> timeframe
> > on that fix.
> >
> > FYI - We are going to work towards open sourcing our work so that
> everyone
> > can see exactly what we have done.
> >
> > -Timothy
> >
> > On Thu, Aug 17, 2017 at 10:56 AM Aaron West <aaron@loadbalancer.org>
> wrote:
> >
> > > Hi Timothy,
> > >
> > > What you had working was it in AWS or VMWare?
> > >
> > > I've pretty much given up trying to get the spoofed reply traffic back
> > > out of AWS it gets dropped always with either DR mode or TUN unless I
> > > send it back via the director and that case I may as well do NAT mode
> > > instead...
> > >
> > > My understanding is that when egress traffic hits their edge they
> > > check the source matches the instance it came from and if it doesn't
> > > the packet gets dropped. However, all works fine if the client is in
> > > the same VPC so it doesn't need to go out past the edge of the VPC.
> > >
> > > Just interested if you found the same in your testing?
> > > Aaron West
> > >
> > > Loadbalancer.org
> > >
> > > www.loadbalancer.org
> > > +1 888 867 9504 / +44 (0)330 380 1064
> > > aaron@loadbalancer.org
> > >
> > > LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG
> > >
> > >
> > > On 10 August 2017 at 18:08, Timothy R. Weiand <
> timothy.weiand@gmail.com>
> > > wrote:
> > > > Julian:
> > > >
> > > > Good news and bad news: this guidance helped me get LVS-TUN working
> with
> > > a
> > > > 3.10.0 kernel. But, I am not able to use the same configuration to
> get
> > > > 4.4.x,4.8.x or 4.12.x working. We are working on doing further
> analysis
> > > to
> > > > ensure it is kernel dependent and not distribution dependent. I hope
> to
> > > > find the kernel version that this configuration stops working.
> > > >
> > > > Note on 4.x machines: If we do not have the correct ip assigned to
> the
> > > > tunnel interface we can see traffic on that interface. But, if we
> use the
> > > > correct VIP then we no longer see the traffic.
> > > >
> > > > I will keep you updated as I find information out.
> > > >
> > > > Thanks!
> > > > -Timothy
> > > >
> > > > On Tue, Jul 25, 2017 at 4:25 PM Julian Anastasov <ja@ssi.bg> wrote:
> > > >
> > > >>
> > > >> Hello,
> > > >>
> > > >> On Fri, 14 Jul 2017, Timothy R. Weiand wrote:
> > > >>
> > > >> > I am building a DR-TUN configuration to load balance DNS traffic.
> My
> > > >> issue
> > > >> > is with the real servers: the ipip packet get unwrapped and
> appears on
> > > >> the
> > > >> > tunnel interface and becomes a martian. Also, I have not been
> able to
> > > >> > determine how to bind to the tunnel interface to capture these
> > > packets. I
> > > >> > have verified details with tcpdump/dmesg. All my configuration is
> > > >> scripted
> > > >> > to ensure reproducibility. tcpdumps below do not show real DNS
> > > traffic;
> > > >> > using netcat to send text.
> > > >> >
> > > >> > Much more detail can be supplied, please request it.
> > > >> >
> > > >> > Configuration:
> > > >> >
> > > >> > ==============
> > > >> >
> > > >> > - All machines are currently Debian 9 (4.9.30-2+deb9u2)
> > > >> >
> > > >> > - Client: 192.168.200.10
> > > >> >
> > > >> > - Director: 192.168.200.11
> > > >> >
> > > >> > - Real Server: 192.168.200.12
> > > >> >
> > > >> > - No VIP: I am using the IP address for my director interface -
> > > >> > 192.168.200.11
> > > >> >
> > > >> >
> > > >> > Director Configuration:
> > > >> >
> > > >> > ==================
> > > >> >
> > > >> > vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
> > > >> >
> > > >> > IP Virtual Server version 1.2.1 (size=4096)
> > > >> >
> > > >> > Prot LocalAddress:Port Scheduler Flags
> > > >> >
> > > >> > -> RemoteAddress:Port Forward Weight ActiveConn
> InActConn
> > > >> >
> > > >> > UDP 192.168.200.11:53 rr
> > > >> >
> > > >> > -> 192.168.200.12:53 Tunnel 1 0 0
> > > >> >
> > > >> >
> > > >> >
> > > >> > Real Server:
> > > >> >
> > > >> > ============
> > > >> >
> > > >> > # modprobe ipip
> > > >> >
> > > >> > # echo 1 > /proc/sys/net/ipv4/ip_forward
> > > >> >
> > > >> > # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
> > > >> > 192.168.200.11
> > > >> >
> > > >> > # ip link set tunl1 up arp off
> > > >> >
> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
> > > >> >
> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
> > > >> >
> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
> > > >> >
> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
> > > >>
> > > >> tunl1/rp_filter=2 is ignored if there are no
> > > >> IPv4 addresses configured on tunl1, you can see
> > > >> __fib_validate_source() for reference:
> > > >>
> > > >> if (no_addr)
> > > >> goto last_resort;
> > > >>
> > > >> As result, packets are dropped as martians due to rpf=2.
> > > >>
> > > >> To check if packets are dropped by routing try:
> > > >>
> > > >> ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1
> > > >>
> > > >> Here is small howto for TUN (outdated):
> > > >>
> > > >> http://ja.ssi.bg/TUN-HOWTO.txt
> > > >>
> > > >> > vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
> > > >> >
> > > >> > tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size
> > > 262144
> > > >> > bytes
> > > >> >
> > > >> > 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags
> [DF],
> > > >> proto
> > > >> > UDP (17), length 40)
> > > >> >
> > > >> > 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226
> > > >> updateMA+
> > > >> > [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
> > > >> >
> > > >> >
> > > >> >
> > > >> > If I add an IP address to the interface the packet never reaches
> tunl1
> > > >> >
> > > >> > # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
> > > >>
> > > >> Try with (note mask /32):
> > > >>
> > > >> ip addr add 192.168.200.11/32 dev tunl1
> > > >>
> > > >> as shown in LVS HOWTO (8.3):
> > > >>
> > > >> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
> > > >>
> > > >> > Notes:
> > > >> >
> > > >> > ======
> > > >> >
> > > >> >
> > > >> >
> > > >> > LVS-DR was not an option because the real servers will be in
> different
> > > >> > subnets (I was able to get LVS-DR working though).
> > > >> >
> > > >> >
> > > >> >
> > > >> > I am looking for advice on how to better understand or resolve
> this
> > > >> issue.
> > > >> > Or, who would be better to answer this question.
> > > >> >
> > > >> >
> > > >> >
> > > >> > I have groked as many articles on how to setup this configuration
> as I
> > > >> can.
> > > >> > And, I have read a good deal of the archives of this mailing list.
> > > >> >
> > > >> >
> > > >> >
> > > >> > Linux distributions I have tried are Debian, Ubuntu and Amazon
> linux.
> > > >> This
> > > >> > is been tested on both AWS and VMware fusion.
> > > >> >
> > > >> >
> > > >> >
> > > >> > ------
> > > >> >
> > > >> >
> > > >> >
> > > >> > I have a feeling I am missing something simple...
> > > >> >
> > > >> >
> > > >> >
> > > >> > Thanks!
> > > >> >
> > > >> > -Timothy
> > > >>
> > > >> Regards
> > > >>
> > > >> --
> > > >> Julian Anastasov <ja@ssi.bg>
> > > >>
> > > > _______________________________________________
> > > > Please read the documentation before posting - it's available at:
> > > > http://www.linuxvirtualserver.org/
> > > >
> > > > LinuxVirtualServer.org mailing list -
> lvs-users@LinuxVirtualServer.org
> > > > Send requests to lvs-users-request@LinuxVirtualServer.org
> > > > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> > >
> > > _______________________________________________
> > > Please read the documentation before posting - it's available at:
> > > http://www.linuxvirtualserver.org/
> > >
> > > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > > Send requests to lvs-users-request@LinuxVirtualServer.org
> > > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> > >
> > _______________________________________________
> > Please read the documentation before posting - it's available at:
> > http://www.linuxvirtualserver.org/
> >
> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > Send requests to lvs-users-request@LinuxVirtualServer.org
> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
Hello Timothy

Aaron has told me that you are in AWS and this being the case I hope
ipip does indeed work with my hints.

It is worth noting that I know in AWS we can not do Layer4 DR however
with the help of OpenVSwitch I know 100% that its possible to do
Layer4 DR to any real server connected with either a VXLAN or GRE
tunnel .

I have personally done this in my private service and stretched a
VXLAN from AWS to my private lab and other remote datacenters where my
real servers reside.




Andruw Smalley

Loadbalancer.org Ltd.

www.loadbalancer.org
+1 888 867 9504 / +44 (0)330 380 1064
asmalley@loadbalancer.org

Leave a Review | Deployment Guides | Blog


On 7 November 2017 at 16:51, Timothy R. Weiand <timothy.weiand@gmail.com> wrote:
> Ooohhhh!! Let me try this.
>
> Thanks for the directions,
> -Timothy
> On Mon, Nov 6, 2017 at 1:55 PM Andrew Smalley <asmalley@loadbalancer.org>
> wrote:
>
>> Hi Timothy,
>>
>> I've been testing ipip (Tun mode) and I have it working,
>>
>> I note that the Weak ES Model needs to be accounted for and the tunl0
>> should be a /32 not a /24 as you have it on the real server.
>>
>> ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
>>
>> becomes
>>
>> ip addr add dev tunl1 192.168.200.11/32 brd 192.168.200.11 dev tunl1
>>
>>
>> I also have this in my /etc/sysctl.conf
>>
>> net.ipv4.conf.all.arp_ignore=1
>> net.ipv4.conf.eth0.arp_ignore=1
>> net.ipv4.conf.tunl1.arp_ignore=1
>> net.ipv4.conf.all.arp_announce=2
>> net.ipv4.conf.eth0.arp_announce=2
>> net.ipv4.conf.tunl1.arp_announce=2
>>
>> net.ipv4.conf.all.log_martians=1
>> net.ipv4.conf.default.log_martians=1
>>
>> And I ran this script portion to enable forwarding and disable the
>> rp_filter
>>
>> if [ $1 = "tunl1" ]; then
>> echo 1 > /proc/sys/net/ipv4/conf/tunl0/forwarding
>> echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
>> fi
>>
>>
>> This is with a 4.4.49 Kernel
>>
>> I hope this help you
>>
>>
>> Andruw Smalley
>>
>> Loadbalancer.org Ltd.
>>
>> www.loadbalancer.org
>> +1 888 867 9504 / +44 (0)330 380 1064
>> asmalley@loadbalancer.org
>>
>> Leave a Review | Deployment Guides | Blog
>>
>>
>> On 22 August 2017 at 12:43, Timothy R. Weiand <timothy.weiand@gmail.com>
>> wrote:
>> >
>> > Aaron:
>> >
>> > I was able to get LVS-TUN working in between subnets in AWS if the real
>> > server is a 3.x kernel (RHEL to be specific). I had to set the
>> destination
>> > side of the tunnel and the address on that interface to the VIP. Then, I
>> > configured unbound to listen for the VIP address.
>> >
>> > We are working with RedHat to get the 4.x kernels fixed. But, no
>> timeframe
>> > on that fix.
>> >
>> > FYI - We are going to work towards open sourcing our work so that
>> everyone
>> > can see exactly what we have done.
>> >
>> > -Timothy
>> >
>> > On Thu, Aug 17, 2017 at 10:56 AM Aaron West <aaron@loadbalancer.org>
>> wrote:
>> >
>> > > Hi Timothy,
>> > >
>> > > What you had working was it in AWS or VMWare?
>> > >
>> > > I've pretty much given up trying to get the spoofed reply traffic back
>> > > out of AWS it gets dropped always with either DR mode or TUN unless I
>> > > send it back via the director and that case I may as well do NAT mode
>> > > instead...
>> > >
>> > > My understanding is that when egress traffic hits their edge they
>> > > check the source matches the instance it came from and if it doesn't
>> > > the packet gets dropped. However, all works fine if the client is in
>> > > the same VPC so it doesn't need to go out past the edge of the VPC.
>> > >
>> > > Just interested if you found the same in your testing?
>> > > Aaron West
>> > >
>> > > Loadbalancer.org
>> > >
>> > > www.loadbalancer.org
>> > > +1 888 867 9504 / +44 (0)330 380 1064
>> > > aaron@loadbalancer.org
>> > >
>> > > LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG
>> > >
>> > >
>> > > On 10 August 2017 at 18:08, Timothy R. Weiand <
>> timothy.weiand@gmail.com>
>> > > wrote:
>> > > > Julian:
>> > > >
>> > > > Good news and bad news: this guidance helped me get LVS-TUN working
>> with
>> > > a
>> > > > 3.10.0 kernel. But, I am not able to use the same configuration to
>> get
>> > > > 4.4.x,4.8.x or 4.12.x working. We are working on doing further
>> analysis
>> > > to
>> > > > ensure it is kernel dependent and not distribution dependent. I hope
>> to
>> > > > find the kernel version that this configuration stops working.
>> > > >
>> > > > Note on 4.x machines: If we do not have the correct ip assigned to
>> the
>> > > > tunnel interface we can see traffic on that interface. But, if we
>> use the
>> > > > correct VIP then we no longer see the traffic.
>> > > >
>> > > > I will keep you updated as I find information out.
>> > > >
>> > > > Thanks!
>> > > > -Timothy
>> > > >
>> > > > On Tue, Jul 25, 2017 at 4:25 PM Julian Anastasov <ja@ssi.bg> wrote:
>> > > >
>> > > >>
>> > > >> Hello,
>> > > >>
>> > > >> On Fri, 14 Jul 2017, Timothy R. Weiand wrote:
>> > > >>
>> > > >> > I am building a DR-TUN configuration to load balance DNS traffic.
>> My
>> > > >> issue
>> > > >> > is with the real servers: the ipip packet get unwrapped and
>> appears on
>> > > >> the
>> > > >> > tunnel interface and becomes a martian. Also, I have not been
>> able to
>> > > >> > determine how to bind to the tunnel interface to capture these
>> > > packets. I
>> > > >> > have verified details with tcpdump/dmesg. All my configuration is
>> > > >> scripted
>> > > >> > to ensure reproducibility. tcpdumps below do not show real DNS
>> > > traffic;
>> > > >> > using netcat to send text.
>> > > >> >
>> > > >> > Much more detail can be supplied, please request it.
>> > > >> >
>> > > >> > Configuration:
>> > > >> >
>> > > >> > ==============
>> > > >> >
>> > > >> > - All machines are currently Debian 9 (4.9.30-2+deb9u2)
>> > > >> >
>> > > >> > - Client: 192.168.200.10
>> > > >> >
>> > > >> > - Director: 192.168.200.11
>> > > >> >
>> > > >> > - Real Server: 192.168.200.12
>> > > >> >
>> > > >> > - No VIP: I am using the IP address for my director interface -
>> > > >> > 192.168.200.11
>> > > >> >
>> > > >> >
>> > > >> > Director Configuration:
>> > > >> >
>> > > >> > ==================
>> > > >> >
>> > > >> > vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
>> > > >> >
>> > > >> > IP Virtual Server version 1.2.1 (size=4096)
>> > > >> >
>> > > >> > Prot LocalAddress:Port Scheduler Flags
>> > > >> >
>> > > >> > -> RemoteAddress:Port Forward Weight ActiveConn
>> InActConn
>> > > >> >
>> > > >> > UDP 192.168.200.11:53 rr
>> > > >> >
>> > > >> > -> 192.168.200.12:53 Tunnel 1 0 0
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > Real Server:
>> > > >> >
>> > > >> > ============
>> > > >> >
>> > > >> > # modprobe ipip
>> > > >> >
>> > > >> > # echo 1 > /proc/sys/net/ipv4/ip_forward
>> > > >> >
>> > > >> > # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12 remote
>> > > >> > 192.168.200.11
>> > > >> >
>> > > >> > # ip link set tunl1 up arp off
>> > > >> >
>> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
>> > > >> >
>> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
>> > > >> >
>> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
>> > > >> >
>> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
>> > > >>
>> > > >> tunl1/rp_filter=2 is ignored if there are no
>> > > >> IPv4 addresses configured on tunl1, you can see
>> > > >> __fib_validate_source() for reference:
>> > > >>
>> > > >> if (no_addr)
>> > > >> goto last_resort;
>> > > >>
>> > > >> As result, packets are dropped as martians due to rpf=2.
>> > > >>
>> > > >> To check if packets are dropped by routing try:
>> > > >>
>> > > >> ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1
>> > > >>
>> > > >> Here is small howto for TUN (outdated):
>> > > >>
>> > > >> http://ja.ssi.bg/TUN-HOWTO.txt
>> > > >>
>> > > >> > vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
>> > > >> >
>> > > >> > tcpdump: listening on tunl1, link-type RAW (Raw IP), capture size
>> > > 262144
>> > > >> > bytes
>> > > >> >
>> > > >> > 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags
>> [DF],
>> > > >> proto
>> > > >> > UDP (17), length 40)
>> > > >> >
>> > > >> > 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok] 26226
>> > > >> updateMA+
>> > > >> > [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > If I add an IP address to the interface the packet never reaches
>> tunl1
>> > > >> >
>> > > >> > # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
>> > > >>
>> > > >> Try with (note mask /32):
>> > > >>
>> > > >> ip addr add 192.168.200.11/32 dev tunl1
>> > > >>
>> > > >> as shown in LVS HOWTO (8.3):
>> > > >>
>> > > >> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
>> > > >>
>> > > >> > Notes:
>> > > >> >
>> > > >> > ======
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > LVS-DR was not an option because the real servers will be in
>> different
>> > > >> > subnets (I was able to get LVS-DR working though).
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > I am looking for advice on how to better understand or resolve
>> this
>> > > >> issue.
>> > > >> > Or, who would be better to answer this question.
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > I have groked as many articles on how to setup this configuration
>> as I
>> > > >> can.
>> > > >> > And, I have read a good deal of the archives of this mailing list.
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > Linux distributions I have tried are Debian, Ubuntu and Amazon
>> linux.
>> > > >> This
>> > > >> > is been tested on both AWS and VMware fusion.
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > ------
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > I have a feeling I am missing something simple...
>> > > >> >
>> > > >> >
>> > > >> >
>> > > >> > Thanks!
>> > > >> >
>> > > >> > -Timothy
>> > > >>
>> > > >> Regards
>> > > >>
>> > > >> --
>> > > >> Julian Anastasov <ja@ssi.bg>
>> > > >>
>> > > > _______________________________________________
>> > > > Please read the documentation before posting - it's available at:
>> > > > http://www.linuxvirtualserver.org/
>> > > >
>> > > > LinuxVirtualServer.org mailing list -
>> lvs-users@LinuxVirtualServer.org
>> > > > Send requests to lvs-users-request@LinuxVirtualServer.org
>> > > > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>> > >
>> > > _______________________________________________
>> > > Please read the documentation before posting - it's available at:
>> > > http://www.linuxvirtualserver.org/
>> > >
>> > > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> > > Send requests to lvs-users-request@LinuxVirtualServer.org
>> > > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>> > >
>> > _______________________________________________
>> > Please read the documentation before posting - it's available at:
>> > http://www.linuxvirtualserver.org/
>> >
>> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> > Send requests to lvs-users-request@LinuxVirtualServer.org
>> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] DR-TUN Issues: Packets become martians on real server [ In reply to ]
So, it turns out that I am currently interviewing to find a new employer. I
will probably not pick this back up again; I will not be able to see if
this works in the “broken” environment.

Thanks for the help!
-Timothy
On Tue, Nov 7, 2017 at 12:38 PM Andrew Smalley <asmalley@loadbalancer.org>
wrote:

> Hello Timothy
>
> Aaron has told me that you are in AWS and this being the case I hope
> ipip does indeed work with my hints.
>
> It is worth noting that I know in AWS we can not do Layer4 DR however
> with the help of OpenVSwitch I know 100% that its possible to do
> Layer4 DR to any real server connected with either a VXLAN or GRE
> tunnel .
>
> I have personally done this in my private service and stretched a
> VXLAN from AWS to my private lab and other remote datacenters where my
> real servers reside.
>
>
>
>
> Andruw Smalley
>
> Loadbalancer.org Ltd.
>
> www.loadbalancer.org
> +1 888 867 9504 / +44 (0)330 380 1064
> asmalley@loadbalancer.org
>
> Leave a Review | Deployment Guides | Blog
>
>
> On 7 November 2017 at 16:51, Timothy R. Weiand <timothy.weiand@gmail.com>
> wrote:
> > Ooohhhh!! Let me try this.
> >
> > Thanks for the directions,
> > -Timothy
> > On Mon, Nov 6, 2017 at 1:55 PM Andrew Smalley <asmalley@loadbalancer.org
> >
> > wrote:
> >
> >> Hi Timothy,
> >>
> >> I've been testing ipip (Tun mode) and I have it working,
> >>
> >> I note that the Weak ES Model needs to be accounted for and the tunl0
> >> should be a /32 not a /24 as you have it on the real server.
> >>
> >> ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
> >>
> >> becomes
> >>
> >> ip addr add dev tunl1 192.168.200.11/32 brd 192.168.200.11 dev tunl1
> >>
> >>
> >> I also have this in my /etc/sysctl.conf
> >>
> >> net.ipv4.conf.all.arp_ignore=1
> >> net.ipv4.conf.eth0.arp_ignore=1
> >> net.ipv4.conf.tunl1.arp_ignore=1
> >> net.ipv4.conf.all.arp_announce=2
> >> net.ipv4.conf.eth0.arp_announce=2
> >> net.ipv4.conf.tunl1.arp_announce=2
> >>
> >> net.ipv4.conf.all.log_martians=1
> >> net.ipv4.conf.default.log_martians=1
> >>
> >> And I ran this script portion to enable forwarding and disable the
> >> rp_filter
> >>
> >> if [ $1 = "tunl1" ]; then
> >> echo 1 > /proc/sys/net/ipv4/conf/tunl0/forwarding
> >> echo 0 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
> >> fi
> >>
> >>
> >> This is with a 4.4.49 Kernel
> >>
> >> I hope this help you
> >>
> >>
> >> Andruw Smalley
> >>
> >> Loadbalancer.org Ltd.
> >>
> >> www.loadbalancer.org
> >> +1 888 867 9504 / +44 (0)330 380 1064
> >> asmalley@loadbalancer.org
> >>
> >> Leave a Review | Deployment Guides | Blog
> >>
> >>
> >> On 22 August 2017 at 12:43, Timothy R. Weiand <timothy.weiand@gmail.com
> >
> >> wrote:
> >> >
> >> > Aaron:
> >> >
> >> > I was able to get LVS-TUN working in between subnets in AWS if the
> real
> >> > server is a 3.x kernel (RHEL to be specific). I had to set the
> >> destination
> >> > side of the tunnel and the address on that interface to the VIP.
> Then, I
> >> > configured unbound to listen for the VIP address.
> >> >
> >> > We are working with RedHat to get the 4.x kernels fixed. But, no
> >> timeframe
> >> > on that fix.
> >> >
> >> > FYI - We are going to work towards open sourcing our work so that
> >> everyone
> >> > can see exactly what we have done.
> >> >
> >> > -Timothy
> >> >
> >> > On Thu, Aug 17, 2017 at 10:56 AM Aaron West <aaron@loadbalancer.org>
> >> wrote:
> >> >
> >> > > Hi Timothy,
> >> > >
> >> > > What you had working was it in AWS or VMWare?
> >> > >
> >> > > I've pretty much given up trying to get the spoofed reply traffic
> back
> >> > > out of AWS it gets dropped always with either DR mode or TUN unless
> I
> >> > > send it back via the director and that case I may as well do NAT
> mode
> >> > > instead...
> >> > >
> >> > > My understanding is that when egress traffic hits their edge they
> >> > > check the source matches the instance it came from and if it doesn't
> >> > > the packet gets dropped. However, all works fine if the client is in
> >> > > the same VPC so it doesn't need to go out past the edge of the VPC.
> >> > >
> >> > > Just interested if you found the same in your testing?
> >> > > Aaron West
> >> > >
> >> > > Loadbalancer.org
> >> > >
> >> > > www.loadbalancer.org
> >> > > +1 888 867 9504 / +44 (0)330 380 1064
> >> > > aaron@loadbalancer.org
> >> > >
> >> > > LEAVE A REVIEW | DEPLOYMENT GUIDES | BLOG
> >> > >
> >> > >
> >> > > On 10 August 2017 at 18:08, Timothy R. Weiand <
> >> timothy.weiand@gmail.com>
> >> > > wrote:
> >> > > > Julian:
> >> > > >
> >> > > > Good news and bad news: this guidance helped me get LVS-TUN
> working
> >> with
> >> > > a
> >> > > > 3.10.0 kernel. But, I am not able to use the same configuration to
> >> get
> >> > > > 4.4.x,4.8.x or 4.12.x working. We are working on doing further
> >> analysis
> >> > > to
> >> > > > ensure it is kernel dependent and not distribution dependent. I
> hope
> >> to
> >> > > > find the kernel version that this configuration stops working.
> >> > > >
> >> > > > Note on 4.x machines: If we do not have the correct ip assigned to
> >> the
> >> > > > tunnel interface we can see traffic on that interface. But, if we
> >> use the
> >> > > > correct VIP then we no longer see the traffic.
> >> > > >
> >> > > > I will keep you updated as I find information out.
> >> > > >
> >> > > > Thanks!
> >> > > > -Timothy
> >> > > >
> >> > > > On Tue, Jul 25, 2017 at 4:25 PM Julian Anastasov <ja@ssi.bg>
> wrote:
> >> > > >
> >> > > >>
> >> > > >> Hello,
> >> > > >>
> >> > > >> On Fri, 14 Jul 2017, Timothy R. Weiand wrote:
> >> > > >>
> >> > > >> > I am building a DR-TUN configuration to load balance DNS
> traffic.
> >> My
> >> > > >> issue
> >> > > >> > is with the real servers: the ipip packet get unwrapped and
> >> appears on
> >> > > >> the
> >> > > >> > tunnel interface and becomes a martian. Also, I have not been
> >> able to
> >> > > >> > determine how to bind to the tunnel interface to capture these
> >> > > packets. I
> >> > > >> > have verified details with tcpdump/dmesg. All my configuration
> is
> >> > > >> scripted
> >> > > >> > to ensure reproducibility. tcpdumps below do not show real DNS
> >> > > traffic;
> >> > > >> > using netcat to send text.
> >> > > >> >
> >> > > >> > Much more detail can be supplied, please request it.
> >> > > >> >
> >> > > >> > Configuration:
> >> > > >> >
> >> > > >> > ==============
> >> > > >> >
> >> > > >> > - All machines are currently Debian 9 (4.9.30-2+deb9u2)
> >> > > >> >
> >> > > >> > - Client: 192.168.200.10
> >> > > >> >
> >> > > >> > - Director: 192.168.200.11
> >> > > >> >
> >> > > >> > - Real Server: 192.168.200.12
> >> > > >> >
> >> > > >> > - No VIP: I am using the IP address for my director interface -
> >> > > >> > 192.168.200.11
> >> > > >> >
> >> > > >> >
> >> > > >> > Director Configuration:
> >> > > >> >
> >> > > >> > ==================
> >> > > >> >
> >> > > >> > vagrant@debian-9-lb:~$ sudo ipvsadm -Ln
> >> > > >> >
> >> > > >> > IP Virtual Server version 1.2.1 (size=4096)
> >> > > >> >
> >> > > >> > Prot LocalAddress:Port Scheduler Flags
> >> > > >> >
> >> > > >> > -> RemoteAddress:Port Forward Weight ActiveConn
> >> InActConn
> >> > > >> >
> >> > > >> > UDP 192.168.200.11:53 rr
> >> > > >> >
> >> > > >> > -> 192.168.200.12:53 Tunnel 1 0 0
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > Real Server:
> >> > > >> >
> >> > > >> > ============
> >> > > >> >
> >> > > >> > # modprobe ipip
> >> > > >> >
> >> > > >> > # echo 1 > /proc/sys/net/ipv4/ip_forward
> >> > > >> >
> >> > > >> > # ip tunnel add tunl1 mode ipip ttl 32 local 192.168.200.12
> remote
> >> > > >> > 192.168.200.11
> >> > > >> >
> >> > > >> > # ip link set tunl1 up arp off
> >> > > >> >
> >> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/all/rp_filter
> >> > > >> >
> >> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/default/rp_filter
> >> > > >> >
> >> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl0/rp_filter
> >> > > >> >
> >> > > >> > # echo 2 > /proc/sys/net/ipv4/conf/tunl1/rp_filter
> >> > > >>
> >> > > >> tunl1/rp_filter=2 is ignored if there are no
> >> > > >> IPv4 addresses configured on tunl1, you can see
> >> > > >> __fib_validate_source() for reference:
> >> > > >>
> >> > > >> if (no_addr)
> >> > > >> goto last_resort;
> >> > > >>
> >> > > >> As result, packets are dropped as martians due to rpf=2.
> >> > > >>
> >> > > >> To check if packets are dropped by routing try:
> >> > > >>
> >> > > >> ip route get from 192.168.200.10 to 192.168.200.11 iif tunl1
> >> > > >>
> >> > > >> Here is small howto for TUN (outdated):
> >> > > >>
> >> > > >> http://ja.ssi.bg/TUN-HOWTO.txt
> >> > > >>
> >> > > >> > vagrant@debian-9-dns:~$ sudo tcpdump -e -n -s 0 -i tunl1 -vv
> >> > > >> >
> >> > > >> > tcpdump: listening on tunl1, link-type RAW (Raw IP), capture
> size
> >> > > 262144
> >> > > >> > bytes
> >> > > >> >
> >> > > >> > 10:16:13.919646 ip: (tos 0x0, ttl 64, id 63545, offset 0, flags
> >> [DF],
> >> > > >> proto
> >> > > >> > UDP (17), length 40)
> >> > > >> >
> >> > > >> > 192.168.200.10.51149 > 192.168.200.11.53: [udp sum ok]
> 26226
> >> > > >> updateMA+
> >> > > >> > [b2&3=0x6f6d] [27753a] [11619q] [25966n] [29706au][|domain]
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > If I add an IP address to the interface the packet never
> reaches
> >> tunl1
> >> > > >> >
> >> > > >> > # ip addr add 192.168.200.11/24 brd 192.168.200.11 dev tunl1
> >> > > >>
> >> > > >> Try with (note mask /32):
> >> > > >>
> >> > > >> ip addr add 192.168.200.11/32 dev tunl1
> >> > > >>
> >> > > >> as shown in LVS HOWTO (8.3):
> >> > > >>
> >> > > >>
> http://www.austintek.com/LVS/LVS-HOWTO/HOWTO/LVS-HOWTO.LVS-Tun.html
> >> > > >>
> >> > > >> > Notes:
> >> > > >> >
> >> > > >> > ======
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > LVS-DR was not an option because the real servers will be in
> >> different
> >> > > >> > subnets (I was able to get LVS-DR working though).
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > I am looking for advice on how to better understand or resolve
> >> this
> >> > > >> issue.
> >> > > >> > Or, who would be better to answer this question.
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > I have groked as many articles on how to setup this
> configuration
> >> as I
> >> > > >> can.
> >> > > >> > And, I have read a good deal of the archives of this mailing
> list.
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > Linux distributions I have tried are Debian, Ubuntu and Amazon
> >> linux.
> >> > > >> This
> >> > > >> > is been tested on both AWS and VMware fusion.
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > ------
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > I have a feeling I am missing something simple...
> >> > > >> >
> >> > > >> >
> >> > > >> >
> >> > > >> > Thanks!
> >> > > >> >
> >> > > >> > -Timothy
> >> > > >>
> >> > > >> Regards
> >> > > >>
> >> > > >> --
> >> > > >> Julian Anastasov <ja@ssi.bg>
> >> > > >>
> >> > > > _______________________________________________
> >> > > > Please read the documentation before posting - it's available at:
> >> > > > http://www.linuxvirtualserver.org/
> >> > > >
> >> > > > LinuxVirtualServer.org mailing list -
> >> lvs-users@LinuxVirtualServer.org
> >> > > > Send requests to lvs-users-request@LinuxVirtualServer.org
> >> > > > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> >> > >
> >> > > _______________________________________________
> >> > > Please read the documentation before posting - it's available at:
> >> > > http://www.linuxvirtualserver.org/
> >> > >
> >> > > LinuxVirtualServer.org mailing list -
> lvs-users@LinuxVirtualServer.org
> >> > > Send requests to lvs-users-request@LinuxVirtualServer.org
> >> > > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> >> > >
> >> > _______________________________________________
> >> > Please read the documentation before posting - it's available at:
> >> > http://www.linuxvirtualserver.org/
> >> >
> >> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> >> > Send requests to lvs-users-request@LinuxVirtualServer.org
> >> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> >>
> >> _______________________________________________
> >> Please read the documentation before posting - it's available at:
> >> http://www.linuxvirtualserver.org/
> >>
> >> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> >> Send requests to lvs-users-request@LinuxVirtualServer.org
> >> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
> >>
> > _______________________________________________
> > Please read the documentation before posting - it's available at:
> > http://www.linuxvirtualserver.org/
> >
> > LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> > Send requests to lvs-users-request@LinuxVirtualServer.org
> > or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users