Mailing List Archive

[lvs-users] Real server not responding back
Hi,

I'm trying to get LVS/IPVS to work for my desired configuration, but facing
a weird problem, most likely due to a simple mistake somewhere.

For now, I've created one load balancer VM (running Ubuntu 18.04 with LVS
director 1.28-3) and one real server VM (running Ubuntu 18.04).

Both the VMs are in different data-centres (different networks), so I'm
trying to make the load balancer and real server work over IP tunneling
mode based on this guide:
https://medium.com/@ppan.brian/ipvs-using-ipip-tunnel-ca180c7f4fd8

I've got it working to the point where if a client sends a request to load
balancer VIP, then it forwards the request to the real server, which is
running a simple HTTP web server 'python3 -m http.server 8000' (has a
'Hello World' index page), but the real server never responds back, and the
request times-out.

Using tcpdump, I can see the request hitting the director, and then hitting
the real server, and the real server responding back to the client IP with
a zero length response (ack?), that goes on for 4-5 times until timeout.
Tunneling seems to be working but the web server doesn't intercept and
respond to the request. Requesting the real server IP directly works fine
though.

I'll appreciate if someone can share some tips or suggestions to resolve
this issue.

Cheers,

Nick
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Nick Wilson wrote:

> Hi,
>
> I'm trying to get LVS/IPVS to work for my desired configuration, but
> facing a weird problem, most likely due to a simple mistake somewhere.
>
> For now, I've created one load balancer VM (running Ubuntu 18.04 with
> LVS director 1.28-3) and one real server VM (running Ubuntu 18.04).
>
> Both the VMs are in different data-centres (different networks), so
> I'm trying to make the load balancer and real server work over IP
> tunneling mode based on this guide:
> https://medium.com/@ppan.brian/ipvs-using-ipip-tunnel-ca180c7f4fd8

FWIW, I have had an LVS on IPIP tunneling setup running for 14-15 years,
currently with some 80 backends, also spread across multiple
datacentres.

> I've got it working to the point where if a client sends a request to
> load balancer VIP, then it forwards the request to the real server,
> which is running a simple HTTP web server 'python3 -m http.server
> 8000' (has a 'Hello World' index page), but the real server never
> responds back, and the request times-out.
> Using tcpdump, I can see the request hitting the director, and then
> hitting the real server, and the real server responding back to the
> client IP with a zero length response (ack?), that goes on for 4-5
> times until timeout. Tunneling seems to be working but the web server
> doesn't intercept and respond to the request. Requesting the real
> server IP directly works fine though.

Wait - you say "hitting the real server, and the real server responding
back to the client IP with a zero length response (ack?)", but
then "but the web server doesn't intercept and respond to the
request" ?

Dunno if this'll help, but maybe:
My setup, very briefly -

2 directors, 80 backends. Each backend is connected via an IPIP tunnel
with a network range 10.0.x.x/30 assigned.

# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 88.198.198.123:25 wlc
-> 10.0.1.146:25 Masq 1000 10 13
-> 10.0.1.142:25 Masq 1000 11 7
-> 10.0.1.138:25 Masq 1000 11 6
-> 10.0.1.134:25 Masq 1000 11 4
-> 10.0.1.130:25 Masq 1000 10 10
[snip]

4: ipip0@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1480 qdisc noqueue
state UNKNOWN group default qlen 1000
link/ipip 46.4.89.115 peer 88.198.198.125
inet 10.0.1.146/30 brd 10.0.1.147 scope global ipip0
valid_lft forever preferred_lft forever
inet6 fe80::200:5efe:2e04:5973/64 scope link
valid_lft forever preferred_lft forever


I have a separate route table :

# ip route show table fe1only
default via 10.0.1.145 dev ipip0
10.0.1.144/30 dev ipip0 scope link src 10.0.1.146
10.0.2.144/30 dev ipip1 scope link src 10.0.2.146
127.0.0.0/8 dev lo scope link

I direct traffic to use that table by setting an fwmark and using an ip
rule.

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
MARK tcp -- 0.0.0.0/0 46.4.89.115 tcp
dpt:10031 MARK set 0x14
MARK tcp -- 10.0.1.144/30 0.0.0.0/0 tcp dpt:25
MARK set 0xa





--
Per Jessen, Zürich (1.6°C)
http://www.hostsuisse.com/ - virtual servers, made in Switzerland.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Hi Nick,

> Using tcpdump, I can see the request hitting the director, and then hitting
> the real server, and the real server responding back to the client IP with
> a zero length response (ack?), that goes on for 4-5 times until timeout.

It sounds like the server's responses aren't making it through,
meaning that a TCP three-way handshake cannot be completed.

What is sitting in front of the real server, and *is it stateful*? A
router? A firewall?

Every time I've built a working LVS/TUN environment, without fail,
I've had to make configuration changes on the router or firewall
sitting in front of the real servers. Without doing so, the router or
firewall drops the return traffic.

When using a Linux router, I always disable the rp_filter. When using
a pfSense firewall, I create floating firewall rules to cover all TCP
flags and 'sloppy state keeping' on the inbound and outbound network
interfaces.

Does the virtual IP address on the real server look 'out of place' in
the context of the rest of the network? For example, if a router
expects to see addresses in 10.0.0.0/24 on eth0 and addresses in
192.168.0.0/24 on eth1 but it starts seeing traffic from 10.0.0.20
coming *in* on eth1 (e.g. from a VIP address) then the router may well
drop the return traffic. There's probably a more rigorous or 'correct'
way to describe this, but those are from my own practical notes on
setting up LVS/TUN environments.

I hope this helps.

Thanks,
Andrew

--
Andrew Howe
Loadbalancer.org Ltd.
www.loadbalancer.org

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Thanks for your reply, Per.

Wait - you say "hitting the real server, and the real server responding
> back to the client IP with a zero length response (ack?)", but
> then "but the web server doesn't intercept and respond to the
> request" ?
>

By 'web server' I meant the python web server software running on the real
server. It doesn't intercept and respond to the tunneled request, but works
fine on a direct request to the real server.

Anyway, I've tried everything I could, but still haven't been able to get
it to work. Andrew's reply suggests packet dropping somewhere, which is my
concern too.

Cheers,

Nick
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Thanks for your reply Andrew.


>
> It sounds like the server's responses aren't making it through,
> meaning that a TCP three-way handshake cannot be completed.
>
> What is sitting in front of the real server, and *is it stateful*? A
> router? A firewall?
>
>
There's no firewall in-front of the real server, while I'm testing LVS.
There should be router I'm guessing. As this a cloud hosted VM, I checked
with the hosting company, and they've confirmed that their network
equipment is not configured to drop any packets, and tunneling should work
fine.

I'll try it out on a different cloud host like DigitalOcean or Linode to
see if that makes a difference.


> When using a Linux router, I always disable the rp_filter. When using
> a pfSense firewall, I create floating firewall rules to cover all TCP
> flags and 'sloppy state keeping' on the inbound and outbound network
> interfaces.
>
>
rp_filter is disabled (set to zero for all interfaces) on the real server.

Does the virtual IP address on the real server look 'out of place' in
> the context of the rest of the network? For example, if a router
> expects to see addresses in 10.0.0.0/24 on eth0 and addresses in
> 192.168.0.0/24 on eth1 but it starts seeing traffic from 10.0.0.20
> coming *in* on eth1 (e.g. from a VIP address) then the router may well
> drop the return traffic.
>

Well, on the load balancer VM I've got two IPs - a static public IP bound
to ens3, and another static public IP (virtual IP) bound to ens3:0. Client
requests are made to the virtual IP. The real server has its own static
public IP bound to ens3, and the virtual IP is bound to tunl0 interface.
Load balancer VM and real server VM are located in different data centers,
so they're on different networks, and hence their IPs and gateways are
different. Nothing out of the ordinary is visible on the real server that
could be dropping the packets. Even this cloud hosting company has
confirmed that their network isn't configured to don't drop packets as such.

Please chime in if anything else comes to mind.

Cheers,

Nick
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Nick Wilson wrote:

> rp_filter is disabled (set to zero for all interfaces) on the real
> server.

On my real servers, rp_filter=2 for my two IPIP interfaces.

Verbatim from the setup script:

# 20190703 this is critical for the ipip tunneling
sysctl -w net.ipv4.conf.ipip0.rp_filter=2
sysctl -w net.ipv4.conf.ipip1.rp_filter=2



--
Per Jessen, Zürich (4.6°C)
http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
On 31 Mar 2020, at 08:47, Nick Wilson <vicnickw@gmail.com> wrote:
> Please chime in if anything else comes to mind.

A couple of things spring immediately to mind:

1. If you setup a tunnel without using LVS, can you get traffic to traverse it and be seen departing the source (director) and and arriving at the target (realserver) inside the tunnel interfaces?

2. If you do see that, do you have the load balanced IP address (VIP) on the loopback adapter on the realserver?

3. If you do have that, is your webserver listening on that IP?

I have more questions dependent on the answers above. They’re all pretty fundamental to get this working.

Graeme
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Hi Nick,

> There's no firewall in-front of the real server, while I'm testing LVS.
> There should be router I'm guessing. As this a cloud hosted VM, I checked
> with the hosting company, and they've confirmed that their network
> equipment is not configured to drop any packets, and tunneling should work
> fine.

Being in the cloud changes things. Not having access to the underlying
infrastructure, to make configuration changes and perform
troubleshooting, may make using LVS/TUN impossible.

Whatever the service provider has sitting at the edge of their/your
network is surely providing at least a basic level of security. Even
if there's just a simple router that is tracking connection states at
the edge it would surely drop the reply traffic from your real server.

Inbound packet:
Source = Load balancer's public IP address
Destination = Real server's public IP address

Whatever is sitting at the edge will forward the inbound packet to the
real server and will start tracking this as a new connection.

Reply traffic:
Source = VIP address (on the real server)
Destination = Client's IP address

Whatever is sitting at the edge is unable to match up the reply packet
with the inbound packet, as the IP addresses are different. The reply
packet looks like a new, unrelated connection: the response half of a
connection to a request that hasn't been observed. I think it would
appear as an unsolicited SYN-ACK and be dropped.

That's my theory, at least.

Could you test using a proxy style load balancer? That way, the
inbound and reply traffic would all match up nicely. If that works, it
would prove that your setup is sound and working and would narrow down
the problem to being TUN specific. For a test, HAProxy is a great
FLOSS proxy style load balancer.

Thanks,
Andrew

--
Andrew Howe
Loadbalancer.org Ltd.
www.loadbalancer.org

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Thanks Per.

I tried rp_filter=2 but it made no difference.

Cheers,

Nick


On Tue, 31 Mar 2020 at 21:12, Per Jessen <per@computer.org> wrote:

> Nick Wilson wrote:
>
> > rp_filter is disabled (set to zero for all interfaces) on the real
> > server.
>
> On my real servers, rp_filter=2 for my two IPIP interfaces.
>
> Verbatim from the setup script:
>
> # 20190703 this is critical for the ipip tunneling
> sysctl -w net.ipv4.conf.ipip0.rp_filter=2
> sysctl -w net.ipv4.conf.ipip1.rp_filter=2
>
>
>
> --
> Per Jessen, Zürich (4.6°C)
> http://www.cloudsuisse.com/ - your owncloud, hosted in Switzerland.
>
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Hi Graeme,

>> 1. If you setup a tunnel without using LVS, can you get traffic to
traverse it and be seen departing the source (director) and and arriving at
the target (realserver) inside the tunnel interfaces?

To be honest, I don't know how to setup a tunnel without using LVS, and
that's why LVS/IPVS as an abstraction seemed interesting to me. Do you know
of a simple guide/tutorial I can follow for setting-up a tunnel without LVS
to test things out? I'm guessing it will involve iptables.

>> 2. If you do see that, do you have the load balanced IP address (VIP) on
the loopback adapter on the realserver?

At the moment with LVS, VIP on the real-server is not set to the loopback
interface but the tunl0 interface.

>> 3. If you do have that, is your webserver listening on that IP?

Yes, python test web server is listening on port 8000 for all interfaces
(0.0.0.0, but also tried VIP specifically). It responds fine on the
real-server IP, but doesn't even intercept the tunneled request if incoming
on VIP. However, tcpdump shows the tunneled request arrive on the
real-server, and responded back with zero-length packets until eventual
timeout.

I can ping the VIP on the real-server.

Cheers,

Nick


On Tue, 31 Mar 2020 at 22:21, Graeme Fowler <graeme@graemef.net> wrote:

> On 31 Mar 2020, at 08:47, Nick Wilson <vicnickw@gmail.com> wrote:
> > Please chime in if anything else comes to mind.
>
> A couple of things spring immediately to mind:
>
> 1. If you setup a tunnel without using LVS, can you get traffic to
> traverse it and be seen departing the source (director) and and arriving at
> the target (realserver) inside the tunnel interfaces?
>
> 2. If you do see that, do you have the load balanced IP address (VIP) on
> the loopback adapter on the realserver?
>
> 3. If you do have that, is your webserver listening on that IP?
>
> I have more questions dependent on the answers above. They’re all pretty
> fundamental to get this working.
>
> Graeme
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Thanks for explaining it Andrew.

My guess is the same -- it's to do with cloud routing/networking, but the
cloud hosting company says that their infrastructure and equipment have no
restrictions in place, and tunneling should work fine.

>> Could you test using a proxy style load balancer? That way, the
>> inbound and reply traffic would all match up nicely. If that works, it
>> would prove that your setup is sound and working and would narrow down
>> the problem to being TUN specific. For a test, HAProxy is a great
>> FLOSS proxy style load balancer.

I just tried HAProxy to load balance to the same real-server, and it works
fine. I don't mind using HAProxy for my requirements, but an LVS tunnel
seemed interesting because the real-server could respond back directly to
the client, bypassing the load balancer.

Cheers,

Nick


On Tue, 31 Mar 2020 at 22:22, Andrew Howe <andrew.howe@loadbalancer.org>
wrote:

> Hi Nick,
>
> > There's no firewall in-front of the real server, while I'm testing LVS.
> > There should be router I'm guessing. As this a cloud hosted VM, I checked
> > with the hosting company, and they've confirmed that their network
> > equipment is not configured to drop any packets, and tunneling should
> work
> > fine.
>
> Being in the cloud changes things. Not having access to the underlying
> infrastructure, to make configuration changes and perform
> troubleshooting, may make using LVS/TUN impossible.
>
> Whatever the service provider has sitting at the edge of their/your
> network is surely providing at least a basic level of security. Even
> if there's just a simple router that is tracking connection states at
> the edge it would surely drop the reply traffic from your real server.
>
> Inbound packet:
> Source = Load balancer's public IP address
> Destination = Real server's public IP address
>
> Whatever is sitting at the edge will forward the inbound packet to the
> real server and will start tracking this as a new connection.
>
> Reply traffic:
> Source = VIP address (on the real server)
> Destination = Client's IP address
>
> Whatever is sitting at the edge is unable to match up the reply packet
> with the inbound packet, as the IP addresses are different. The reply
> packet looks like a new, unrelated connection: the response half of a
> connection to a request that hasn't been observed. I think it would
> appear as an unsolicited SYN-ACK and be dropped.
>
> That's my theory, at least.
>
> Could you test using a proxy style load balancer? That way, the
> inbound and reply traffic would all match up nicely. If that works, it
> would prove that your setup is sound and working and would narrow down
> the problem to being TUN specific. For a test, HAProxy is a great
> FLOSS proxy style load balancer.
>
> Thanks,
> Andrew
>
> --
> Andrew Howe
> Loadbalancer.org Ltd.
> www.loadbalancer.org
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Hello,

On Thu, 2 Apr 2020, Nick Wilson wrote:

> Hi Graeme,
>
> >> 1. If you setup a tunnel without using LVS, can you get traffic to
> traverse it and be seen departing the source (director) and and arriving at
> the target (realserver) inside the tunnel interfaces?
>
> To be honest, I don't know how to setup a tunnel without using LVS, and
> that's why LVS/IPVS as an abstraction seemed interesting to me. Do you know
> of a simple guide/tutorial I can follow for setting-up a tunnel without LVS
> to test things out? I'm guessing it will involve iptables.

I have an old document for tunnel troubleshooting. I just added
some configuration examples including recently supported tunnel modes, HTH:

http://ja.ssi.bg/TUN-HOWTO.txt

> At the moment with LVS, VIP on the real-server is not set to the loopback
> interface but the tunl0 interface.

It works, as long as tunl0/hidden is set to 1, if this flag is
used to stop ARP replies. Otherwise, the recommended device for VIPs
is lo.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Hi Julian,

> I have an old document for tunnel troubleshooting. I just added
> > some configuration examples including recently supported tunnel modes,
> HTH:
> >
> > http://ja.ssi.bg/TUN-HOWTO.txt <http://ja.ssi.bg/TUN-HOWTO.txt>
> >
> > At the moment with LVS, VIP on the real-server is not set to the loopback
> > interface but the tunl0 interface.
> >
> > It works, as long as tunl0/hidden is set to 1, if this flag is
> > used to stop ARP replies. Otherwise, the recommended device for VIPs
> > is lo.
> >
>

Thanks a lot for sharing that how-to document. It was pretty clear and
concise.

I followed the document to setup LVS once again from scratch, but
unfortunately it didn't resolve the response issue :(

This time I tried binding the VIP on 'lo' interface instead of 'tunl0' on
the real-server, and still bring tunl0 up as in your doc, but no luck.

All the troubleshooting steps in your doc, like 'ip route get...' resolve
fine.

I don't see any IPIP packet decoding happening on the real-server when I do
a tcpdump. Here's how it looks:

tunl0: CIP -> VIP (packet length 40; checksum correct)
ens3: VIP -> CIP (packet length 0; checksum correct)

This goes on for 4-5 times until timeout on the client.

Anyway, please share if anything else comes to your mind.

Have a good day.

Cheers,

Nick
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Hello,

On Fri, 3 Apr 2020, Nick Wilson wrote:

> I followed the document to setup LVS once again from scratch, but
> unfortunately it didn't resolve the response issue :(
>
> This time I tried binding the VIP on 'lo' interface instead of 'tunl0' on
> the real-server, and still bring tunl0 up as in your doc, but no luck.
>
> All the troubleshooting steps in your doc, like 'ip route get...' resolve
> fine.
>
> I don't see any IPIP packet decoding happening on the real-server when I do
> a tcpdump. Here's how it looks:
>
> tunl0: CIP -> VIP (packet length 40; checksum correct)

If you see traffic on tunl0 then the IPIP header is already
removed and you see CIP->VIP TCP packet. Before that, you should see
IPIP DIP->RIP packet on the ens3 (input device).

> ens3: VIP -> CIP (packet length 0; checksum correct)

OK, kernel sends SYN+ACK ? Note that the server application (the
listener) may run in mode where it wants to see the first data, so
the server may not wakeup for this first packet. In this case, the
kernel still sends the SYN+ACK (3-way handshake performed without
wakeup). Wakeup occurs on 3th packet which can come with data, eg.
GET request (if HTTP). Such mode is suitable for servers that
expect first data from client, eg. HTTP. OTOH, for SMTP, the
first packet is sent by server, so this mode should not be used
by the listener (TCP_DEFER_ACCEPT).

> This goes on for 4-5 times until timeout on the client.

So, if you see VIP->CIP SYN+ACK sent by real server, it
means the ISP filters the packet and it does not reach the
client. Client retries. Problem in ISP.

Check the procedure under Q.3. traceroute will send UDP
traffic VIP->CIP which should generate ICMP errors. Such ICMP
errors are sent by every hop in the path to client. Then you
know which hop receives the traffic from real server. Still,
some hops may refuse to send ICMP, so such test can be confusing.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Real server not responding back [ In reply to ]
Thanks for your notes.

If you see traffic on tunl0 then the IPIP header is already
> removed and you see CIP->VIP TCP packet. Before that, you should see
> IPIP DIP->RIP packet on the ens3 (input device).
>
>
My bad, I can see IPIP with a wider tcpdump filter. Flow is like:

ens3: DIP -> RIP (proto IPIP)
tunl0: CIP -> VIP
ens3: VIP -> CIP (length 0)

OK, kernel sends SYN+ACK ? Note that the server application (the
> listener) may run in mode where it wants to see the first data, so
> the server may not wakeup for this first packet. In this case, the
> kernel still sends the SYN+ACK (3-way handshake performed without
> wakeup). Wakeup occurs on 3th packet which can come with data, eg.
> GET request (if HTTP). Such mode is suitable for servers that
> expect first data from client, eg. HTTP. OTOH, for SMTP, the
> first packet is sent by server, so this mode should not be used
> by the listener (TCP_DEFER_ACCEPT).
>
>
It does like a SYN+ACK. Application on the real-server I'm using to test is
a simple 'python3 -m http.server', which responds to curl on RIP:8000 but
not on VIP:8000.

> This goes on for 4-5 times until timeout on the client.
>
> So, if you see VIP->CIP SYN+ACK sent by real server, it
> means the ISP filters the packet and it does not reach the
> client. Client retries. Problem in ISP.
>
>
ISP filtering is the most likely cause of this problem, although they say
otherwise.


> Check the procedure under Q.3. traceroute will send UDP
> traffic VIP->CIP which should generate ICMP errors. Such ICMP
> errors are sent by every hop in the path to client. Then you
> know which hop receives the traffic from real server. Still,
> some hops may refuse to send ICMP, so such test can be confusing.
>
>
I couldn't get the 'traceroute -n -s VIP CIP' command (how-to Q.3) to work,
because the traceroute package on real-server's Ubuntu 18.04 doesn't
support the '-s' (source IP) argument.

Cheers,

Nick
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users