Mailing List Archive

[lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t
Formerly both of my load balancers were running RHEL 5.2 and everything has been awesome for years. I reformatted them and installed RHEL 6.3 (because that's what's on the rest of my servers) and now we're seeing application pauses and freezes. If we bypass the load balancer and point the client application directly at one of the tomcat servers, it performs like a dream. If we re-point the application back at the load balancer, the pauses and freezes return.

Connectivity from the client to the load balancer is solid. Using ping -f (ping flooding) or tcping, we can send half a million packets with zero drops and no high latency. Likewise, connectivity from the load balancer to the tomcat realservers is just as solid. However, we have observed TCP retransmissions in the communication between the load balancer and the realservers, which may correspond to application freezes.

Aside from the upgrade to RHEL 6.3, we also added two additional NICs and enabled bonding (mode 1). The clients connect to the load balancer on bond1. The load balancer forwards the packets to the realservers on bond0.

Is anyone aware of issues with LVS and bonding? Or problems with LVS and RHEL 6.3?

--Eric Robinson



_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...

Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:29 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:34 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:38 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:43 lb01a kernel: IPVS: WRR: no destination available: no destinations present
Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no destinations present

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
Eric,

What kind of health checks are you using, it sounds like no backend
servers available in the cluster?

On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@psmnv.com> wrote:
> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>
> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:29 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:34 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:38 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:43 lb01a kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no destinations present
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
That's the odd part. We don't have any WRR health checks at all. Everything is WLC. And the ldirectord.cf file did not change, not did the version of ldirectord itself. I have checked through the file, and there are no virtual servers that are not backed by realservers, and there are no realservers showing in a down state. I would post the file but it is quite long. We have approximately 1700 virtual services configured.

--Eric


-----Original Message-----
From: lvs-users-bounces@linuxvirtualserver.org [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
Sent: Friday, September 18, 2015 2:20 PM
To: LinuxVirtualServer.org users mailing list. <lvs-users@linuxvirtualserver.org>
Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t

Eric,

What kind of health checks are you using, it sounds like no backend servers available in the cluster?

On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@psmnv.com> wrote:
> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>
> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no
> destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
> destination available: no destinations present Sep 18 09:06:29 lb01a
> kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no
> destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
> destination available: no destinations present Sep 18 09:06:34 lb01a
> kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no
> destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
> destination available: no destinations present Sep 18 09:06:38 lb01a
> kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no
> destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
> destination available: no destinations present Sep 18 09:06:43 lb01a
> kernel: IPVS: WRR: no destination available: no destinations present
> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no
> destinations present
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
Eric,

Ah , OK I thought that might be too simple :-).

Some general advice from our end is:
We have had problems with both LRO & GRO in the past (we disable them
both), and we also run irqbalance (to save on CPU overhead from
network traffic).
Also we tend to use Intel NICs with the latest drivers.
We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm
afraid I can't be much help.





On 18 September 2015 at 23:54, Eric Robinson <eric.robinson@psmnv.com> wrote:
> That's the odd part. We don't have any WRR health checks at all. Everything is WLC. And the ldirectord.cf file did not change, not did the version of ldirectord itself. I have checked through the file, and there are no virtual servers that are not backed by realservers, and there are no realservers showing in a down state. I would post the file but it is quite long. We have approximately 1700 virtual services configured.
>
> --Eric
>
>
> -----Original Message-----
> From: lvs-users-bounces@linuxvirtualserver.org [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
> Sent: Friday, September 18, 2015 2:20 PM
> To: LinuxVirtualServer.org users mailing list. <lvs-users@linuxvirtualserver.org>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t
>
> Eric,
>
> What kind of health checks are you using, it sounds like no backend servers available in the cluster?
>
> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@psmnv.com> wrote:
>> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>>
>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:29 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:34 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:38 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:43 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
Malcom -- My wife makes wonderful fudge at Christmas time. Where should I send yours?

Because...

ethtool -K p2p1 gro off
ethtool -K p2p2 gro off
ethtool -K p2p3 gro off
ethtool -K p2p4 gro off

...totally fixed it! The change was instantaneous. I didn't even have to down/up the interfaces.

I'm sitting here stunned. The application isn't freezing any more. I hardly know what to do with myself.

Thanks!

--Eric

-----Original Message-----
From: lvs-users-bounces@linuxvirtualserver.org [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
Sent: Saturday, September 19, 2015 12:05 AM
To: LinuxVirtualServer.org users mailing list. <lvs-users@linuxvirtualserver.org>
Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t

Eric,

Ah , OK I thought that might be too simple :-).

Some general advice from our end is:
We have had problems with both LRO & GRO in the past (we disable them both), and we also run irqbalance (to save on CPU overhead from network traffic).
Also we tend to use Intel NICs with the latest drivers.
We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm afraid I can't be much help.





On 18 September 2015 at 23:54, Eric Robinson <eric.robinson@psmnv.com> wrote:
> That's the odd part. We don't have any WRR health checks at all. Everything is WLC. And the ldirectord.cf file did not change, not did the version of ldirectord itself. I have checked through the file, and there are no virtual servers that are not backed by realservers, and there are no realservers showing in a down state. I would post the file but it is quite long. We have approximately 1700 virtual services configured.
>
> --Eric
>
>
> -----Original Message-----
> From: lvs-users-bounces@linuxvirtualserver.org
> [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm
> Turnbull
> Sent: Friday, September 18, 2015 2:20 PM
> To: LinuxVirtualServer.org users mailing list.
> <lvs-users@linuxvirtualserver.org>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
> to Sh*t
>
> Eric,
>
> What kind of health checks are you using, it sounds like no backend servers available in the cluster?
>
> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@psmnv.com> wrote:
>> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>>
>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:29 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:34 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:38 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
>> destination available: no destinations present Sep 18 09:06:43 lb01a
>> kernel: IPVS: WRR: no destination available: no destinations present
>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no
>> destinations present
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list -
>> lvs-users@LinuxVirtualServer.org Send requests to
>> lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
Eric,

That's great news, glad to hear that it is working!

Fudge sounds nice :-). But maybe just spread the word that
Loadbalancer.org are nice people?
http://loadbalancer.org/blog/so-just-how-should-you-compare-load-balancers

I remember now that we did some work with Horms to fix GRO for a
customer of ours who needed it (but RHEL 6.3 probably doesn't have it)

Dug out of my email:
In all 2.6.39 was a rather busy release for LVS with a total of 92 patches.
Some highlights are:

Features
* Network namespace support
* New synchronisation protocol
* Optimised statistics reading

Bug Fixes
* Allow transmit of GRO aggregated skbs
* Allow fragmented packets to be handled by SIP persistence engine
* Correct zeroing of statistics


On 19 September 2015 at 08:28, Eric Robinson <eric.robinson@psmnv.com> wrote:
> Malcom -- My wife makes wonderful fudge at Christmas time. Where should I send yours?
>
> Because...
>
> ethtool -K p2p1 gro off
> ethtool -K p2p2 gro off
> ethtool -K p2p3 gro off
> ethtool -K p2p4 gro off
>
> ...totally fixed it! The change was instantaneous. I didn't even have to down/up the interfaces.
>
> I'm sitting here stunned. The application isn't freezing any more. I hardly know what to do with myself.
>
> Thanks!
>
> --Eric
>
> -----Original Message-----
> From: lvs-users-bounces@linuxvirtualserver.org [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
> Sent: Saturday, September 19, 2015 12:05 AM
> To: LinuxVirtualServer.org users mailing list. <lvs-users@linuxvirtualserver.org>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t
>
> Eric,
>
> Ah , OK I thought that might be too simple :-).
>
> Some general advice from our end is:
> We have had problems with both LRO & GRO in the past (we disable them both), and we also run irqbalance (to save on CPU overhead from network traffic).
> Also we tend to use Intel NICs with the latest drivers.
> We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm afraid I can't be much help.
>
>
>
>
>
> On 18 September 2015 at 23:54, Eric Robinson <eric.robinson@psmnv.com> wrote:
>> That's the odd part. We don't have any WRR health checks at all. Everything is WLC. And the ldirectord.cf file did not change, not did the version of ldirectord itself. I have checked through the file, and there are no virtual servers that are not backed by realservers, and there are no realservers showing in a down state. I would post the file but it is quite long. We have approximately 1700 virtual services configured.
>>
>> --Eric
>>
>>
>> -----Original Message-----
>> From: lvs-users-bounces@linuxvirtualserver.org
>> [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm
>> Turnbull
>> Sent: Friday, September 18, 2015 2:20 PM
>> To: LinuxVirtualServer.org users mailing list.
>> <lvs-users@linuxvirtualserver.org>
>> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
>> to Sh*t
>>
>> Eric,
>>
>> What kind of health checks are you using, it sounds like no backend servers available in the cluster?
>>
>> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@psmnv.com> wrote:
>>> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>>>
>>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:29 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:34 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:38 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:43 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available: no
>>> destinations present
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list -
>>> lvs-users@LinuxVirtualServer.org Send requests to
>>> lvs-users-request@LinuxVirtualServer.org
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>>
>>
>> --
>> Regards,
>>
>> Malcolm Turnbull.
>>
>> Loadbalancer.org Ltd.
>> Phone: +44 (0)330 1604540
>> http://www.loadbalancer.org/
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
I've been looking for an alternative to ldirectord because it is single-threaded. My config file is so long that it now takes about 3 or 4 minutes to health-check all my servers, which means a server could be down for that long before the load balancer notices. Ldirectord does have the advantage of being free, which is nice, but I might be interested in something that isn't too expensive. Is loadbalancer.org what I'm looking for?

--Eric

-----Original Message-----
From: lvs-users-bounces@linuxvirtualserver.org [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
Sent: Saturday, September 19, 2015 1:24 AM
To: LinuxVirtualServer.org users mailing list. <lvs-users@linuxvirtualserver.org>
Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t

Eric,

That's great news, glad to hear that it is working!

Fudge sounds nice :-). But maybe just spread the word that Loadbalancer.org are nice people?
http://loadbalancer.org/blog/so-just-how-should-you-compare-load-balancers

I remember now that we did some work with Horms to fix GRO for a customer of ours who needed it (but RHEL 6.3 probably doesn't have it)

Dug out of my email:
In all 2.6.39 was a rather busy release for LVS with a total of 92 patches.
Some highlights are:

Features
* Network namespace support
* New synchronisation protocol
* Optimised statistics reading

Bug Fixes
* Allow transmit of GRO aggregated skbs
* Allow fragmented packets to be handled by SIP persistence engine
* Correct zeroing of statistics


On 19 September 2015 at 08:28, Eric Robinson <eric.robinson@psmnv.com> wrote:
> Malcom -- My wife makes wonderful fudge at Christmas time. Where should I send yours?
>
> Because...
>
> ethtool -K p2p1 gro off
> ethtool -K p2p2 gro off
> ethtool -K p2p3 gro off
> ethtool -K p2p4 gro off
>
> ...totally fixed it! The change was instantaneous. I didn't even have to down/up the interfaces.
>
> I'm sitting here stunned. The application isn't freezing any more. I hardly know what to do with myself.
>
> Thanks!
>
> --Eric
>
> -----Original Message-----
> From: lvs-users-bounces@linuxvirtualserver.org
> [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm
> Turnbull
> Sent: Saturday, September 19, 2015 12:05 AM
> To: LinuxVirtualServer.org users mailing list.
> <lvs-users@linuxvirtualserver.org>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
> to Sh*t
>
> Eric,
>
> Ah , OK I thought that might be too simple :-).
>
> Some general advice from our end is:
> We have had problems with both LRO & GRO in the past (we disable them both), and we also run irqbalance (to save on CPU overhead from network traffic).
> Also we tend to use Intel NICs with the latest drivers.
> We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm afraid I can't be much help.
>
>
>
>
>
> On 18 September 2015 at 23:54, Eric Robinson <eric.robinson@psmnv.com> wrote:
>> That's the odd part. We don't have any WRR health checks at all. Everything is WLC. And the ldirectord.cf file did not change, not did the version of ldirectord itself. I have checked through the file, and there are no virtual servers that are not backed by realservers, and there are no realservers showing in a down state. I would post the file but it is quite long. We have approximately 1700 virtual services configured.
>>
>> --Eric
>>
>>
>> -----Original Message-----
>> From: lvs-users-bounces@linuxvirtualserver.org
>> [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of
>> Malcolm Turnbull
>> Sent: Friday, September 18, 2015 2:20 PM
>> To: LinuxVirtualServer.org users mailing list.
>> <lvs-users@linuxvirtualserver.org>
>> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
>> to Sh*t
>>
>> Eric,
>>
>> What kind of health checks are you using, it sounds like no backend servers available in the cluster?
>>
>> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@psmnv.com> wrote:
>>> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>>>
>>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available:
>>> no destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:29 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available:
>>> no destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:34 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available:
>>> no destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:38 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available:
>>> no destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
>>> destination available: no destinations present Sep 18 09:06:43 lb01a
>>> kernel: IPVS: WRR: no destination available: no destinations present
>>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available:
>>> no destinations present
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list -
>>> lvs-users@LinuxVirtualServer.org Send requests to
>>> lvs-users-request@LinuxVirtualServer.org
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>>
>>
>> --
>> Regards,
>>
>> Malcolm Turnbull.
>>
>> Loadbalancer.org Ltd.
>> Phone: +44 (0)330 1604540
>> http://www.loadbalancer.org/
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list -
>> lvs-users@LinuxVirtualServer.org Send requests to
>> lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list -
>> lvs-users@LinuxVirtualServer.org Send requests to
>> lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t [ In reply to ]
Eric,

Our usual advice is that for anything with over 200 health checks
(which your configuration sounds like it definitely is) then you
should look carefully at your options for simplifying the checks.

We do use a version of Ldirectord but each virtual service runs its
health check in a sub process, so the real limitation is number of
backend servers per virtual service.

Maybe you could send me your configuration file off list
(malcolm@loadbalancer.org) and I'll get someone to test the
performance of the health checks on our system?

It sounds like the kind of configuration that would definitely
highlight any scalability issues in our product :-).











On 19 September 2015 at 16:50, Eric Robinson <eric.robinson@psmnv.com> wrote:
> I've been looking for an alternative to ldirectord because it is single-threaded. My config file is so long that it now takes about 3 or 4 minutes to health-check all my servers, which means a server could be down for that long before the load balancer notices. Ldirectord does have the advantage of being free, which is nice, but I might be interested in something that isn't too expensive. Is loadbalancer.org what I'm looking for?
>
> --Eric
>
> -----Original Message-----
> From: lvs-users-bounces@linuxvirtualserver.org [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm Turnbull
> Sent: Saturday, September 19, 2015 1:24 AM
> To: LinuxVirtualServer.org users mailing list. <lvs-users@linuxvirtualserver.org>
> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went to Sh*t
>
> Eric,
>
> That's great news, glad to hear that it is working!
>
> Fudge sounds nice :-). But maybe just spread the word that Loadbalancer.org are nice people?
> http://loadbalancer.org/blog/so-just-how-should-you-compare-load-balancers
>
> I remember now that we did some work with Horms to fix GRO for a customer of ours who needed it (but RHEL 6.3 probably doesn't have it)
>
> Dug out of my email:
> In all 2.6.39 was a rather busy release for LVS with a total of 92 patches.
> Some highlights are:
>
> Features
> * Network namespace support
> * New synchronisation protocol
> * Optimised statistics reading
>
> Bug Fixes
> * Allow transmit of GRO aggregated skbs
> * Allow fragmented packets to be handled by SIP persistence engine
> * Correct zeroing of statistics
>
>
> On 19 September 2015 at 08:28, Eric Robinson <eric.robinson@psmnv.com> wrote:
>> Malcom -- My wife makes wonderful fudge at Christmas time. Where should I send yours?
>>
>> Because...
>>
>> ethtool -K p2p1 gro off
>> ethtool -K p2p2 gro off
>> ethtool -K p2p3 gro off
>> ethtool -K p2p4 gro off
>>
>> ...totally fixed it! The change was instantaneous. I didn't even have to down/up the interfaces.
>>
>> I'm sitting here stunned. The application isn't freezing any more. I hardly know what to do with myself.
>>
>> Thanks!
>>
>> --Eric
>>
>> -----Original Message-----
>> From: lvs-users-bounces@linuxvirtualserver.org
>> [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of Malcolm
>> Turnbull
>> Sent: Saturday, September 19, 2015 12:05 AM
>> To: LinuxVirtualServer.org users mailing list.
>> <lvs-users@linuxvirtualserver.org>
>> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
>> to Sh*t
>>
>> Eric,
>>
>> Ah , OK I thought that might be too simple :-).
>>
>> Some general advice from our end is:
>> We have had problems with both LRO & GRO in the past (we disable them both), and we also run irqbalance (to save on CPU overhead from network traffic).
>> Also we tend to use Intel NICs with the latest drivers.
>> We do use Centos 6.3 (roughly) but with a more recent Kernel so I'm afraid I can't be much help.
>>
>>
>>
>>
>>
>> On 18 September 2015 at 23:54, Eric Robinson <eric.robinson@psmnv.com> wrote:
>>> That's the odd part. We don't have any WRR health checks at all. Everything is WLC. And the ldirectord.cf file did not change, not did the version of ldirectord itself. I have checked through the file, and there are no virtual servers that are not backed by realservers, and there are no realservers showing in a down state. I would post the file but it is quite long. We have approximately 1700 virtual services configured.
>>>
>>> --Eric
>>>
>>>
>>> -----Original Message-----
>>> From: lvs-users-bounces@linuxvirtualserver.org
>>> [mailto:lvs-users-bounces@linuxvirtualserver.org] On Behalf Of
>>> Malcolm Turnbull
>>> Sent: Friday, September 18, 2015 2:20 PM
>>> To: LinuxVirtualServer.org users mailing list.
>>> <lvs-users@linuxvirtualserver.org>
>>> Subject: Re: [lvs-users] Upgraded my Load Balancers, Performance Went
>>> to Sh*t
>>>
>>> Eric,
>>>
>>> What kind of health checks are you using, it sounds like no backend servers available in the cluster?
>>>
>>> On 18 September 2015 at 17:11, Eric Robinson <eric.robinson@psmnv.com> wrote:
>>>> ...oh, and one other thing. After the upgrade, I am now seeing these syslog messages once approximately per second...
>>>>
>>>> Sep 18 09:06:26 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:28 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:29 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:31 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:32 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:34 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:35 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:37 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:38 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:40 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present Sep 18 09:06:41 lb01a kernel: IPVS: WRR: no
>>>> destination available: no destinations present Sep 18 09:06:43 lb01a
>>>> kernel: IPVS: WRR: no destination available: no destinations present
>>>> Sep 18 09:06:44 lb01a kernel: IPVS: WRR: no destination available:
>>>> no destinations present
>>>>
>>>> _______________________________________________
>>>> Please read the documentation before posting - it's available at:
>>>> http://www.linuxvirtualserver.org/
>>>>
>>>> LinuxVirtualServer.org mailing list -
>>>> lvs-users@LinuxVirtualServer.org Send requests to
>>>> lvs-users-request@LinuxVirtualServer.org
>>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Malcolm Turnbull.
>>>
>>> Loadbalancer.org Ltd.
>>> Phone: +44 (0)330 1604540
>>> http://www.loadbalancer.org/
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list -
>>> lvs-users@LinuxVirtualServer.org Send requests to
>>> lvs-users-request@LinuxVirtualServer.org
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>>
>>> _______________________________________________
>>> Please read the documentation before posting - it's available at:
>>> http://www.linuxvirtualserver.org/
>>>
>>> LinuxVirtualServer.org mailing list -
>>> lvs-users@LinuxVirtualServer.org Send requests to
>>> lvs-users-request@LinuxVirtualServer.org
>>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>>
>>
>> --
>> Regards,
>>
>> Malcolm Turnbull.
>>
>> Loadbalancer.org Ltd.
>> Phone: +44 (0)330 1604540
>> http://www.loadbalancer.org/
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>>
>> _______________________________________________
>> Please read the documentation before posting - it's available at:
>> http://www.linuxvirtualserver.org/
>>
>> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
>> Send requests to lvs-users-request@LinuxVirtualServer.org
>> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
>
>
> --
> Regards,
>
> Malcolm Turnbull.
>
> Loadbalancer.org Ltd.
> Phone: +44 (0)330 1604540
> http://www.loadbalancer.org/
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
>
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
> Send requests to lvs-users-request@LinuxVirtualServer.org
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users



--
Regards,

Malcolm Turnbull.

Loadbalancer.org Ltd.
Phone: +44 (0)330 1604540
http://www.loadbalancer.org/

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users