Mailing List Archive

[lvs-users] Kernel Panic when adding a service
Dear list,

I've used IPVS with ldirectord on Ubuntu for a couple of years, never had much trouble. Recently, however, I've tried to setup a new HA Load Balancer cluster for DNS and I'm running into reproducable kernel panics.

I've narrowed the panics down to the point where i run "sudo /sbin/ipvsadm -A -u 10.0.50.4:53" and a panic immediately occurs.

The last part of the stack track can be seen in this image: http://cl.ly/image/1K3P1Q0Y242t

My setup is a Virtual Machine (VMWare) running a clean install of Ubuntu 13.04, all updates installed, running kernel 3.8.0-31-generic.

I've installed no extra packages, except for heartbeat and ldirectord.

Running ipvsadm -v produces the following result:
ipvsadm v1.26 2008/5/15 (compiled with popt and IPVS v1.2.1)

Running ldirectord -v produces the following result:
Linux Director v1.186-ha

I've tried googling, but could find no results for my specific scenario. Any help would be greatly appreciated.

Best regards,
Luc van Donkersgoed
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Kernel Panic when adding a service [ In reply to ]
> Dear list,
>
> I've used IPVS with ldirectord on Ubuntu for a couple of years, never had much trouble. Recently, however, I've tried to setup a new HA Load Balancer cluster for DNS and I'm running into reproducable kernel panics.
>
> I've narrowed the panics down to the point where i run "sudo /sbin/ipvsadm -A -u 10.0.50.4:53" and a panic immediately occurs.
>
> The last part of the stack track can be seen in this image: http://cl.ly/image/1K3P1Q0Y242t
>
> My setup is a Virtual Machine (VMWare) running a clean install of Ubuntu 13.04, all updates installed, running kernel 3.8.0-31-generic.
>
> I've installed no extra packages, except for heartbeat and ldirectord.
>
> Running ipvsadm -v produces the following result:
> ipvsadm v1.26 2008/5/15 (compiled with popt and IPVS v1.2.1)
>
> Running ldirectord -v produces the following result:
> Linux Director v1.186-ha
>
> I've tried googling, but could find no results for my specific scenario. Any help would be greatly appreciated.
>
> Best regards,
> Luc van Donkersgoed
>

After some more research, my problem seems to lie with the kernel. I cleaned the virtual machine and reinstalled Ubuntu 13.04 from an older ISO. After installation, the VM was running kernel 3.8.0-19-generic.

I then installed ldirectord and heartbeat, added a service through 'sudo /sbin/ipvsadm -A -u 10.0.50.4:53', and everything worked fine. I then ran a 'sudo apt-get dist-upgrade', which prompted the following:

The following NEW packages will be installed:
linux-headers-3.8.0-31 linux-headers-3.8.0-31-generic linux-image-3.8.0-31-generic linux-image-extra-3.8.0-31-generic
The following packages will be upgraded:
apport apport-symptoms apt apt-transport-https apt-utils apt-xapian-index bind9-host command-not-found command-not-found-data curl dbus dnsutils gnupg gpgv ifupdown isc-dhcp-client
isc-dhcp-common language-selector-common libapt-inst1.5 libapt-pkg4.12 libasn1-8-heimdal libbind9-90 libcurl3 libcurl3-gnutls libdbus-1-3 libdns95 libdrm2 libgcrypt11 libgnutls26
libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libisc92 libisccc90 libisccfg90 libkrb5-26-heimdal libldap-2.4-2 liblwres90
libplymouth2 libpolkit-agent-1-0 libpolkit-backend-1-0 libpolkit-gobject-1-0 libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3.3-minimal libpython3.3-stdlib
libroken18-heimdal libsasl2-2 libsasl2-modules libssl1.0.0 libudev1 libwhoopsie0 libwind0-heimdal libx11-6 libx11-data libxcb1 libxext6 libxml2 linux-generic linux-headers-3.8.0-19
linux-headers-3.8.0-19-generic linux-headers-generic linux-image-3.8.0-19-generic linux-image-extra-3.8.0-19-generic linux-image-generic login lsb-base lsb-release openssl passwd plymouth
plymouth-theme-ubuntu-text policykit-1 python-openssl python2.7 python2.7-minimal python3-apport python3-commandnotfound python3-distupgrade python3-problem-report
python3-software-properties python3-update-manager python3.3 python3.3-minimal rsyslog software-properties-common tzdata ubuntu-release-upgrader-core update-manager-core whoopsie

After the installation of the kernel and packages, I added the service again, which still worked fine. Then I rebooted the machine, tried to add the service, and instantly got a kernel panic.

My conclusion is that the 3.8.0-31 kernel or one of the upgraded packages must be the culprit.

I would love to hear your thoughts on this.

Best,
Luc van Donkersgoed


_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Re: [lvs-users] Kernel Panic when adding a service [ In reply to ]
> > My conclusion is that the 3.8.0-31 kernel or one of the upgraded
> > packages must be the culprit.
>
> Probably, but you could perhaps test by upgrading only the kernel.
>
> > I would love to hear your thoughts on this.
>
> This should be reported to Ubuntu. Without knowing their kernels, it's
> hard to tell if this is an upstream Linux problem or not.
> --
> Regards,
> Feri.

I have determined the panic occurs on Ubuntu kernel 3.8.0-29 and higher, and have filed a bug report at https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1238494

Thank you for your help.

Best,
Luc

_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/

LinuxVirtualServer.org mailing list - lvs-users@LinuxVirtualServer.org
Send requests to lvs-users-request@LinuxVirtualServer.org
or go to http://lists.graemef.net/mailman/listinfo/lvs-users