Mailing List Archive: Network dies and kernel errors

Network dies and kernel errors

Jul 25, 2011, 12:18 PM

Post #1 of 12 (630 views)

Have a new amd 6100 based server.
http://www.supermicro.com/Aplus/system/2U/2022/AS-2022G-URF.cfm
Running debian squeeze with debian 2.6.32 xen kernel
Running xen 4.1.1 built from source from xen.org

I'm seeing 2 errors.
during boot get this:

[ 0.004823] ------------[ cut here ]------------
[ 0.004833] WARNING:
at /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/build/source_amd64_xen/arch/x86/xen/enlighten.c:726
init_hw_perf_events+0x32d/0x3cd()
[ 0.004838] Hardware name: H8DGU
[ 0.004841] Modules linked in:
[ 0.004847] Pid: 0, comm: swapper Not tainted 2.6.32-5-xen-amd64 #1
[ 0.004850] Call Trace:
[ 0.004857] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
[ 0.004862] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
[ 0.004870] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
[ 0.004875] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
[ 0.004881] [<ffffffff813044dc>] ? identify_cpu+0x2f7/0x300
[ 0.004888] [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
[ 0.004895] [<ffffffff810e81d5>] ? kmem_cache_alloc+0x8c/0xf0
[ 0.004900] [<ffffffff81510a16>] ? identify_boot_cpu+0x15/0x3e
[ 0.004904] [<ffffffff81510baa>] ? check_bugs+0x9/0x2e
[ 0.004910] [<ffffffff81509cce>] ? start_kernel+0x3cd/0x3e8
[ 0.004915] [<ffffffff8150bc93>] ? xen_start_kernel+0x586/0x58a
[ 0.004926] ---[ end trace a7919e7f17c0a725 ]---
[ 0.004930] ... version: 0
[ 0.004932] ... bit width: 48
[ 0.004935] ... generic registers: 4
[ 0.004938] ... value mask: 0000ffffffffffff
[ 0.004940] ... max period: 00007fffffffffff
[ 0.004943] ... fixed-purpose events: 0
[ 0.004946] ... event mask: 000000000000000f

Concerning the enlighten.c error. While it does not seem to cause problems it
concerns me power management may be it's the real cause network problems
that are getting my attention.

I got the debian kernel source and line 726 is the line with the *****

static void xen_clts(void)
{
struct multicall_space mcs; *****

mcs = xen_mc_entry(0);

MULTI_fpu_taskswitch(mcs.mc, 0);

xen_mc_issue(PARAVIRT_LAZY_CPU);
}

Means nothing to me :-(

What is it?
What can be done?

Then next one may not be xen but I only had the problem after running a domu.
After a while I get kernel error and networking stops.
This is the error:
[ 1411.813376] ------------[ cut here ]------------
[ 1411.813398] WARNING:
at /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/build/source_amd64_xen/net/sched/s
ch_generic.c:261 dev_watchdog+0xe2/0x194()
[ 1411.813410] Hardware name: H8DGU
[ 1411.813417] NETDEV WATCHDOG: peth0 (igb): transmit queue 1 timed out
[ 1411.813424] Modules linked in: xt_physdev iptable_filter tun ip_tables
x_tables bridge stp sg sr_mod cdrom xfs exportfs ipmi_si i
pmi_devintf ipmi_watchdog ipmi_msghandler xen_evtchn blktap xenfs loop snd_pcm
snd_timer snd soundcore snd_page_alloc pcspkr psmouse joydev evdev serio_raw
i2c_piix
4 edac_core k10temp edac_mce_amd i2c_core processor button acpi_processor ext4
mbcache jbd2 crc16 usbhid hid dm_mod raid1 md_mod sd_mod crc_t10dif
ata_generic usb_s
torage pata_atiixp ahci ohci_hcd libata ehci_hcd usbcore nls_base scsi_mod igb
dca thermal thermal_sys [last unloaded: scsi_wait_scan]
[ 1411.813656] Pid: 4, comm: ksoftirqd/0 Tainted: G W
2.6.32-5-xen-amd64 #1
[ 1411.813664] Call Trace:
[ 1411.813671] <IRQ> [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
[ 1411.813697] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
[ 1411.813711] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
[ 1411.813724] [<ffffffff81272d60>] ? dev_watchdog+0x0/0x194
[ 1411.813736] [<ffffffff8104ef88>] ? warn_slowpath_fmt+0x51/0x59
[ 1411.813751] [<ffffffff8130d42a>] ? _spin_unlock_irqrestore+0xd/0xe
[ 1411.813762] [<ffffffff8104b41e>] ? try_to_wake_up+0x289/0x29b
[ 1411.813778] [<ffffffff81272d34>] ? netif_tx_lock+0x3d/0x69
[ 1411.813791] [<ffffffff8125d7da>] ? netdev_drivername+0x3b/0x40
[ 1411.813803] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
[ 1411.813816] [<ffffffff8100ece2>] ? check_events+0x12/0x20
[ 1411.813827] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
[ 1411.813841] [<ffffffff8105b5ef>] ? run_timer_softirq+0x1c9/0x268
[ 1411.813855] [<ffffffff81054c9b>] ? __do_softirq+0xdd/0x1a6
[ 1411.813867] [<ffffffff81012cac>] ? call_softirq+0x1c/0x30
[ 1411.813873] <EOI> [<ffffffff8101422b>] ? do_softirq+0x3f/0x7c
[ 1411.813893] [<ffffffff810548c2>] ? ksoftirqd+0x5f/0xd3
[ 1411.813905] [<ffffffff81054863>] ? ksoftirqd+0x0/0xd3
[ 1411.813915] [<ffffffff81065c39>] ? kthread+0x79/0x81
[ 1411.813926] [<ffffffff81012baa>] ? child_rip+0xa/0x20
[ 1411.813937] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
[ 1411.813948] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
[ 1411.813958] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
[ 1411.813966] ---[ end trace a7919e7f17c0a727 ]---
[ 1412.052253] eth0: port 1(peth0) entering disabled state
[ 1635.796207] frontend_changed: backend/vbd/3/768: prepare for reconnect
[ 1647.137513] eth0: port 3(vif3.0) entering disabled state
[ 1647.157527] eth0: port 3(vif3.0) entering disabled state
Kernel logging (proc) stopped.

In this case dom0 locked up. Some times just networking stops and some times
networking recovers.

Looks like it uses msi-x interrupts.

Concerning igb error I have tried the following one at a time:
New igb driver from Intel site.
kernel parameter pcie_aspm=off
ethtool -K eth0 tx off on dom0
ethtool -K eth0 gro off on dom0

It has never died doing iperf from dom0 or domu <> external.
Never died during network backup.

Usually takes a least a few hours and has never made it a day running a domu.
Wish I could get it to die faster :-)
Any ideas?
I'm pretty much down to trying different network cards

Any ideas?

John

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network dies and kernel errors [ In reply to ]

konrad.wilk at oracle

Jul 29, 2011, 8:03 AM

Post #2 of 12 (624 views)

Permalink

On Mon, Jul 25, 2011 at 02:18:21PM -0500, John McMonagle wrote:
> Have a new amd 6100 based server.
> http://www.supermicro.com/Aplus/system/2U/2022/AS-2022G-URF.cfm
> Running debian squeeze with debian 2.6.32 xen kernel
> Running xen 4.1.1 built from source from xen.org
>
> I'm seeing 2 errors.
> during boot get this:
>
> [ 0.004823] ------------[ cut here ]------------
> [ 0.004833] WARNING:
> at /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/build/source_amd64_xen/arch/x86/xen/enlighten.c:726
> init_hw_perf_events+0x32d/0x3cd()
> [ 0.004838] Hardware name: H8DGU
> [ 0.004841] Modules linked in:
> [ 0.004847] Pid: 0, comm: swapper Not tainted 2.6.32-5-xen-amd64 #1
> [ 0.004850] Call Trace:
> [ 0.004857] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> [ 0.004862] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> [ 0.004870] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> [ 0.004875] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> [ 0.004881] [<ffffffff813044dc>] ? identify_cpu+0x2f7/0x300
> [ 0.004888] [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
> [ 0.004895] [<ffffffff810e81d5>] ? kmem_cache_alloc+0x8c/0xf0
> [ 0.004900] [<ffffffff81510a16>] ? identify_boot_cpu+0x15/0x3e
> [ 0.004904] [<ffffffff81510baa>] ? check_bugs+0x9/0x2e
> [ 0.004910] [<ffffffff81509cce>] ? start_kernel+0x3cd/0x3e8
> [ 0.004915] [<ffffffff8150bc93>] ? xen_start_kernel+0x586/0x58a

You can ignore that one. It just means that you can't do profiling which we haven't
yet up-ported.

..
> Then next one may not be xen but I only had the problem after running a domu.
> After a while I get kernel error and networking stops.

And some other user with a bnx2 driver seems to see a similar problem. Let me CC them here.

> This is the error:
> [ 1411.813376] ------------[ cut here ]------------
> [ 1411.813398] WARNING:
> at /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/build/source_amd64_xen/net/sched/s
> ch_generic.c:261 dev_watchdog+0xe2/0x194()

OK, this is one is more worrysome.

> [ 1411.813410] Hardware name: H8DGU
> [ 1411.813417] NETDEV WATCHDOG: peth0 (igb): transmit queue 1 timed out
> [ 1411.813424] Modules linked in: xt_physdev iptable_filter tun ip_tables
> x_tables bridge stp sg sr_mod cdrom xfs exportfs ipmi_si i
> pmi_devintf ipmi_watchdog ipmi_msghandler xen_evtchn blktap xenfs loop snd_pcm
> snd_timer snd soundcore snd_page_alloc pcspkr psmouse joydev evdev serio_raw
> i2c_piix
> 4 edac_core k10temp edac_mce_amd i2c_core processor button acpi_processor ext4
> mbcache jbd2 crc16 usbhid hid dm_mod raid1 md_mod sd_mod crc_t10dif
> ata_generic usb_s
> torage pata_atiixp ahci ohci_hcd libata ehci_hcd usbcore nls_base scsi_mod igb
> dca thermal thermal_sys [last unloaded: scsi_wait_scan]
> [ 1411.813656] Pid: 4, comm: ksoftirqd/0 Tainted: G W
> 2.6.32-5-xen-amd64 #1
> [ 1411.813664] Call Trace:
> [ 1411.813671] <IRQ> [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> [ 1411.813697] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> [ 1411.813711] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> [ 1411.813724] [<ffffffff81272d60>] ? dev_watchdog+0x0/0x194
> [ 1411.813736] [<ffffffff8104ef88>] ? warn_slowpath_fmt+0x51/0x59
> [ 1411.813751] [<ffffffff8130d42a>] ? _spin_unlock_irqrestore+0xd/0xe
> [ 1411.813762] [<ffffffff8104b41e>] ? try_to_wake_up+0x289/0x29b
> [ 1411.813778] [<ffffffff81272d34>] ? netif_tx_lock+0x3d/0x69
> [ 1411.813791] [<ffffffff8125d7da>] ? netdev_drivername+0x3b/0x40
> [ 1411.813803] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> [ 1411.813816] [<ffffffff8100ece2>] ? check_events+0x12/0x20
> [ 1411.813827] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
> [ 1411.813841] [<ffffffff8105b5ef>] ? run_timer_softirq+0x1c9/0x268
> [ 1411.813855] [<ffffffff81054c9b>] ? __do_softirq+0xdd/0x1a6
> [ 1411.813867] [<ffffffff81012cac>] ? call_softirq+0x1c/0x30
> [ 1411.813873] <EOI> [<ffffffff8101422b>] ? do_softirq+0x3f/0x7c
> [ 1411.813893] [<ffffffff810548c2>] ? ksoftirqd+0x5f/0xd3
> [ 1411.813905] [<ffffffff81054863>] ? ksoftirqd+0x0/0xd3
> [ 1411.813915] [<ffffffff81065c39>] ? kthread+0x79/0x81
> [ 1411.813926] [<ffffffff81012baa>] ? child_rip+0xa/0x20
> [ 1411.813937] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
> [ 1411.813948] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
> [ 1411.813958] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
> [ 1411.813966] ---[ end trace a7919e7f17c0a727 ]---
> [ 1412.052253] eth0: port 1(peth0) entering disabled state
> [ 1635.796207] frontend_changed: backend/vbd/3/768: prepare for reconnect
> [ 1647.137513] eth0: port 3(vif3.0) entering disabled state
> [ 1647.157527] eth0: port 3(vif3.0) entering disabled state
> Kernel logging (proc) stopped.
>
> In this case dom0 locked up. Some times just networking stops and some times
> networking recovers.
>
> Looks like it uses msi-x interrupts.
>
> Concerning igb error I have tried the following one at a time:
> New igb driver from Intel site.
> kernel parameter pcie_aspm=off
> ethtool -K eth0 tx off on dom0
> ethtool -K eth0 gro off on dom0
>
OK.
> It has never died doing iperf from dom0 or domu <> external.
> Never died during network backup.
>
> Usually takes a least a few hours and has never made it a day running a domu.
> Wish I could get it to die faster :-)
> Any ideas?
> I'm pretty much down to trying different network cards

Did you try that? Did that make any difference?
>
> Any ideas?

There is a Xen parameter called 'noirqbalance' . Try that. Also see if you can
limit the CPUs in the dom0 using these two arguments on Xen hypervisor:

dom0_vcpus=2 dom0_vcpus_pin=1

It would be interesting to narrow down _when_ you trigger this failure. B/c we
can pull Xen to see what the MSI's are 'xl debug-keys M' _before_ and _after_ your
failure to see if something is amiss.

Mainly to figure out if the vectors are moving around the CPUs (or not)

(XEN) MSI 29 vec=21 lowest edge assert log lowest dest=00000001 mask=0/0/-1

and also 'xl debug-keys i' to see if the domain has ACK-ed the interrupt:
(XEN) IRQ: 29 affinity:00000000,00000000,00000000,00000001 vec:21 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(----),

(the last '----' might have something else in in them - if so that is a sign that
dom0 hasn't picked up the event/vector).

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network dies and kernel errors [ In reply to ]

johnm at advocap

Jul 29, 2011, 8:38 AM

Post #3 of 12 (626 views)

Permalink

On Friday, July 29, 2011 10:03:48 am Konrad Rzeszutek Wilk wrote:
> On Mon, Jul 25, 2011 at 02:18:21PM -0500, John McMonagle wrote:
> > Have a new amd 6100 based server.
> > http://www.supermicro.com/Aplus/system/2U/2022/AS-2022G-URF.cfm
> > Running debian squeeze with debian 2.6.32 xen kernel
> > Running xen 4.1.1 built from source from xen.org
> >
> > I'm seeing 2 errors.
> > during boot get this:
> >
> > [ 0.004823] ------------[ cut here ]------------
> > [ 0.004833] WARNING:
> > at
> > /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/b
> > uild/source_amd64_xen/arch/x86/xen/enlighten.c:726
> > init_hw_perf_events+0x32d/0x3cd()
> > [ 0.004838] Hardware name: H8DGU
> > [ 0.004841] Modules linked in:
> > [ 0.004847] Pid: 0, comm: swapper Not tainted 2.6.32-5-xen-amd64 #1
> > [ 0.004850] Call Trace:
> > [ 0.004857] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> > [ 0.004862] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> > [ 0.004870] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> > [ 0.004875] [<ffffffff81510efc>] ? init_hw_perf_events+0x32d/0x3cd
> > [ 0.004881] [<ffffffff813044dc>] ? identify_cpu+0x2f7/0x300
> > [ 0.004888] [<ffffffff8100eccf>] ? xen_restore_fl_direct_end+0x0/0x1
> > [ 0.004895] [<ffffffff810e81d5>] ? kmem_cache_alloc+0x8c/0xf0
> > [ 0.004900] [<ffffffff81510a16>] ? identify_boot_cpu+0x15/0x3e
> > [ 0.004904] [<ffffffff81510baa>] ? check_bugs+0x9/0x2e
> > [ 0.004910] [<ffffffff81509cce>] ? start_kernel+0x3cd/0x3e8
> > [ 0.004915] [<ffffffff8150bc93>] ? xen_start_kernel+0x586/0x58a
>
> You can ignore that one. It just means that you can't do profiling which we
> haven't yet up-ported.
>
> ..
>
> > Then next one may not be xen but I only had the problem after running a
> > domu. After a while I get kernel error and networking stops.
>
> And some other user with a bnx2 driver seems to see a similar problem. Let
> me CC them here.
>
> > This is the error:
> > [ 1411.813376] ------------[ cut here ]------------
> > [ 1411.813398] WARNING:
> > at
> > /build/buildd-linux-2.6_2.6.32-35-amd64-aZSlKL/linux-2.6-2.6.32/debian/b
> > uild/source_amd64_xen/net/sched/s ch_generic.c:261
> > dev_watchdog+0xe2/0x194()
>
> OK, this is one is more worrysome.
>
> > [ 1411.813410] Hardware name: H8DGU
> > [ 1411.813417] NETDEV WATCHDOG: peth0 (igb): transmit queue 1 timed out
> > [ 1411.813424] Modules linked in: xt_physdev iptable_filter tun ip_tables
> > x_tables bridge stp sg sr_mod cdrom xfs exportfs ipmi_si i
> > pmi_devintf ipmi_watchdog ipmi_msghandler xen_evtchn blktap xenfs loop
> > snd_pcm snd_timer snd soundcore snd_page_alloc pcspkr psmouse joydev
> > evdev serio_raw i2c_piix
> > 4 edac_core k10temp edac_mce_amd i2c_core processor button acpi_processor
> > ext4 mbcache jbd2 crc16 usbhid hid dm_mod raid1 md_mod sd_mod crc_t10dif
> > ata_generic usb_s
> > torage pata_atiixp ahci ohci_hcd libata ehci_hcd usbcore nls_base
> > scsi_mod igb dca thermal thermal_sys [last unloaded: scsi_wait_scan]
> >
> > [ 1411.813656] Pid: 4, comm: ksoftirqd/0 Tainted: G W
> >
> > 2.6.32-5-xen-amd64 #1
> > [ 1411.813664] Call Trace:
> > [ 1411.813671] <IRQ> [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> > [ 1411.813697] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> > [ 1411.813711] [<ffffffff8104ef00>] ? warn_slowpath_common+0x77/0xa3
> > [ 1411.813724] [<ffffffff81272d60>] ? dev_watchdog+0x0/0x194
> > [ 1411.813736] [<ffffffff8104ef88>] ? warn_slowpath_fmt+0x51/0x59
> > [ 1411.813751] [<ffffffff8130d42a>] ? _spin_unlock_irqrestore+0xd/0xe
> > [ 1411.813762] [<ffffffff8104b41e>] ? try_to_wake_up+0x289/0x29b
> > [ 1411.813778] [<ffffffff81272d34>] ? netif_tx_lock+0x3d/0x69
> > [ 1411.813791] [<ffffffff8125d7da>] ? netdev_drivername+0x3b/0x40
> > [ 1411.813803] [<ffffffff81272e42>] ? dev_watchdog+0xe2/0x194
> > [ 1411.813816] [<ffffffff8100ece2>] ? check_events+0x12/0x20
> > [ 1411.813827] [<ffffffff81040e42>] ? check_preempt_wakeup+0x0/0x268
> > [ 1411.813841] [<ffffffff8105b5ef>] ? run_timer_softirq+0x1c9/0x268
> > [ 1411.813855] [<ffffffff81054c9b>] ? __do_softirq+0xdd/0x1a6
> > [ 1411.813867] [<ffffffff81012cac>] ? call_softirq+0x1c/0x30
> > [ 1411.813873] <EOI> [<ffffffff8101422b>] ? do_softirq+0x3f/0x7c
> > [ 1411.813893] [<ffffffff810548c2>] ? ksoftirqd+0x5f/0xd3
> > [ 1411.813905] [<ffffffff81054863>] ? ksoftirqd+0x0/0xd3
> > [ 1411.813915] [<ffffffff81065c39>] ? kthread+0x79/0x81
> > [ 1411.813926] [<ffffffff81012baa>] ? child_rip+0xa/0x20
> > [ 1411.813937] [<ffffffff81011d61>] ? int_ret_from_sys_call+0x7/0x1b
> > [ 1411.813948] [<ffffffff8101251d>] ? retint_restore_args+0x5/0x6
> > [ 1411.813958] [<ffffffff81012ba0>] ? child_rip+0x0/0x20
> > [ 1411.813966] ---[ end trace a7919e7f17c0a727 ]---
> > [ 1412.052253] eth0: port 1(peth0) entering disabled state
> > [ 1635.796207] frontend_changed: backend/vbd/3/768: prepare for reconnect
> > [ 1647.137513] eth0: port 3(vif3.0) entering disabled state
> > [ 1647.157527] eth0: port 3(vif3.0) entering disabled state
> >
> > Kernel logging (proc) stopped.
> >
> > In this case dom0 locked up. Some times just networking stops and some
> > times networking recovers.
> >
> > Looks like it uses msi-x interrupts.
> >
> > Concerning igb error I have tried the following one at a time:
> > New igb driver from Intel site.
> > kernel parameter pcie_aspm=off
> > ethtool -K eth0 tx off on dom0
> > ethtool -K eth0 gro off on dom0
>
> OK.
>
> > It has never died doing iperf from dom0 or domu <> external.
> > Never died during network backup.
> >
> > Usually takes a least a few hours and has never made it a day running a
> > domu. Wish I could get it to die faster :-)
> > Any ideas?
> > I'm pretty much down to trying different network cards
>
> Did you try that? Did that make any difference?

Not tested I did install one.

I think I found a way to keep it running.
On the new igb driver I built from new intel source added module parameter
IntMode=1.

This puts it in msi mode. It was in msi-x mode.
It's never died with that setting.
It's up now over a day.
No real experience with msi-x. I think it's the first time I have seen a
driver use msi-x interrupts.
Maybe that gives you more ideas?

>
> > Any ideas?
>
> There is a Xen parameter called 'noirqbalance' . Try that. Also see if you
> can limit the CPUs in the dom0 using these two arguments on Xen
> hypervisor:
>
Should I turn off the irqbalence daemon also?
Just in case you wonder it does with out it.

> dom0_vcpus=2 dom0_vcpus_pin=1
>
>
> It would be interesting to narrow down _when_ you trigger this failure. B/c
> we can pull Xen to see what the MSI's are 'xl debug-keys M' _before_ and
> _after_ your failure to see if something is amiss.
>
> Mainly to figure out if the vectors are moving around the CPUs (or not)
>
> (XEN) MSI 29 vec=21 lowest edge assert log lowest dest=00000001
> mask=0/0/-1
>
> and also 'xl debug-keys i' to see if the domain has ACK-ed the interrupt:
> (XEN) IRQ: 29 affinity:00000000,00000000,00000000,00000001 vec:21
> type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(----),
>
> (the last '----' might have something else in in them - if so that is a
> sign that dom0 hasn't picked up the event/vector).

Much of my frustration is that I have not found a way to get it to fail other
than waiting a long time :-(

Thanks

John

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network dies and kernel errors [ In reply to ]

konrad.wilk at oracle

Jul 29, 2011, 10:31 AM

Post #4 of 12 (624 views)

Permalink

> > Did you try that? Did that make any difference?
>
> Not tested I did install one.
>
> I think I found a way to keep it running.
> On the new igb driver I built from new intel source added module parameter
> IntMode=1.
>
> This puts it in msi mode. It was in msi-x mode.
> It's never died with that setting.
> It's up now over a day.
> No real experience with msi-x. I think it's the first time I have seen a
> driver use msi-x interrupts.
> Maybe that gives you more ideas?

That was my thought - the MSI-X aren't somehow being ACKed properly. But I don't
know if the issue with Dom0 or Xen.
>
>
> >
> > > Any ideas?
> >
> > There is a Xen parameter called 'noirqbalance' . Try that. Also see if you
> > can limit the CPUs in the dom0 using these two arguments on Xen
> > hypervisor:
> >
> Should I turn off the irqbalence daemon also?

Sure.

> Just in case you wonder it does with out it.
>
> > dom0_vcpus=2 dom0_vcpus_pin=1
> >
> >
> > It would be interesting to narrow down _when_ you trigger this failure. B/c
> > we can pull Xen to see what the MSI's are 'xl debug-keys M' _before_ and
> > _after_ your failure to see if something is amiss.
> >
> > Mainly to figure out if the vectors are moving around the CPUs (or not)
> >
> > (XEN) MSI 29 vec=21 lowest edge assert log lowest dest=00000001
> > mask=0/0/-1
> >
> > and also 'xl debug-keys i' to see if the domain has ACK-ed the interrupt:
> > (XEN) IRQ: 29 affinity:00000000,00000000,00000000,00000001 vec:21
> > type=PCI-MSI status=00000010 in-flight=0 domain-list=0:275(----),
> >
> > (the last '----' might have something else in in them - if so that is a
> > sign that dom0 hasn't picked up the event/vector).
>
> Much of my frustration is that I have not found a way to get it to fail other
> than waiting a long time :-(

Ah that sucks. Well, just make a nice shell script that will run those continously
(and also 'xl dmesg') and pipe the log to a file.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network dies and kernel errors [ In reply to ]

johnm at advocap

Jul 29, 2011, 2:12 PM

Post #5 of 12 (620 views)

Permalink

On Friday, July 29, 2011 12:31:29 pm you wrote:
> > > Did you try that? Did that make any difference?
> >
> > Not tested I did install one.
> >
> > I think I found a way to keep it running.
> > On the new igb driver I built from new intel source added module
> > parameter IntMode=1.
> >
> > This puts it in msi mode. It was in msi-x mode.
> > It's never died with that setting.
> > It's up now over a day.
> > No real experience with msi-x. I think it's the first time I have seen a
> > driver use msi-x interrupts.
> > Maybe that gives you more ideas?
>
> That was my thought - the MSI-X aren't somehow being ACKed properly. But I
> don't know if the issue with Dom0 or Xen.
>
> > > > Any ideas?
> > >
> > > There is a Xen parameter called 'noirqbalance' . Try that. Also see if
> > > you can limit the CPUs in the dom0 using these two arguments on Xen
> >
> > > hypervisor:
> > Should I turn off the irqbalence daemon also?
>
> Sure.
>
> > Just in case you wonder it does with out it.
> >
> > > dom0_vcpus=2 dom0_vcpus_pin=1
> > >
> > >
> > > It would be interesting to narrow down _when_ you trigger this failure.
> > > B/c we can pull Xen to see what the MSI's are 'xl debug-keys M'
> > > _before_ and _after_ your failure to see if something is amiss.
> > >
> > > Mainly to figure out if the vectors are moving around the CPUs (or not)
> > >
> > > (XEN) MSI 29 vec=21 lowest edge assert log lowest dest=00000001
> > > mask=0/0/-1
> > >
> > > and also 'xl debug-keys i' to see if the domain has ACK-ed the
> > > interrupt: (XEN) IRQ: 29
> > > affinity:00000000,00000000,00000000,00000001 vec:21 type=PCI-MSI
> > > status=00000010 in-flight=0 domain-list=0:275(----),
> > >
> > > (the last '----' might have something else in in them - if so that is a
> > > sign that dom0 hasn't picked up the event/vector).
> >
> > Much of my frustration is that I have not found a way to get it to fail
> > other than waiting a long time :-(
>
> Ah that sucks. Well, just make a nice shell script that will run those
> continously (and also 'xl dmesg') and pipe the log to a file.
I was just setting up to run your test.
About 10 minutes after removing irqbalence I lost networking.
I'm remote and few minutes later lost ipmi sol so odds I lost the serial port
interrupt also. So went to ikvm.

Was able to able to restore network with
ifdown xenbr0
rmmod igb
modprobe igb
ifup xenbr0

The runnning domu still had no networking.

Attached is
dom0.dmesg.gz dom0 dmesg I see nothing in particular myself.
xen.dmesg.gz xl dmesg I had done the M and i earlier and ran them just
before running xl dmesg so it should have before and after.
Did these before bringing the network back up.

I'll reboot and run as you requested.

John

Re: Network dies and kernel errors [ In reply to ]

johnm at advocap

Jul 31, 2011, 4:21 PM

Post #6 of 12 (620 views)

Permalink

Konrad

I ran as you requested and after 2 days it's still up.

Attached are xen.dmesg and dom0.dmesg all the command line parameters are in
them.

At this point I'd be happy if was on line with a reasonably efficient stable
configuration.

Supermicro has been helpful on other issues but not with this.
Suspect I'll need to have some idea what the problem is to be able to get it
sent to the right person.

Any suggestions?

John

On Friday, July 29, 2011 12:31:29 pm Konrad Rzeszutek Wilk wrote:
> > > Did you try that? Did that make any difference?
> >
> > Not tested I did install one.
> >
> > I think I found a way to keep it running.
> > On the new igb driver I built from new intel source added module
> > parameter IntMode=1.
> >
> > This puts it in msi mode. It was in msi-x mode.
> > It's never died with that setting.
> > It's up now over a day.
> > No real experience with msi-x. I think it's the first time I have seen a
> > driver use msi-x interrupts.
> > Maybe that gives you more ideas?
>
> That was my thought - the MSI-X aren't somehow being ACKed properly. But I
> don't know if the issue with Dom0 or Xen.
>
> > > > Any ideas?
> > >
> > > There is a Xen parameter called 'noirqbalance' . Try that. Also see if
> > > you can limit the CPUs in the dom0 using these two arguments on Xen
> >
> > > hypervisor:
> > Should I turn off the irqbalence daemon also?
>
> Sure.
>
> > Just in case you wonder it does with out it.
> >
> > > dom0_vcpus=2 dom0_vcpus_pin=1
> > >
> > >
> > > It would be interesting to narrow down _when_ you trigger this failure.
> > > B/c we can pull Xen to see what the MSI's are 'xl debug-keys M'
> > > _before_ and _after_ your failure to see if something is amiss.
> > >
> > > Mainly to figure out if the vectors are moving around the CPUs (or not)
> > >
> > > (XEN) MSI 29 vec=21 lowest edge assert log lowest dest=00000001
> > > mask=0/0/-1
> > >
> > > and also 'xl debug-keys i' to see if the domain has ACK-ed the
> > > interrupt: (XEN) IRQ: 29
> > > affinity:00000000,00000000,00000000,00000001 vec:21 type=PCI-MSI
> > > status=00000010 in-flight=0 domain-list=0:275(----),
> > >
> > > (the last '----' might have something else in in them - if so that is a
> > > sign that dom0 hasn't picked up the event/vector).
> >
> > Much of my frustration is that I have not found a way to get it to fail
> > other than waiting a long time :-(
>
> Ah that sucks. Well, just make a nice shell script that will run those
> continously (and also 'xl dmesg') and pipe the log to a file.

Re: Network dies and kernel errors [ In reply to ]

konrad.wilk at oracle

Aug 2, 2011, 9:17 AM

Post #7 of 12 (613 views)

Permalink

On Sun, Jul 31, 2011 at 06:21:47PM -0500, John McMonagle wrote:
> Konrad
>
> I ran as you requested and after 2 days it's still up.

Good!
>
> Attached are xen.dmesg and dom0.dmesg all the command line parameters are in
> them.

> At this point I'd be happy if was on line with a reasonably efficient stable
> configuration.

So it looks like from your runs (before you had irqbalance):

(XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
(XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
(XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
(XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
(XEN) IRQ: 31 affinity:00000000,00000000,00000000,00004000 vec:b5 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:273(----),
(XEN) MSI 31 vec=39 fixed edge assert phys cpu dest=00000020 mask=1/0/0
(XEN) IRQ: 31 affinity:00000000,00000000,00000000,00000100 vec:39 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(----),

The affinity (and the corresponding vector) moves from one CPU to another
and then dom0 somehow does not get it. Do you have another of these
boxes available remotly to debug this further?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Network dies and kernel errors [ In reply to ]

johnm at advocap

Aug 2, 2011, 11:07 AM

Post #8 of 12 (610 views)

Permalink

On Tuesday, August 02, 2011 11:17:36 am Konrad Rzeszutek Wilk wrote:
> On Sun, Jul 31, 2011 at 06:21:47PM -0500, John McMonagle wrote:
> > Konrad
> >
> > I ran as you requested and after 2 days it's still up.
>
> Good!
>
> > Attached are xen.dmesg and dom0.dmesg all the command line parameters
> > are in them.
> >
> > At this point I'd be happy if was on line with a reasonably efficient
> > stable configuration.
>
> So it looks like from your runs (before you had irqbalance):
>
> (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023
> mask=1/0/0 (XEN) MSI 31 vec=ad fixed edge assert phys cpu
> dest=00000023 mask=1/0/0 (XEN) MSI 31 vec=ad fixed edge assert
> phys cpu dest=00000023 mask=1/0/0 (XEN) MSI 31 vec=ad fixed edge
> assert phys cpu dest=00000023 mask=1/0/0 (XEN) IRQ: 31
> affinity:00000000,00000000,00000000,00004000 vec:b5 type=PCI-MSI
> status=00000010 in-flight=0 domain-list=0:273(----), (XEN) MSI 31
> vec=39 fixed edge assert phys cpu dest=00000020 mask=1/0/0 (XEN)
> IRQ: 31 affinity:00000000,00000000,00000000,00000100 vec:39 type=PCI-MSI
> status=00000050 in-flight=0 domain-list=0:273(----),
>
> The affinity (and the corresponding vector) moves from one CPU to another
> and then dom0 somehow does not get it. Do you have another of these
> boxes available remotly to debug this further?

Konrad

Just one box but I'm willing to try more things.
A shame this is otherwise a really nice box for the price.
I sent Supermicro another email with updated info.
I'm not holding my breath :-(
They did fix the ipmi firmware for me so I guess I can hope for a fix.

In my last test shouldn't dom0_vcpus=2 have been dom0_max_vcpus=2?
Dom0 still has 16 cpus.

I'd prefer not pining dom0 but stability is more important :-)

Thanks for all the help so far.

John

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Network dies and kernel errors [ In reply to ]

tinnycloud at hotmail

Aug 3, 2011, 2:01 AM

Post #9 of 12 (610 views)

Permalink

in my test environment, the irqbalance is already stopped.

So my bnx bug may not related to this.

Many thanks.

> Date: Tue, 2 Aug 2011 12:17:36 -0400
> From: konrad.wilk@oracle.com
> To: johnm@advocap.org
> CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Network dies and kernel errors
>
> On Sun, Jul 31, 2011 at 06:21:47PM -0500, John McMonagle wrote:
> > Konrad
> >
> > I ran as you requested and after 2 days it's still up.
>
> Good!
> >
> > Attached are xen.dmesg and dom0.dmesg all the command line parameters are in
> > them.
>
> > At this point I'd be happy if was on line with a reasonably efficient stable
> > configuration.
>
> So it looks like from your runs (before you had irqbalance):
>
> (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00004000 vec:b5 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:273(----),
> (XEN) MSI 31 vec=39 fixed edge assert phys cpu dest=00000020 mask=1/0/0
> (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00000100 vec:39 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(----),
>
> The affinity (and the corresponding vector) moves from one CPU to another
> and then dom0 somehow does not get it. Do you have another of these
> boxes available remotly to debug this further?
>

Re: Network dies and kernel errors [ In reply to ]

konrad.wilk at oracle

Aug 3, 2011, 6:37 AM

Post #10 of 12 (605 views)

Permalink

On Wed, Aug 03, 2011 at 05:01:38PM +0800, MaoXiaoyun wrote:
>
> in my test environment, the irqbalance is already stopped.
>
> So my bnx bug may not related to this.

Did you use 'noirqbalance' on the Xen hypervisor line?
>
> Many thanks.
>
>
> > Date: Tue, 2 Aug 2011 12:17:36 -0400
> > From: konrad.wilk@oracle.com
> > To: johnm@advocap.org
> > CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > Subject: Re: [Xen-devel] Network dies and kernel errors
> >
> > On Sun, Jul 31, 2011 at 06:21:47PM -0500, John McMonagle wrote:
> > > Konrad
> > >
> > > I ran as you requested and after 2 days it's still up.
> >
> > Good!
> > >
> > > Attached are xen.dmesg and dom0.dmesg all the command line parameters are in
> > > them.
> >
> > > At this point I'd be happy if was on line with a reasonably efficient stable
> > > configuration.
> >
> > So it looks like from your runs (before you had irqbalance):
> >
> > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00004000 vec:b5 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:273(----),
> > (XEN) MSI 31 vec=39 fixed edge assert phys cpu dest=00000020 mask=1/0/0
> > (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00000100 vec:39 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(----),
> >
> > The affinity (and the corresponding vector) moves from one CPU to another
> > and then dom0 somehow does not get it. Do you have another of these
> > boxes available remotly to debug this further?
> >
>

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: Network dies and kernel errors [ In reply to ]

tinnycloud at hotmail

Aug 3, 2011, 8:55 PM

Post #11 of 12 (601 views)

Permalink

> Date: Wed, 3 Aug 2011 09:37:57 -0400
> From: konrad.wilk@oracle.com
> To: tinnycloud@hotmail.com
> CC: johnm@advocap.org; xen-devel@lists.xensource.com
> Subject: Re: [Xen-devel] Network dies and kernel errors
>
> On Wed, Aug 03, 2011 at 05:01:38PM +0800, MaoXiaoyun wrote:
> >
> > in my test environment, the irqbalance is already stopped.
> >
> > So my bnx bug may not related to this.
>
>
> Did you use 'noirqbalance' on the Xen hypervisor line?

No. I just stop irqbalance service, any difference with "noirqbalance"?
If so, I would like to have a try.

Thanks.

> >
> > Many thanks.
> >
> >
> > > Date: Tue, 2 Aug 2011 12:17:36 -0400
> > > From: konrad.wilk@oracle.com
> > > To: johnm@advocap.org
> > > CC: tinnycloud@hotmail.com; xen-devel@lists.xensource.com
> > > Subject: Re: [Xen-devel] Network dies and kernel errors
> > >
> > > On Sun, Jul 31, 2011 at 06:21:47PM -0500, John McMonagle wrote:
> > > > Konrad
> > > >
> > > > I ran as you requested and after 2 days it's still up.
> > >
> > > Good!
> > > >
> > > > Attached are xen.dmesg and dom0.dmesg all the command line parameters are in
> > > > them.
> > >
> > > > At this point I'd be happy if was on line with a reasonably efficient stable
> > > > configuration.
> > >
> > > So it looks like from your runs (before you had irqbalance):
> > >
> > > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > > (XEN) MSI 31 vec=ad fixed edge assert phys cpu dest=00000023 mask=1/0/0
> > > (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00004000 vec:b5 type=PCI-MSI status=00000010 in-flight=0 domain-list=0:273(----),
> > > (XEN) MSI 31 vec=39 fixed edge assert phys cpu dest=00000020 mask=1/0/0
> > > (XEN) IRQ: 31 affinity:00000000,00000000,00000000,00000100 vec:39 type=PCI-MSI status=00000050 in-flight=0 domain-list=0:273(----),
> > >
> > > The affinity (and the corresponding vector) moves from one CPU to another
> > > and then dom0 somehow does not get it. Do you have another of these
> > > boxes available remotly to debug this further?
> > >
> >
>
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.xensource.com
> > http://lists.xensource.com/xen-devel
>

Re: Network dies and kernel errors [ In reply to ]

konrad.wilk at oracle

Aug 4, 2011, 11:03 AM

Post #12 of 12 (595 views)

Permalink

> > > in my test environment, the irqbalance is already stopped.
> > >
> > > So my bnx bug may not related to this.
> >
> >
> > Did you use 'noirqbalance' on the Xen hypervisor line?
>
> No. I just stop irqbalance service, any difference with "noirqbalance"?
> If so, I would like to have a try.

No that is OK.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Mailing List Archive

Attached Files:

Attached Files: