http://bugzilla.xensource.com/bugzilla/show_bug.cgi?id=1746
Summary: Dom0 Locked up for 4 hours "BUG: soft lockup - CPU#3
stuck for 61s!"
Product: Xen
Version: 1.0
Platform: x86-64
OS/Version: All
Status: NEW
Severity: critical
Priority: P2
Component: Cloud Xen
AssignedTo: xen-bugs@lists.xensource.com
ReportedBy: jfrias@gmail.com
We have a deployment of about 10 xcp 1.0 beta xen servers, and just had one
server had a very odd issue. Dom0 became unresponsive ( although xenapi
somewhat worked for querying ) for approximately 4 hours. It then recovered
itself.
We had done no changes on Dom0 and only had changed one of the Domu's to turn
off irqbalance per discussion here
http://forums.citrix.com/thread.jspa?threadID=272708&tstart=0
we are running XCP release 1.0.0-38754c (xcp)
Here's some output from dmesg + sar
======= DMESG ==========
BUG: soft lockup - CPU#3 stuck for 61s! [swapper:0]
Modules linked in: nls_utf8 hfsplus bonding tun lockd sunrpc bridge stp llc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp
x_tables binfmt_misc dm_mirror video output sbs sbshc fan container battery ac
parport_pc lp parport nvram joydev sr_mod cdrom evdev usb_storage usb_libusual
usbhid sg thermal button processor thermal_sys bnx2 serio_raw 8250_pnp rtc_cmos
8250 serial_core rtc_core tpm_tis rtc_lib tpm tpm_bios pcspkr dm_region_hash
dm_log dm_mod ide_gd_mod megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd usbcore fbcon font tileblit bitblit softcursor [last
unloaded: ip_tables]
Pid: 0, comm: swapper Not tainted (2.6.32.12-0.7.1.xs1.0.0.298.170582xen #1)
PowerEdge R710
EIP: 0061:[<c01013a7>] EFLAGS: 00000246 CPU: 3
EIP is at 0xc01013a7
EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: ee853f78
ESI: 00117f39 EDI: 00000003 EBP: ee853f90 ESP: ee853f74
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
CR0: 8005003b CR2: b7736000 CR3: 0e713000 CR4: 00002660
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
Call Trace:
[<c0107035>] ? xen_safe_halt+0xb5/0x150
[<c010ac7e>] xen_idle+0x1e/0x50
[<c0102a7b>] cpu_idle+0x3b/0x60
[<c037b00d>] cpu_bringup_and_idle+0xd/0x10
BUG: soft lockup - CPU#3 stuck for 61s! [swapper:0]
Modules linked in: nls_utf8 hfsplus bonding tun lockd sunrpc bridge stp llc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp
x_tables binfmt_misc dm_mirror video output sbs sbshc fan container battery ac
parport_pc lp parport nvram joydev sr_mod cdrom evdev usb_storage usb_libusual
usbhid sg thermal button processor thermal_sys bnx2 serio_raw 8250_pnp rtc_cmos
8250 serial_core rtc_core tpm_tis rtc_lib tpm tpm_bios pcspkr dm_region_hash
dm_log dm_mod ide_gd_mod megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd usbcore fbcon font tileblit bitblit softcursor [last
unloaded: ip_tables]
======= SAR -B ==========
06:30:01 AM pgpgin/s pgpgout/s fault/s majflt/s
07:20:01 AM 106.28 8604.10 19.54 0.00
07:30:01 AM 161.79 8180.32 34.22 0.00
11:33:56 AM 11315.18 1588.36 63.54 0.00
11:40:01 AM 28.68 798.89 694.91 0.03
11:50:01 AM 0.29 804.20 72.90 0.00
06:30:01 AM CPU %user %nice %system %iowait %steal
%idle
07:20:01 AM all 0.18 0.00 2.30 0.15 1.14
96.23
07:30:01 AM all 0.15 0.00 2.53 0.08 1.08
96.16
11:33:56 AM all 0.08 0.00 0.93 0.01 0.78
98.20
11:40:01 AM all 0.96 0.00 0.42 0.05 0.33
98.24
06:30:01 AM tps rtps wtps bread/s bwrtn/s
07:20:01 AM 2828.30 114.11 2714.19 451.26 34424.36
07:30:01 AM 3015.90 98.68 2917.23 670.15 32729.35
11:33:56 AM 497.94 299.28 198.66 45260.70 6356.23
11:40:01 AM 11165955.42 11757009.62 11187596.79 11681878.85 5099265.21
11:50:01 AM 174.25 0.08 174.17 1.15 3218.26
12:00:01 PM 192.56 0.05 192.51 0.19 3386.10
06:30:01 AM proc/s
07:20:01 AM 0.14
07:30:01 AM 0.34
11:33:56 AM 0.70
11:40:01 AM 3.49
11:50:01 AM 0.27
--
Configure bugmail: http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
_______________________________________________
Xen-bugs mailing list
Xen-bugs@lists.xensource.com
http://lists.xensource.com/xen-bugs
Summary: Dom0 Locked up for 4 hours "BUG: soft lockup - CPU#3
stuck for 61s!"
Product: Xen
Version: 1.0
Platform: x86-64
OS/Version: All
Status: NEW
Severity: critical
Priority: P2
Component: Cloud Xen
AssignedTo: xen-bugs@lists.xensource.com
ReportedBy: jfrias@gmail.com
We have a deployment of about 10 xcp 1.0 beta xen servers, and just had one
server had a very odd issue. Dom0 became unresponsive ( although xenapi
somewhat worked for querying ) for approximately 4 hours. It then recovered
itself.
We had done no changes on Dom0 and only had changed one of the Domu's to turn
off irqbalance per discussion here
http://forums.citrix.com/thread.jspa?threadID=272708&tstart=0
we are running XCP release 1.0.0-38754c (xcp)
Here's some output from dmesg + sar
======= DMESG ==========
BUG: soft lockup - CPU#3 stuck for 61s! [swapper:0]
Modules linked in: nls_utf8 hfsplus bonding tun lockd sunrpc bridge stp llc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp
x_tables binfmt_misc dm_mirror video output sbs sbshc fan container battery ac
parport_pc lp parport nvram joydev sr_mod cdrom evdev usb_storage usb_libusual
usbhid sg thermal button processor thermal_sys bnx2 serio_raw 8250_pnp rtc_cmos
8250 serial_core rtc_core tpm_tis rtc_lib tpm tpm_bios pcspkr dm_region_hash
dm_log dm_mod ide_gd_mod megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd usbcore fbcon font tileblit bitblit softcursor [last
unloaded: ip_tables]
Pid: 0, comm: swapper Not tainted (2.6.32.12-0.7.1.xs1.0.0.298.170582xen #1)
PowerEdge R710
EIP: 0061:[<c01013a7>] EFLAGS: 00000246 CPU: 3
EIP is at 0xc01013a7
EAX: 00000000 EBX: 00000001 ECX: 00000000 EDX: ee853f78
ESI: 00117f39 EDI: 00000003 EBP: ee853f90 ESP: ee853f74
DS: 007b ES: 007b FS: 00d8 GS: 0000 SS: 0069
CR0: 8005003b CR2: b7736000 CR3: 0e713000 CR4: 00002660
DR0: 00000000 DR1: 00000000 DR2: 00000000 DR3: 00000000
DR6: ffff0ff0 DR7: 00000400
Call Trace:
[<c0107035>] ? xen_safe_halt+0xb5/0x150
[<c010ac7e>] xen_idle+0x1e/0x50
[<c0102a7b>] cpu_idle+0x3b/0x60
[<c037b00d>] cpu_bringup_and_idle+0xd/0x10
BUG: soft lockup - CPU#3 stuck for 61s! [swapper:0]
Modules linked in: nls_utf8 hfsplus bonding tun lockd sunrpc bridge stp llc
ipt_REJECT nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp
x_tables binfmt_misc dm_mirror video output sbs sbshc fan container battery ac
parport_pc lp parport nvram joydev sr_mod cdrom evdev usb_storage usb_libusual
usbhid sg thermal button processor thermal_sys bnx2 serio_raw 8250_pnp rtc_cmos
8250 serial_core rtc_core tpm_tis rtc_lib tpm tpm_bios pcspkr dm_region_hash
dm_log dm_mod ide_gd_mod megaraid_sas sd_mod scsi_mod ext3 jbd uhci_hcd
ohci_hcd ehci_hcd usbcore fbcon font tileblit bitblit softcursor [last
unloaded: ip_tables]
======= SAR -B ==========
06:30:01 AM pgpgin/s pgpgout/s fault/s majflt/s
07:20:01 AM 106.28 8604.10 19.54 0.00
07:30:01 AM 161.79 8180.32 34.22 0.00
11:33:56 AM 11315.18 1588.36 63.54 0.00
11:40:01 AM 28.68 798.89 694.91 0.03
11:50:01 AM 0.29 804.20 72.90 0.00
06:30:01 AM CPU %user %nice %system %iowait %steal
%idle
07:20:01 AM all 0.18 0.00 2.30 0.15 1.14
96.23
07:30:01 AM all 0.15 0.00 2.53 0.08 1.08
96.16
11:33:56 AM all 0.08 0.00 0.93 0.01 0.78
98.20
11:40:01 AM all 0.96 0.00 0.42 0.05 0.33
98.24
06:30:01 AM tps rtps wtps bread/s bwrtn/s
07:20:01 AM 2828.30 114.11 2714.19 451.26 34424.36
07:30:01 AM 3015.90 98.68 2917.23 670.15 32729.35
11:33:56 AM 497.94 299.28 198.66 45260.70 6356.23
11:40:01 AM 11165955.42 11757009.62 11187596.79 11681878.85 5099265.21
11:50:01 AM 174.25 0.08 174.17 1.15 3218.26
12:00:01 PM 192.56 0.05 192.51 0.19 3386.10
06:30:01 AM proc/s
07:20:01 AM 0.14
07:30:01 AM 0.34
11:33:56 AM 0.70
11:40:01 AM 3.49
11:50:01 AM 0.27
--
Configure bugmail: http://bugzilla.xensource.com/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.
_______________________________________________
Xen-bugs mailing list
Xen-bugs@lists.xensource.com
http://lists.xensource.com/xen-bugs