Mailing List Archive: pv-on-hvm kexec, howto reregister timer interrupt

pv-on-hvm kexec, howto reregister timer interrupt

Jul 22, 2011, 11:49 AM

Post #1 of 5 (580 views)

What is the best way to reregister the timer interrupt in the kexec
kernel in a pv-on-hvm guest? Right now the crash shown below happens
because EVTCHNOP_bind_virq returns -EEXIST. Can the timer on the cpus be
unregistered during shutdown? xen_teardown_timer() does not allow that
for cpu #0. I added some silly loop which tries to match vcpu/cpu in
bind_virq_to_irq() using EVTCHNOP_status to find the currently used port
number. This helps and booting proceeds, but I feel that cant be the
right approach.

Olaf

....
[ 0.000000] No AGP bridge found
[ 0.000000] Memory: 486288k/524288k available (2812k kernel code, 452k absent, 37548k reserved, 3557k data, 480k init)
[ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=8, Nodes=1
[ 0.000000] Hierarchical RCU implementation.
[ 0.000000] NR_IRQS:4352 nr_irqs:1152 16
[ 0.000000] Xen HVM callback vector for event delivery is enabled
[ 0.000000] Console: colour VGA+ 80x25
[ 0.000000] console [ttyS0] enabled
[ 0.000000] Detected 2793.044 MHz processor.
[ 0.004000] Calibrating delay loop (skipped), value calculated using timer frequency.. 5586.08 BogoMIPS (lpj=11172176)
[ 0.004000] pid_max: default: 32768 minimum: 301
[ 0.004000] Mount-cache hash table entries: 256
[ 0.004231] CPU: Physical Processor ID: 0
[ 0.005459] CPU: Processor Core ID: 0
[ 0.008671] ACPI: Core revision 20110413
[ 0.020172] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
[ 0.061644] CPU0: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz stepping 05
[ 0.063779] Xen: using vcpuop timer interface
[ 0.064006] installing Xen timer for CPU 0
[ 0.065166] ------------[ cut here ]------------
[ 0.066468] kernel BUG at drivers/xen/events.c:902!
[ 0.067855] invalid opcode: 0000 [#1] SMP
[ 0.068003] CPU 0
[ 0.068003] Modules linked in:
[ 0.068003]
[ 0.068003] Pid: 1, comm: swapper Not tainted 3.0.0-20110722.1852.20-kernel-linux-3_0 #1 Xen HVM domU
[ 0.068003] RIP: 0010:[<ffffffff811b41dc>] [<ffffffff811b41dc>] bind_virq_to_irq+0xac/0x170
[ 0.068003] RSP: 0018:ffff88001d89fe20 EFLAGS: 00010282
[ 0.068003] RAX: ffffffffffffffef RBX: 0000000000000040 RCX: 0000000000000246
[ 0.068003] RDX: 00000000ffffffef RSI: ffff88001d89fe40 RDI: 0000000000000001
[ 0.068003] RBP: 0000000000000000 R08: ffff88001d807500 R09: ffffffff818ee2e8
[ 0.068003] R10: 000000000000005a R11: 0000000000000000 R12: 0000000000000000
[ 0.068003] R13: 0000000000000000 R14: 000000000000ebe0 R15: 0000000000000000
[ 0.068003] FS: 0000000000000000(0000) GS:ffff88001ea00000(0000) knlGS:0000000000000000
[ 0.068003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 0.068003] CR2: 00007f7c9b6e0ae0 CR3: 0000000001603000 CR4: 00000000000006b0
[ 0.068003] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 0.068003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 0.068003] Process swapper (pid: 1, threadinfo ffff88001d89e000, task ffff88001d8a0000)
[ 0.068003] Stack:
[ 0.068003] ffff88001d89fe80 0000000000000000 0000000000000000 0000000000000000
[ 0.068003] 0000000000000000 0000000000000000 ffffffff81636120 0000000000000000
[ 0.068003] 0000000000000000 ffff88001d830050 000000000001ce20 ffffffff81007870
[ 0.068003] Call Trace:
[ 0.068003] [<ffffffff81007870>] ? xen_hvm_setup_cpu_clockevents+0x20/0x20
[ 0.068003] [<ffffffff811b479d>] ? bind_virq_to_irqhandler+0x2d/0x80
[ 0.068003] [<ffffffff81007801>] ? xen_setup_timer+0x51/0xa0
[ 0.068003] [<ffffffff816500e2>] ? xen_time_init+0x8a/0xaf
[ 0.068003] [<ffffffff81659031>] ? native_smp_prepare_cpus+0x216/0x374
[ 0.068003] [<ffffffff816508bc>] ? xen_hvm_smp_prepare_cpus+0x9/0x2e
[ 0.068003] [<ffffffff8164c551>] ? kernel_init+0x3f/0x128
[ 0.068003] [<ffffffff812bc604>] ? kernel_thread_helper+0x4/0x10
[ 0.068003] [<ffffffff8164c512>] ? parse_early_options+0x1a/0x1a
[ 0.068003] [<ffffffff812bc600>] ? gs_change+0x13/0x13
[ 0.068003] Code: 63 81 e8 f8 dc ec ff 44 89 64 24 20 44 89 6c 24 24 bf 01 00 00 00 48 8d 74 24 20 e8 2f d2 e4 ff 83 f8 da 89 c2 74 6e 85 d2 74 04 <0f> 0b eb fe 89 df 8b 6c 24 28 e8 b5 e5 ff ff 48 89 c1 8b 40 10
[ 0.068003] RIP [<ffffffff811b41dc>] bind_virq_to_irq+0xac/0x170
[ 0.068003] RSP <ffff88001d89fe20>
[ 0.132010] ---[ end trace 4eaa2a86a8e2da22 ]---

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pv-on-hvm kexec, howto reregister timer interrupt [ In reply to ]

keir.xen at gmail

Jul 22, 2011, 11:57 PM

Post #2 of 5 (586 views)

Permalink

On 22/07/2011 19:49, "Olaf Hering" <olaf@aepfle.de> wrote:

> What is the best way to reregister the timer interrupt in the kexec
> kernel in a pv-on-hvm guest? Right now the crash shown below happens
> because EVTCHNOP_bind_virq returns -EEXIST. Can the timer on the cpus be
> unregistered during shutdown? xen_teardown_timer() does not allow that
> for cpu #0. I added some silly loop which tries to match vcpu/cpu in
> bind_virq_to_irq() using EVTCHNOP_status to find the currently used port
> number. This helps and booting proceeds, but I feel that cant be the
> right approach.

Sounbds like it's simply some Linux-guest issue here to be untangled. All
Xen requires you to do is EVTCHNOP_close the old virq-evtchn binding.

-- Keir

> Olaf
>
>
>
> ....
> [ 0.000000] No AGP bridge found
> [ 0.000000] Memory: 486288k/524288k available (2812k kernel code, 452k
> absent, 37548k reserved, 3557k data, 480k init)
> [ 0.000000] SLUB: Genslabs=15, HWalign=64, Order=0-3, MinObjects=0, CPUs=8,
> Nodes=1
> [ 0.000000] Hierarchical RCU implementation.
> [ 0.000000] NR_IRQS:4352 nr_irqs:1152 16
> [ 0.000000] Xen HVM callback vector for event delivery is enabled
> [ 0.000000] Console: colour VGA+ 80x25
> [ 0.000000] console [ttyS0] enabled
> [ 0.000000] Detected 2793.044 MHz processor.
> [ 0.004000] Calibrating delay loop (skipped), value calculated using timer
> frequency.. 5586.08 BogoMIPS (lpj=11172176)
> [ 0.004000] pid_max: default: 32768 minimum: 301
> [ 0.004000] Mount-cache hash table entries: 256
> [ 0.004231] CPU: Physical Processor ID: 0
> [ 0.005459] CPU: Processor Core ID: 0
> [ 0.008671] ACPI: Core revision 20110413
> [ 0.020172] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=0 pin2=0
> [ 0.061644] CPU0: Intel(R) Core(TM) i7 CPU 860 @ 2.80GHz stepping
> 05
> [ 0.063779] Xen: using vcpuop timer interface
> [ 0.064006] installing Xen timer for CPU 0
> [ 0.065166] ------------[ cut here ]------------
> [ 0.066468] kernel BUG at drivers/xen/events.c:902!
> [ 0.067855] invalid opcode: 0000 [#1] SMP
> [ 0.068003] CPU 0
> [ 0.068003] Modules linked in:
> [ 0.068003]
> [ 0.068003] Pid: 1, comm: swapper Not tainted
> 3.0.0-20110722.1852.20-kernel-linux-3_0 #1 Xen HVM domU
> [ 0.068003] RIP: 0010:[<ffffffff811b41dc>] [<ffffffff811b41dc>]
> bind_virq_to_irq+0xac/0x170
> [ 0.068003] RSP: 0018:ffff88001d89fe20 EFLAGS: 00010282
> [ 0.068003] RAX: ffffffffffffffef RBX: 0000000000000040 RCX:
> 0000000000000246
> [ 0.068003] RDX: 00000000ffffffef RSI: ffff88001d89fe40 RDI:
> 0000000000000001
> [ 0.068003] RBP: 0000000000000000 R08: ffff88001d807500 R09:
> ffffffff818ee2e8
> [ 0.068003] R10: 000000000000005a R11: 0000000000000000 R12:
> 0000000000000000
> [ 0.068003] R13: 0000000000000000 R14: 000000000000ebe0 R15:
> 0000000000000000
> [ 0.068003] FS: 0000000000000000(0000) GS:ffff88001ea00000(0000)
> knlGS:0000000000000000
> [ 0.068003] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
> [ 0.068003] CR2: 00007f7c9b6e0ae0 CR3: 0000000001603000 CR4:
> 00000000000006b0
> [ 0.068003] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
> 0000000000000000
> [ 0.068003] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
> 0000000000000400
> [ 0.068003] Process swapper (pid: 1, threadinfo ffff88001d89e000, task
> ffff88001d8a0000)
> [ 0.068003] Stack:
> [ 0.068003] ffff88001d89fe80 0000000000000000 0000000000000000
> 0000000000000000
> [ 0.068003] 0000000000000000 0000000000000000 ffffffff81636120
> 0000000000000000
> [ 0.068003] 0000000000000000 ffff88001d830050 000000000001ce20
> ffffffff81007870
> [ 0.068003] Call Trace:
> [ 0.068003] [<ffffffff81007870>] ? xen_hvm_setup_cpu_clockevents+0x20/0x20
> [ 0.068003] [<ffffffff811b479d>] ? bind_virq_to_irqhandler+0x2d/0x80
> [ 0.068003] [<ffffffff81007801>] ? xen_setup_timer+0x51/0xa0
> [ 0.068003] [<ffffffff816500e2>] ? xen_time_init+0x8a/0xaf
> [ 0.068003] [<ffffffff81659031>] ? native_smp_prepare_cpus+0x216/0x374
> [ 0.068003] [<ffffffff816508bc>] ? xen_hvm_smp_prepare_cpus+0x9/0x2e
> [ 0.068003] [<ffffffff8164c551>] ? kernel_init+0x3f/0x128
> [ 0.068003] [<ffffffff812bc604>] ? kernel_thread_helper+0x4/0x10
> [ 0.068003] [<ffffffff8164c512>] ? parse_early_options+0x1a/0x1a
> [ 0.068003] [<ffffffff812bc600>] ? gs_change+0x13/0x13
> [ 0.068003] Code: 63 81 e8 f8 dc ec ff 44 89 64 24 20 44 89 6c 24 24 bf 01
> 00 00 00 48 8d 74 24 20 e8 2f d2 e4 ff 83 f8 da 89 c2 74 6e 85 d2 74 04 <0f>
> 0b eb fe 89 df 8b 6c 24 28 e8 b5 e5 ff ff 48 89 c1 8b 40 10
> [ 0.068003] RIP [<ffffffff811b41dc>] bind_virq_to_irq+0xac/0x170
> [ 0.068003] RSP <ffff88001d89fe20>
> [ 0.132010] ---[ end trace 4eaa2a86a8e2da22 ]---
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pv-on-hvm kexec, howto reregister timer interrupt [ In reply to ]

Ian.Campbell at citrix

Jul 23, 2011, 1:01 AM

Post #3 of 5 (587 views)

Permalink

On Sat, 2011-07-23 at 07:57 +0100, Keir Fraser wrote:
> On 22/07/2011 19:49, "Olaf Hering" <olaf@aepfle.de> wrote:
>
> > What is the best way to reregister the timer interrupt in the kexec
> > kernel in a pv-on-hvm guest? Right now the crash shown below happens
> > because EVTCHNOP_bind_virq returns -EEXIST. Can the timer on the cpus be
> > unregistered during shutdown? xen_teardown_timer() does not allow that
> > for cpu #0. I added some silly loop which tries to match vcpu/cpu in
> > bind_virq_to_irq() using EVTCHNOP_status to find the currently used port
> > number. This helps and booting proceeds, but I feel that cant be the
> > right approach.
>
> Sounbds like it's simply some Linux-guest issue here to be untangled. All
> Xen requires you to do is EVTCHNOP_close the old virq-evtchn binding.

That might be tricky to do in the target kernel, which may not know what
needs closing. There's EVTCHNOP_reset which looks like it would be a
sensible thing to call early on in the target but seems like it would
close to much -- e.g. the xenstore evtchn?

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pv-on-hvm kexec, howto reregister timer interrupt [ In reply to ]

olaf at aepfle

Jul 23, 2011, 1:52 AM

Post #4 of 5 (584 views)

Permalink

On Sat, Jul 23, Ian Campbell wrote:

> On Sat, 2011-07-23 at 07:57 +0100, Keir Fraser wrote:
> > On 22/07/2011 19:49, "Olaf Hering" <olaf@aepfle.de> wrote:
> >
> > > What is the best way to reregister the timer interrupt in the kexec
> > > kernel in a pv-on-hvm guest? Right now the crash shown below happens
> > > because EVTCHNOP_bind_virq returns -EEXIST. Can the timer on the cpus be
> > > unregistered during shutdown? xen_teardown_timer() does not allow that
> > > for cpu #0. I added some silly loop which tries to match vcpu/cpu in
> > > bind_virq_to_irq() using EVTCHNOP_status to find the currently used port
> > > number. This helps and booting proceeds, but I feel that cant be the
> > > right approach.
> >
> > Sounbds like it's simply some Linux-guest issue here to be untangled. All
> > Xen requires you to do is EVTCHNOP_close the old virq-evtchn binding.
>
> That might be tricky to do in the target kernel, which may not know what
> needs closing. There's EVTCHNOP_reset which looks like it would be a
> sensible thing to call early on in the target but seems like it would
> close to much -- e.g. the xenstore evtchn?

After some more poking around in the reboot code path I have figured it
out, a syscore_ops has to be registerd to shutdown the irqs.
I will post some patches for review.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: pv-on-hvm kexec, howto reregister timer interrupt [ In reply to ]

keir.xen at gmail

Jul 23, 2011, 2:55 AM

Post #5 of 5 (591 views)

Permalink

On 23/07/2011 09:01, "Ian Campbell" <Ian.Campbell@citrix.com> wrote:

>> Sounbds like it's simply some Linux-guest issue here to be untangled. All
>> Xen requires you to do is EVTCHNOP_close the old virq-evtchn binding.
>
> That might be tricky to do in the target kernel, which may not know what
> needs closing. There's EVTCHNOP_reset which looks like it would be a
> sensible thing to call early on in the target but seems like it would
> close to much -- e.g. the xenstore evtchn?

If you can't do it in the source kernel (e.g., because it's crashed) the
best bet is to call EVTCHN_status on every port to find the channels you
want to clobber. Which is what Olaf hacked up already I believe, just needs
integrating cleanly.

-- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel