Mailing List Archive

On 22/11/2011 15:07, "Olaf Hering" <olaf@aepfle.de> wrote:

> On Tue, Nov 22, Keir Fraser wrote:
>
>> On 22/11/2011 13:54, "Olaf Hering" <olaf@aepfle.de> wrote:
>>
>>> On Tue, Nov 22, Keir Fraser wrote:
>>>
>>>> I think I checked before, but: also unresponsive to serial debug keys?
>>>
>>> Good point, I will check that. So far I havent used these keys.
>>
>> If they work then 'd' will give you a backtrace on every CPU, and 'q' will
>> dump domain and vcpu states. That should make things easier!
>
> They do indeed work. The backtrace below is from another system.
> Looks like hpet_broadcast_exit() is involved.
>
> Does that output below give any good hints?

It tells us that the hypervisor itself is in good shape. The deterministic
RIP in hpet_broadcast_exit() is simply because the serial rx interrupt is
always waking us from the idle loop. That RIP value will simply be the first
possible interruption point after the HLT instruction.

I have a new theory, which is that if we go round the for-loop in
wait_event() more than once, the vcpu's pause counter gets messed up and
goes negative, condemning it to sleep forever.

I have *just* pushed a change to the debug 'q' key (ignore the changeset
comment referring to 'd' key, I got that wrong!) which will print per-vcpu
and per-domain pause_count values. Please get the system stuck again, and
send the output from 'q' key with that new changeset (c/s 24178).

Finally, I don't really know what the prep/wake/done messages from your logs
mean, as you didn't send the patch that prints them.

-- Keir

>> Try the attached patch (please also try reducing the size of the new
>> parameter to the inline asm from PAGE_SIZE down to e.g. 2000 to force the
>> domain-crashing path).
>
> Thanks, I will try it.
>
>
> Olaf
>
>
> ..........
>
> (XEN) 'q' pressed -> dumping domain info (now=0x5E:F50D77F8)
> (XEN) General information for domain 0:
> (XEN) refcnt=3 dying=0 nr_pages=1852873 xenheap_pages=5 dirty_cpus={}
> max_pages=4294967295
> (XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=00000004
> (XEN) Rangesets belonging to domain 0:
> (XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-3f7, 400-807, 80c-cfb,
> d00-ffff }
> (XEN) Interrupts { 0-207 }
> (XEN) I/O Memory { 0-febff, fec01-fedff, fee01-ffffffffffffffff }
> (XEN) Memory pages belonging to domain 0:
> (XEN) DomPage list too long to display
> (XEN) XenPage 000000000021e6d9: caf=c000000000000002, taf=7400000000000002
> (XEN) XenPage 000000000021e6d8: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000021e6d7: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000021e6d6: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 00000000000db2fe: caf=c000000000000002, taf=7400000000000002
> (XEN) VCPU information and callbacks for domain 0:
> (XEN) VCPU0: CPU0 [has=F] flags=0 poll=0 upcall_pend = 01, upcall_mask =
> 00 dirty_cpus={} cpu_affinity={0}
> (XEN) 250 Hz periodic timer (period 4 ms)
> (XEN) General information for domain 1:
> (XEN) refcnt=3 dying=0 nr_pages=3645 xenheap_pages=6 dirty_cpus={}
> max_pages=131328
> (XEN) handle=d80155e4-8f8b-94e1-8382-94084b7f1e51 vm_assist=00000000
> (XEN) paging assistance: hap refcounts log_dirty translate external
> (XEN) Rangesets belonging to domain 1:
> (XEN) I/O Ports { }
> (XEN) Interrupts { }
> (XEN) I/O Memory { }
> (XEN) Memory pages belonging to domain 1:
> (XEN) DomPage list too long to display
> (XEN) PoD entries=0 cachesize=0
> (XEN) XenPage 000000000020df70: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020e045: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020c58c: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020c5a4: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 0000000000019f1e: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020eb23: caf=c000000000000001, taf=7400000000000001
> (XEN) VCPU information and callbacks for domain 1:
> (XEN) VCPU0: CPU0 [has=F] flags=4 poll=0 upcall_pend = 00, upcall_mask =
> 00 dirty_cpus={} cpu_affinity={0}
> (XEN) paging assistance: hap, 4 levels
> (XEN) No periodic timer
> (XEN) Notifying guest 0:0 (virq 1, port 0, stat 0/-1/-1)
> (XEN) Notifying guest 1:0 (virq 1, port 0, stat 0/0/0)
> (XEN) 'q' pressed -> dumping domain info (now=0x60:A7DD8B08)
> (XEN) General information for domain 0:
> (XEN) refcnt=3 dying=0 nr_pages=1852873 xenheap_pages=5 dirty_cpus={}
> max_pages=4294967295
> (XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=00000004
> (XEN) Rangesets belonging to domain 0:
> (XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-3f7, 400-807, 80c-cfb,
> d00-ffff }
> (XEN) Interrupts { 0-207 }
> (XEN) I/O Memory { 0-febff, fec01-fedff, fee01-ffffffffffffffff }
> (XEN) Memory pages belonging to domain 0:
> (XEN) DomPage list too long to display
> (XEN) XenPage 000000000021e6d9: caf=c000000000000002, taf=7400000000000002
> (XEN) XenPage 000000000021e6d8: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000021e6d7: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000021e6d6: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 00000000000db2fe: caf=c000000000000002, taf=7400000000000002
> (XEN) VCPU information and callbacks for domain 0:
> (XEN) VCPU0: CPU0 [has=F] flags=0 poll=0 upcall_pend = 01, upcall_mask =
> 00 dirty_cpus={} cpu_affinity={0}
> (XEN) 250 Hz periodic timer (period 4 ms)
> (XEN) General information for domain 1:
> (XEN) refcnt=3 dying=0 nr_pages=3645 xenheap_pages=6 dirty_cpus={}
> max_pages=131328
> (XEN) handle=d80155e4-8f8b-94e1-8382-94084b7f1e51 vm_assist=00000000
> (XEN) paging assistance: hap refcounts log_dirty translate external
> (XEN) Rangesets belonging to domain 1:
> (XEN) I/O Ports { }
> (XEN) Interrupts { }
> (XEN) I/O Memory { }
> (XEN) Memory pages belonging to domain 1:
> (XEN) DomPage list too long to display
> (XEN) PoD entries=0 cachesize=0
> (XEN) XenPage 000000000020df70: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020e045: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020c58c: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020c5a4: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 0000000000019f1e: caf=c000000000000001, taf=7400000000000001
> (XEN) XenPage 000000000020eb23: caf=c000000000000001, taf=7400000000000001
> (XEN) VCPU information and callbacks for domain 1:
> (XEN) VCPU0: CPU0 [has=F] flags=4 poll=0 upcall_pend = 00, upcall_mask =
> 00 dirty_cpus={} cpu_affinity={0}
> (XEN) paging assistance: hap, 4 levels
> (XEN) No periodic timer
> (XEN) Notifying guest 0:0 (virq 1, port 0, stat 0/-1/-1)
> (XEN) Notifying guest 1:0 (virq 1, port 0, stat 0/0/0)
> (XEN) 'd' pressed -> dumping registers
> (XEN)
> (XEN) *** Dumping CPU0 host state: ***
> (XEN) ----[ Xen-4.2.24169-20111122.144218 x86_64 debug=y Tainted: C
> ]----
> (XEN) CPU: 0
> (XEN) RIP: e008:[<ffff82c48019bfe6>] hpet_broadcast_exit+0x0/0x1f9
> (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
> (XEN) rax: 0000000000003b40 rbx: 000000674742e72d rcx: 0000000000000001
> (XEN) rdx: 0000000000000000 rsi: ffff82c48030f000 rdi: ffff82c4802bfea0
> (XEN) rbp: ffff82c4802bfee0 rsp: ffff82c4802bfe78 r8: 000000008c858211
> (XEN) r9: 0000000000000003 r10: ffff82c4803064e0 r11: 000000676bf885a3
> (XEN) r12: ffff83021e70e840 r13: ffff83021e70e8d0 r14: 00000067471bdb62
> (XEN) r15: ffff82c48030e440 cr0: 000000008005003b cr4: 00000000000026f0
> (XEN) cr3: 00000000db4c4000 cr2: 0000000000beb000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
> (XEN) Xen stack trace from rsp=ffff82c4802bfe78:
> (XEN) ffff82c48019f0ca ffff82c4802bff18 ffffffffffffffff ffff82c4802bfed0
> (XEN) 0000000180124b57 0000000000000000 0000000000000000 ffff82c48025b200
> (XEN) 0000152900006fe3 ffff82c4802bff18 ffff82c48025b200 ffff82c4802bff18
> (XEN) ffff82c48030e468 ffff82c4802bff10 ffff82c48015a88d 0000000000000000
> (XEN) ffff8300db6c6000 ffff8300db6c6000 ffffffffffffffff ffff82c4802bfe00
> (XEN) 0000000000000000 0000000000001000 0000000000001000 0000000000000000
> (XEN) 8000000000000427 ffff8801d8579010 0000000000000246 00000000deadbeef
> (XEN) ffff8801d8579000 ffff8801d8579000 00000000fffffffe ffffffff8000302a
> (XEN) 00000000deadbeef 00000000deadbeef 00000000deadbeef 0000010000000000
> (XEN) ffffffff8000302a 000000000000e033 0000000000000246 ffff8801a515bd10
> (XEN) 000000000000e02b 000000000000beef 000000000000beef 000000000000beef
> (XEN) 000000000000beef 0000000000000000 ffff8300db6c6000 0000000000000000
> (XEN) 0000000000000000
> (XEN) Xen call trace:
> (XEN) [<ffff82c48019bfe6>] hpet_broadcast_exit+0x0/0x1f9
> (XEN) [<ffff82c48015a88d>] idle_loop+0x6c/0x7b
> (XEN)
> (XEN) 'd' pressed -> dumping registers
> (XEN)
> (XEN) *** Dumping CPU0 host state: ***
> (XEN) ----[ Xen-4.2.24169-20111122.144218 x86_64 debug=y Tainted: C
> ]----
> (XEN) CPU: 0
> (XEN) RIP: e008:[<ffff82c48019bfe6>] hpet_broadcast_exit+0x0/0x1f9
> (XEN) RFLAGS: 0000000000000246 CONTEXT: hypervisor
> (XEN) rax: 0000000000003b40 rbx: 00000078f4fbe7ed rcx: 0000000000000001
> (XEN) rdx: 0000000000000000 rsi: ffff82c48030f000 rdi: ffff82c4802bfea0
> (XEN) rbp: ffff82c4802bfee0 rsp: ffff82c4802bfe78 r8: 00000000cd4f8db6
> (XEN) r9: 0000000000000002 r10: ffff82c480308780 r11: 000000790438291d
> (XEN) r12: ffff83021e70e840 r13: ffff83021e70e8d0 r14: 00000078f412a61c
> (XEN) r15: ffff82c48030e440 cr0: 000000008005003b cr4: 00000000000026f0
> (XEN) cr3: 00000000db4c4000 cr2: 0000000000beb000
> (XEN) ds: 0000 es: 0000 fs: 0000 gs: 0000 ss: e010 cs: e008
> (XEN) Xen stack trace from rsp=ffff82c4802bfe78:
> (XEN) ffff82c48019f0ca ffff82c4802bff18 ffffffffffffffff ffff82c4802bfed0
> (XEN) 0000000180124b57 0000000000000000 0000000000000000 ffff82c48025b200
> (XEN) 0000239e00007657 ffff82c4802bff18 ffff82c48025b200 ffff82c4802bff18
> (XEN) ffff82c48030e468 ffff82c4802bff10 ffff82c48015a88d 0000000000000000
> (XEN) ffff8300db6c6000 ffff8300db6c6000 ffffffffffffffff ffff82c4802bfe00
> (XEN) 0000000000000000 0000000000001000 0000000000001000 0000000000000000
> (XEN) 8000000000000427 ffff8801d8579010 0000000000000246 00000000deadbeef
> (XEN) ffff8801d8579000 ffff8801d8579000 00000000fffffffe ffffffff8000302a
> (XEN) 00000000deadbeef 00000000deadbeef 00000000deadbeef 0000010000000000
> (XEN) ffffffff8000302a 000000000000e033 0000000000000246 ffff8801a515bd10
> (XEN) 000000000000e02b 000000000000beef 000000000000beef 000000000000beef
> (XEN) 000000000000beef 0000000000000000 ffff8300db6c6000 0000000000000000
> (XEN) 0000000000000000
> (XEN) Xen call trace:
> (XEN) [<ffff82c48019bfe6>] hpet_broadcast_exit+0x0/0x1f9
> (XEN) [<ffff82c48015a88d>] idle_loop+0x6c/0x7b
> (XEN)
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/2011 15:40, "Keir Fraser" <keir@xen.org> wrote:

> I have a new theory, which is that if we go round the for-loop in
> wait_event() more than once, the vcpu's pause counter gets messed up and
> goes negative, condemning it to sleep forever.

Further to this, can you please try moving the call to __prepare_to_wait()
from just after the spinlock region to just before (i.e., immediately after
the ASSERT), in prepare_to_wait(). That could well makes things work better.
On UP at least -- for SMP systems I will also need to fix the broken usage
of wqv->esp...

-- Keir




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Nov 22, Keir Fraser wrote:

> I have a new theory, which is that if we go round the for-loop in
> wait_event() more than once, the vcpu's pause counter gets messed up and
> goes negative, condemning it to sleep forever.

I have added a check for that, its not negative.

> I have *just* pushed a change to the debug 'q' key (ignore the changeset
> comment referring to 'd' key, I got that wrong!) which will print per-vcpu
> and per-domain pause_count values. Please get the system stuck again, and
> send the output from 'q' key with that new changeset (c/s 24178).

To me it looks like dom0 gets paused, perhaps due to some uneven pause/unpause calls.
I will see if I can figure it out.

Olaf

(XEN) 'q' pressed -> dumping domain info (now=0xA1:4BC733CC)
(XEN) General information for domain 0:
(XEN) refcnt=3 dying=0 pause_count=0
(XEN) nr_pages=5991502 xenheap_pages=5 dirty_cpus={} max_pages=4294967295
(XEN) handle=00000000-0000-0000-0000-000000000000 vm_assist=00000004
(XEN) Rangesets belonging to domain 0:
(XEN) I/O Ports { 0-1f, 22-3f, 44-60, 62-9f, a2-3f7, 400-407, 40c-cfb, d00-ffff }
(XEN) Interrupts { 0-303 }
(XEN) I/O Memory { 0-febff, fec01-fec8f, fec91-fedff, fee01-ffffffffffffffff }
(XEN) Memory pages belonging to domain 0:
(XEN) DomPage list too long to display
(XEN) XenPage 000000000036ff8d: caf=c000000000000002, taf=7400000000000002
(XEN) XenPage 000000000036ff8c: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 000000000036ff8b: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 000000000036ff8a: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 000000000008befd: caf=c000000000000002, taf=7400000000000002
(XEN) VCPU information and callbacks for domain 0:
(XEN) VCPU0: CPU0 [has=F] poll=0 upcall_pend = 01, upcall_mask = 00 dirty_cpus={} cpu_affinity={0}
(XEN) pause_count=1 pause_flags=0
(XEN) 250 Hz periodic timer (period 4 ms)
(XEN) General information for domain 1:
(XEN) refcnt=3 dying=0 pause_count=0
(XEN) nr_pages=17549 xenheap_pages=6 dirty_cpus={} max_pages=131328
(XEN) handle=5499728e-7f38-dbb0-b6cc-22866a6864f3 vm_assist=00000000
(XEN) paging assistance: hap refcounts translate external
(XEN) Rangesets belonging to domain 1:
(XEN) I/O Ports { }
(XEN) Interrupts { }
(XEN) I/O Memory { }
(XEN) Memory pages belonging to domain 1:
(XEN) DomPage list too long to display
(XEN) PoD entries=0 cachesize=0
(XEN) XenPage 0000000000200b7c: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 0000000000203bfe: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 0000000000200b48: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 000000000021291d: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 000000000003ebfc: caf=c000000000000001, taf=7400000000000001
(XEN) XenPage 0000000000202ef4: caf=c000000000000001, taf=7400000000000001
(XEN) VCPU information and callbacks for domain 1:
(XEN) VCPU0: CPU0 [has=F] poll=0 upcall_pend = 00, upcall_mask = 00 dirty_cpus={} cpu_affinity={0}
(XEN) pause_count=0 pause_flags=4
(XEN) paging assistance: hap, 4 levels
(XEN) No periodic timer
(XEN) Notifying guest 0:0 (virq 1, port 0, stat 0/-1/-1)
(XEN) Notifying guest 1:0 (virq 1, port 0, stat 0/0/0)


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/2011 17:36, "Olaf Hering" <olaf@aepfle.de> wrote:

> On Tue, Nov 22, Keir Fraser wrote:
>
>> I have a new theory, which is that if we go round the for-loop in
>> wait_event() more than once, the vcpu's pause counter gets messed up and
>> goes negative, condemning it to sleep forever.
>
> I have added a check for that, its not negative.
>
>> I have *just* pushed a change to the debug 'q' key (ignore the changeset
>> comment referring to 'd' key, I got that wrong!) which will print per-vcpu
>> and per-domain pause_count values. Please get the system stuck again, and
>> send the output from 'q' key with that new changeset (c/s 24178).
>
> To me it looks like dom0 gets paused, perhaps due to some uneven pause/unpause
> calls.
> I will see if I can figure it out.

Could it have ended up on the waitqueue?

-- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Nov 22, Keir Fraser wrote:

> Could it have ended up on the waitqueue?

Unlikely, but I will add checks for that as well.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Nov 22, Olaf Hering wrote:

> On Tue, Nov 22, Keir Fraser wrote:
>
> > Could it have ended up on the waitqueue?
>
> Unlikely, but I will add checks for that as well.

I posted three changes which make use of the wait queues.
For some reason the code at the very end of p2m_mem_paging_populate()
triggers when d is dom0, so its vcpu is put to sleep.


Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 22/11/2011 21:15, "Olaf Hering" <olaf@aepfle.de> wrote:

> On Tue, Nov 22, Olaf Hering wrote:
>
>> On Tue, Nov 22, Keir Fraser wrote:
>>
>>> Could it have ended up on the waitqueue?
>>
>> Unlikely, but I will add checks for that as well.
>
> I posted three changes which make use of the wait queues.
> For some reason the code at the very end of p2m_mem_paging_populate()
> triggers when d is dom0, so its vcpu is put to sleep.

We obviously can't have dom0 going to sleep on paging work. This, at least,
isn't a wait-queue bug.

> Olaf



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Tue, Nov 22, Keir Fraser wrote:

> We obviously can't have dom0 going to sleep on paging work. This, at least,
> isn't a wait-queue bug.

I had to rearrange some code in p2m_mem_paging_populate for my debug
stuff. This led to an uninitialized req, and as a result req.flags
sometimes had MEM_EVENT_FLAG_VCPU_PAUSED set. For some reason gcc did
not catch that..
Now waitqueues appear to work ok for me. Thanks!


What do you think about C99 initializers in p2m_mem_paging_populate,
just to avoid such mistakes?

mem_event_request_t req = { .type = MEM_EVENT_TYPE_PAGING };

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23/11/2011 17:00, "Olaf Hering" <olaf@aepfle.de> wrote:

> On Tue, Nov 22, Keir Fraser wrote:
>
>> We obviously can't have dom0 going to sleep on paging work. This, at least,
>> isn't a wait-queue bug.
>
> I had to rearrange some code in p2m_mem_paging_populate for my debug
> stuff. This led to an uninitialized req, and as a result req.flags
> sometimes had MEM_EVENT_FLAG_VCPU_PAUSED set. For some reason gcc did
> not catch that..
> Now waitqueues appear to work ok for me. Thanks!

Great. However, while eyeballing wait.c I spotted at least two bugs. I'm
pretty sure that the hypervisor will blow up pretty quickly when you resume
testing with multiple physical CPUs, for example. I need to create a couple
of fixup patches which I will then send to you for test.

By the way, did you test my patch to domain_crash when the stack-save area
isn't large enough?

> What do you think about C99 initializers in p2m_mem_paging_populate,
> just to avoid such mistakes?
>
> mem_event_request_t req = { .type = MEM_EVENT_TYPE_PAGING };

We like them.

-- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 23, Keir Fraser wrote:

> On 23/11/2011 17:00, "Olaf Hering" <olaf@aepfle.de> wrote:
>
> > On Tue, Nov 22, Keir Fraser wrote:
> >
> >> We obviously can't have dom0 going to sleep on paging work. This, at least,
> >> isn't a wait-queue bug.
> >
> > I had to rearrange some code in p2m_mem_paging_populate for my debug
> > stuff. This led to an uninitialized req, and as a result req.flags
> > sometimes had MEM_EVENT_FLAG_VCPU_PAUSED set. For some reason gcc did
> > not catch that..
> > Now waitqueues appear to work ok for me. Thanks!
>
> Great. However, while eyeballing wait.c I spotted at least two bugs. I'm
> pretty sure that the hypervisor will blow up pretty quickly when you resume
> testing with multiple physical CPUs, for example. I need to create a couple
> of fixup patches which I will then send to you for test.

Good, I will look forward for these fixes.

> By the way, did you test my patch to domain_crash when the stack-save area
> isn't large enough?

I ran into the ->esp == 0 case right away, but I need to retest with a
clean tree.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23/11/2011 17:16, "Keir Fraser" <keir.xen@gmail.com> wrote:

> On 23/11/2011 17:00, "Olaf Hering" <olaf@aepfle.de> wrote:
>
>> On Tue, Nov 22, Keir Fraser wrote:
>>
>>> We obviously can't have dom0 going to sleep on paging work. This, at least,
>>> isn't a wait-queue bug.
>>
>> I had to rearrange some code in p2m_mem_paging_populate for my debug
>> stuff. This led to an uninitialized req, and as a result req.flags
>> sometimes had MEM_EVENT_FLAG_VCPU_PAUSED set. For some reason gcc did
>> not catch that..
>> Now waitqueues appear to work ok for me. Thanks!
>
> Great. However, while eyeballing wait.c I spotted at least two bugs. I'm
> pretty sure that the hypervisor will blow up pretty quickly when you resume
> testing with multiple physical CPUs, for example. I need to create a couple
> of fixup patches which I will then send to you for test.

We have quite a big waitqueue problem actually. The current scheme of
per-cpu stacks doesn't work nicely, as the stack pointer will change if a
vcpu goes to sleep and then wakes up on a different cpu. This really doesn't
work nicely with preempted C code, which may implement frame pointers and/or
arbitrarily take the address of on-stack variables. The result will be
hideous cross-stack corruptions, as these frame pointers and cached
addresses of automatic variables will reference the wrong cpu's stack!
Fixing or detecting this in general is not possible afaics.

So, we'll have to switch to per-vcpu stacks, probably with separate per-cpu
irq stacks (as a later followup). That's quite a nuisance!

-- Keir

> By the way, did you test my patch to domain_crash when the stack-save area
> isn't large enough?
>
>> What do you think about C99 initializers in p2m_mem_paging_populate,
>> just to avoid such mistakes?
>>
>> mem_event_request_t req = { .type = MEM_EVENT_TYPE_PAGING };
>
> We like them.
>
> -- Keir
>
>



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23/11/2011 18:06, "Olaf Hering" <olaf@aepfle.de> wrote:

> On Wed, Nov 23, Keir Fraser wrote:
>
>> On 23/11/2011 17:00, "Olaf Hering" <olaf@aepfle.de> wrote:
>>
>>> On Tue, Nov 22, Keir Fraser wrote:
>>>
>>>> We obviously can't have dom0 going to sleep on paging work. This, at least,
>>>> isn't a wait-queue bug.
>>>
>>> I had to rearrange some code in p2m_mem_paging_populate for my debug
>>> stuff. This led to an uninitialized req, and as a result req.flags
>>> sometimes had MEM_EVENT_FLAG_VCPU_PAUSED set. For some reason gcc did
>>> not catch that..
>>> Now waitqueues appear to work ok for me. Thanks!
>>
>> Great. However, while eyeballing wait.c I spotted at least two bugs. I'm
>> pretty sure that the hypervisor will blow up pretty quickly when you resume
>> testing with multiple physical CPUs, for example. I need to create a couple
>> of fixup patches which I will then send to you for test.
>
> Good, I will look forward for these fixes.
>
>> By the way, did you test my patch to domain_crash when the stack-save area
>> isn't large enough?
>
> I ran into the ->esp == 0 case right away, but I need to retest with a
> clean tree.

I think I have a test the wrong way round. This doesn't really matter now
anyway. As I say in my previous email, stack management will have to be
redone for waitqueues.

-- Keir

> Olaf



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 23, Keir Fraser wrote:

> We have quite a big waitqueue problem actually. The current scheme of
> per-cpu stacks doesn't work nicely, as the stack pointer will change if a
> vcpu goes to sleep and then wakes up on a different cpu. This really doesn't
> work nicely with preempted C code, which may implement frame pointers and/or
> arbitrarily take the address of on-stack variables. The result will be
> hideous cross-stack corruptions, as these frame pointers and cached
> addresses of automatic variables will reference the wrong cpu's stack!
> Fixing or detecting this in general is not possible afaics.

Yes, I was thinking about that wakeup on different cpu as well.
As a quick fix/hack, perhaps the scheduler could make sure the vcpu
wakes up on the same cpu?

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23/11/2011 18:31, "Olaf Hering" <olaf@aepfle.de> wrote:

> On Wed, Nov 23, Keir Fraser wrote:
>
>> We have quite a big waitqueue problem actually. The current scheme of
>> per-cpu stacks doesn't work nicely, as the stack pointer will change if a
>> vcpu goes to sleep and then wakes up on a different cpu. This really doesn't
>> work nicely with preempted C code, which may implement frame pointers and/or
>> arbitrarily take the address of on-stack variables. The result will be
>> hideous cross-stack corruptions, as these frame pointers and cached
>> addresses of automatic variables will reference the wrong cpu's stack!
>> Fixing or detecting this in general is not possible afaics.
>
> Yes, I was thinking about that wakeup on different cpu as well.
> As a quick fix/hack, perhaps the scheduler could make sure the vcpu
> wakes up on the same cpu?

Could save old affinity and then vcpu_set_affinity. That will have to do for
now. Actually it should work okay as long as toolstack doesn't mess with
affinity meanwhile. I'll sort out a patch for this.

-- Keir

> Olaf



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23/11/2011 19:21, "Keir Fraser" <keir.xen@gmail.com> wrote:

> On 23/11/2011 18:31, "Olaf Hering" <olaf@aepfle.de> wrote:
>
>> On Wed, Nov 23, Keir Fraser wrote:
>>
>>> We have quite a big waitqueue problem actually. The current scheme of
>>> per-cpu stacks doesn't work nicely, as the stack pointer will change if a
>>> vcpu goes to sleep and then wakes up on a different cpu. This really doesn't
>>> work nicely with preempted C code, which may implement frame pointers and/or
>>> arbitrarily take the address of on-stack variables. The result will be
>>> hideous cross-stack corruptions, as these frame pointers and cached
>>> addresses of automatic variables will reference the wrong cpu's stack!
>>> Fixing or detecting this in general is not possible afaics.
>>
>> Yes, I was thinking about that wakeup on different cpu as well.
>> As a quick fix/hack, perhaps the scheduler could make sure the vcpu
>> wakes up on the same cpu?
>
> Could save old affinity and then vcpu_set_affinity. That will have to do for
> now. Actually it should work okay as long as toolstack doesn't mess with
> affinity meanwhile. I'll sort out a patch for this.

Attached three patches for you to try. They apply in sequence.
00: A fixed version of "domain_crash on stack overflow"
01: Reorders prepare_to_wait so that the vcpu will always be on the
waitqueue on exit (even if it has just been woken).
02: Ensures the vcpu wakes up on the same cpu that it slept on.

We need all of these. Just need testing to make sure they aren't horribly
broken. You should be able to test multi-processor host again with these.

-- Keir

> -- Keir
>
>> Olaf
>
>

On Wed, Nov 23, Keir Fraser wrote:

> Attached three patches for you to try. They apply in sequence.
> 00: A fixed version of "domain_crash on stack overflow"
> 01: Reorders prepare_to_wait so that the vcpu will always be on the
> waitqueue on exit (even if it has just been woken).
> 02: Ensures the vcpu wakes up on the same cpu that it slept on.
>
> We need all of these. Just need testing to make sure they aren't horribly
> broken. You should be able to test multi-processor host again with these.

Thanks Keir.

In a first test they work ok with multi-processor.
I get vcpu hangs when I balloon up and down with mem-set. Thats most
likely caused by uneven vcpu_pause/unpause calls in my changes which use
wait queue in mem_event handling and ept_get_entry. I will debug that
further.

After the vcpu hung I killed the guest and tried to start a new one.
Oddly enough I wasnt able to fully kill the guest, it remained in --p--d
state. Most vcpus were in paused state before that.

In another attempt I was able to run firefox in a guest. But after
trying to open all "latest headlines" in tabs the guest crashed. qemu-dm
log had alot of this (but nothing in xen dmesg):
track_dirty_vram(f0000000, 12c) failed (-1, 3)

xl vcpu-list shows
(null) 1 0 - --p 47.3 any cpu
(null) 1 1 12 --- 13.4 any cpu
(null) 1 2 - --p 4.3 any cpu
(null) 1 3 - --p 7.8 any cpu
(null) 1 4 - --p 3.5 any cpu
(null) 1 5 - --p 1.9 any cpu
(null) 1 6 - --p 1.6 any cpu
(null) 1 7 - --p 1.4 any cpu

Hmm, qemu-dm doesnt get killed in all cases, killing it destroys the guest..
I have seen that before already.


I will provide more test results tomorrow.


Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 23/11/2011 22:30, "Olaf Hering" <olaf@aepfle.de> wrote:

> After the vcpu hung I killed the guest and tried to start a new one.
> Oddly enough I wasnt able to fully kill the guest, it remained in --p--d
> state. Most vcpus were in paused state before that.

Dying but kept as a zombie by memory references...

> In another attempt I was able to run firefox in a guest. But after
> trying to open all "latest headlines" in tabs the guest crashed. qemu-dm
> log had alot of this (but nothing in xen dmesg):
> track_dirty_vram(f0000000, 12c) failed (-1, 3)
>
> xl vcpu-list shows
> (null) 1 0 - --p 47.3 any cpu
> (null) 1 1 12 --- 13.4 any cpu
> (null) 1 2 - --p 4.3 any cpu
> (null) 1 3 - --p 7.8 any cpu
> (null) 1 4 - --p 3.5 any cpu
> (null) 1 5 - --p 1.9 any cpu
> (null) 1 6 - --p 1.6 any cpu
> (null) 1 7 - --p 1.4 any cpu
>
> Hmm, qemu-dm doesnt get killed in all cases, killing it destroys the guest..
> I have seen that before already.

...from qemu-dm. Problem is that toolstack is not killing qemu-dm, or
qemu-dm is not responding to some shutdown signal.

> I will provide more test results tomorrow.

Thanks.

-- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 23.11.11 at 22:03, Keir Fraser <keir.xen@gmail.com> wrote:
> On 23/11/2011 19:21, "Keir Fraser" <keir.xen@gmail.com> wrote:
>
>> On 23/11/2011 18:31, "Olaf Hering" <olaf@aepfle.de> wrote:
>>
>>> On Wed, Nov 23, Keir Fraser wrote:
>>>
>>>> We have quite a big waitqueue problem actually. The current scheme of
>>>> per-cpu stacks doesn't work nicely, as the stack pointer will change if a
>>>> vcpu goes to sleep and then wakes up on a different cpu. This really doesn't
>>>> work nicely with preempted C code, which may implement frame pointers and/or
>>>> arbitrarily take the address of on-stack variables. The result will be
>>>> hideous cross-stack corruptions, as these frame pointers and cached
>>>> addresses of automatic variables will reference the wrong cpu's stack!
>>>> Fixing or detecting this in general is not possible afaics.
>>>
>>> Yes, I was thinking about that wakeup on different cpu as well.
>>> As a quick fix/hack, perhaps the scheduler could make sure the vcpu
>>> wakes up on the same cpu?
>>
>> Could save old affinity and then vcpu_set_affinity. That will have to do for
>> now. Actually it should work okay as long as toolstack doesn't mess with
>> affinity meanwhile. I'll sort out a patch for this.
>
> Attached three patches for you to try. They apply in sequence.
> 00: A fixed version of "domain_crash on stack overflow"
> 01: Reorders prepare_to_wait so that the vcpu will always be on the
> waitqueue on exit (even if it has just been woken).
> 02: Ensures the vcpu wakes up on the same cpu that it slept on.

Didn't we (long ago) settle on not permitting new calls to
domain_crash_synchronous()? Is it really impossible to just
domain_crash() in any of the instances these add?

Jan

> We need all of these. Just need testing to make sure they aren't horribly
> broken. You should be able to test multi-processor host again with these.
>
> -- Keir
>
>> -- Keir
>>
>>> Olaf
>>
>>




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/11/2011 09:15, "Jan Beulich" <JBeulich@suse.com> wrote:

>>
>> Attached three patches for you to try. They apply in sequence.
>> 00: A fixed version of "domain_crash on stack overflow"
>> 01: Reorders prepare_to_wait so that the vcpu will always be on the
>> waitqueue on exit (even if it has just been woken).
>> 02: Ensures the vcpu wakes up on the same cpu that it slept on.
>
> Didn't we (long ago) settle on not permitting new calls to
> domain_crash_synchronous()? Is it really impossible to just
> domain_crash() in any of the instances these add?

It's safe because you must be in a context that is safe to preempt. That's a
pre-condition for using a waitqueue. It's not safe to use domain_crash()
because the caller of wait_event() may not handle the exceptional return.

-- Keir



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 24/11/2011 09:15, "Jan Beulich" <JBeulich@suse.com> wrote:

>> Attached three patches for you to try. They apply in sequence.
>> 00: A fixed version of "domain_crash on stack overflow"
>> 01: Reorders prepare_to_wait so that the vcpu will always be on the
>> waitqueue on exit (even if it has just been woken).
>> 02: Ensures the vcpu wakes up on the same cpu that it slept on.
>
> Didn't we (long ago) settle on not permitting new calls to
> domain_crash_synchronous()?

This was a reaction to lazy patches which sprinkled d_c_s calls around
liberally, and in unsafe locations, as a dodge around proper error handling.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Nov 23, Keir Fraser wrote:

> ...from qemu-dm. Problem is that toolstack is not killing qemu-dm, or
> qemu-dm is not responding to some shutdown signal.

In the first crash there was no qemu-dm process left from what I
remember. I will see if it happens again.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Thu, Nov 24, Olaf Hering wrote:

> On Wed, Nov 23, Keir Fraser wrote:
>
> > ...from qemu-dm. Problem is that toolstack is not killing qemu-dm, or
> > qemu-dm is not responding to some shutdown signal.
>
> In the first crash there was no qemu-dm process left from what I
> remember. I will see if it happens again.

I see the patches were already commited. Thanks.

After more investigation my config file has on_crash="preserve".
To me it looks like the guest kills itself, since nothing is in the
logs. So after all thats not a waitqueue issue, and most likely also not
a paging bug.

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

One more thing:

Is the BUG_ON in destroy_waitqueue_vcpu really required? If the vcpu
happens to be in a queue by the time xl destroy is called the hypervisor
will crash.

Perhaps there should be some sort of domain destructor for each
waitqueue?

Olaf

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 25/11/2011 18:26, "Olaf Hering" <olaf@aepfle.de> wrote:

>
> One more thing:
>
> Is the BUG_ON in destroy_waitqueue_vcpu really required? If the vcpu
> happens to be in a queue by the time xl destroy is called the hypervisor
> will crash.

We could fix this by having waitqueues that contain a vcpu hold a reference
to that vcpu's domain.

> Perhaps there should be some sort of domain destructor for each
> waitqueue?

Not sure what you mean.

-- Keir

> Olaf



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel