Mailing List Archive: [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug

[PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug

Feb 2, 2007, 9:49 PM

Post #1 of 5 (76 views)

Irq nunmber for per-vcpu event (virq/ipi) needs kept
accross vcpu plug/unplug, once allocated. We just
reuse this irq number and bind it to a new event port.
Or else, /proc/interrupt exports messed statistics
like:

ot@localhost ~]# cat /proc/interrupts
CPU0 CPU1 CPU2 CPU3
256: 423066 0 0 0 Dynamic-irq timer0
257: 3014 0 0 0 Dynamic-irq
resched0
258: 37 0 0 0 Dynamic-irq
callfunc0
259: 713 0 0 0 Dynamic-irq xenbus
260: 343 1323 65 50 Dynamic-irq
xencons
261: 270 62 0 0 Dynamic-irq blkif
262: 0 1053818 1220 855 Dynamic-irq
resched1
263: 0 0 2165 53 Dynamic-irq
callfunc1
264: 0 3218 69 0 Dynamic-irq timer1
265: 0 0 42768 228 Dynamic-irq
resched2
266: 0 45 0 1867 Dynamic-irq
callfunc2
267: 0 0 39543 256 Dynamic-irq timer2
268: 0 2244 46 33027 Dynamic-irq
resched3
269: 1585 0 0 0 Dynamic-irq
callfunc3
270: 11273 0 1885 1073 Dynamic-irq timer3
NMI: 0 0 0 0
LOC: 0 0 0 0
ERR: 0
MIS: 0

You can see from above that each virq/ipi have
statistics on several vcpus, when that data on
other vcpus belong to other virq/ipi before unplug.

This also applies to save/restore for same issue.

Signed-off-by Kevin Tian <kevin.tian@intel.com>

Re: [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug [ In reply to ]

Keir.Fraser at cl

Feb 3, 2007, 9:51 PM

Post #2 of 5 (77 views)

Permalink

On 3/2/07 04:49, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> Irq nunmber for per-vcpu event (virq/ipi) needs kept
> accross vcpu plug/unplug, once allocated. We just
> reuse this irq number and bind it to a new event port.
> Or else, /proc/interrupt exports messed statistics
> like:

Noone cares about absolute /proc/interrupt numbers, only rate of change. If
pushed, I would argue that the stats should be zeroed when an interrupt line
is freed (since the interrupt then stops appearing in /proc/interrupts,
which logically implies that the stats lifetime has ended, and so a reuse of
that interrupt is a new lifetime starting from zero). After all, all
irq-evtchn bindings are dynamic running on Xen: a Linux irq may
theoretically get used for all sorts of different devices during the
lifetime of the Linux guest. Should all these uses get aggregated over time?

Zeroing the stats would potentially be a patch for lkml, or we could do it
for dynirqs ourselves in unbind_from_irq(). That's a patch I would accept.

-- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug [ In reply to ]

kevin.tian at intel

Feb 4, 2007, 5:28 AM

Post #3 of 5 (76 views)

Permalink

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007Äê2ÔÂ4ÈÕ 12:52
>
>On 3/2/07 04:49, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> Irq nunmber for per-vcpu event (virq/ipi) needs kept
>> accross vcpu plug/unplug, once allocated. We just
>> reuse this irq number and bind it to a new event port.
>> Or else, /proc/interrupt exports messed statistics
>> like:
>
>Noone cares about absolute /proc/interrupt numbers, only rate
>of change. If
>pushed, I would argue that the stats should be zeroed when an
>interrupt line
>is freed (since the interrupt then stops appearing in /proc/interrupts,
>which logically implies that the stats lifetime has ended, and
>so a reuse of
>that interrupt is a new lifetime starting from zero). After all, all
>irq-evtchn bindings are dynamic running on Xen: a Linux irq may
>theoretically get used for all sorts of different devices during the
>lifetime of the Linux guest. Should all these uses get
>aggregated over time?
>
>Zeroing the stats would potentially be a patch for lkml, or we
>could do it
>for dynirqs ourselves in unbind_from_irq(). That's a patch I
>would accept.
>
> -- Keir

Basically I agree with you, but I'm not sure about the usage model
on such stats. Isn't it up to application to decide? App may decide
to do some action (like balance) by rate of change, so does the
aggregated value. If both usage models exist, do we need consider
both for compatibility to app's assumption?

Another point I'm not for sure is the save/restore and migration.
From user perspective, he has no knowledge that cpu1...N are
unplugged and then plugged along the process. Thus will anyone
feel strange about the fact that all cpus except cpu0 gets their
per-vcpu interrupt stats empty after restored or migrated? This
seems not an explicit lifetime restart...

So I still think virq/ipi are a bit special, and it's reasonable to
zero stats for the rest. :-)

Thanks,
Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug [ In reply to ]

Keir.Fraser at cl

Feb 4, 2007, 7:39 AM

Post #4 of 5 (76 views)

Permalink

On 4/2/07 12:28, "Tian, Kevin" <kevin.tian@intel.com> wrote:

> Another point I'm not for sure is the save/restore and migration.
> From user perspective, he has no knowledge that cpu1...N are
> unplugged and then plugged along the process. Thus will anyone
> feel strange about the fact that all cpus except cpu0 gets their
> per-vcpu interrupt stats empty after restored or migrated? This
> seems not an explicit lifetime restart...

...and the same is true for his network and block-device interrupts.
Currently these may come back with different IRQs too, and 'fixing' this
would be somewhat painful (e.g., it would probably need some evtchn
interface changes). Why do you consider the VIRQs/IPIs so special?

Your original point that having non-zero counts for CPU!=n for an interrupt
that is specific to CPU==n looks weird and may confuse people is a fair one.
Zeroing stats is a nice simple solution to that.

As you say /proc/interrupts is used for IRQ load balancing, where rate of
change matters. I'm not aware of any other application uses of
/proc/interrupts -- certainly none that are 'stateful' in terms of requiring
a given interrupt source to stick with its original IRQ, or require accurate
aggregate counts.

-- Keir

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

RE: [PATCH] Reuse irq number for virq/ipi after vcpu unplug/plug [ In reply to ]

kevin.tian at intel

Feb 4, 2007, 7:49 PM

Post #5 of 5 (76 views)

Permalink

So, attached patch zeros irq stats at bind_evtchn_to_cpu, since
rate of change is the real concern. By this way, we don't need to
care whether stale stats are on multiple cpus at irq unbinding.

Thanks,
Kevin.

>From: Keir Fraser [mailto:Keir.Fraser@cl.cam.ac.uk]
>Sent: 2007Äê2ÔÂ4ÈÕ 22:39
>
>On 4/2/07 12:28, "Tian, Kevin" <kevin.tian@intel.com> wrote:
>
>> Another point I'm not for sure is the save/restore and migration.
>> From user perspective, he has no knowledge that cpu1...N are
>> unplugged and then plugged along the process. Thus will anyone
>> feel strange about the fact that all cpus except cpu0 gets their
>> per-vcpu interrupt stats empty after restored or migrated? This
>> seems not an explicit lifetime restart...
>
>...and the same is true for his network and block-device interrupts.
>Currently these may come back with different IRQs too, and
>'fixing' this
>would be somewhat painful (e.g., it would probably need some evtchn
>interface changes). Why do you consider the VIRQs/IPIs so special?
>
>Your original point that having non-zero counts for CPU!=n for
>an interrupt
>that is specific to CPU==n looks weird and may confuse people
>is a fair one.
>Zeroing stats is a nice simple solution to that.
>
>As you say /proc/interrupts is used for IRQ load balancing,
>where rate of
>change matters. I'm not aware of any other application uses of
>/proc/interrupts -- certainly none that are 'stateful' in
>terms of requiring
>a given interrupt source to stick with its original IRQ, or
>require accurate
>aggregate counts.
>
> -- Keir
>

Mailing List Archive

Attached Files:

Attached Files: