Mailing List Archive: Interrupt Coalescence

Interrupt Coalescence

Nov 11, 2004, 10:13 AM

Post #1 of 4 (283 views)

I've found that Interrupt Coalescence (IC) has a detrimental effect on some benchmarks with the e1000 driver. So I tried disabling it with the argument "InterruptThrottleRate=0,0".

This had the wanted effect in native linux, ie. full speed, 116 MB/s. However, in Xen0, disabling IC has the effect of severely reducing bandwidth down to around 1 MB/s.

Turning of IC has the effect that an interrupt is generated from the NIC for each received or sent MTU (1500 bytes). On a gigabit network this means that an interrupt may be generated, according to [http://www.pam2004.org/papers/265.pdf], every 12 us.

I looked at #interrupts generated with and without IC, and they are ~ 1100 and ~ 400 respectively when sending a 2 MB message with ttcp.

So I'm trying to explain what's happening. Is Xen slower in handling the interrupts than native? The CPU is an Intel P4 @ 3400 MHz. Does it need more than 12 us to do a context switch when using Xen?

Håvard

-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

RE: Interrupt Coalescence [ In reply to ]

m+Ian.Pratt at cl

Nov 12, 2004, 1:33 AM

Post #2 of 4 (284 views)

Permalink

> I've found that Interrupt Coalescence (IC) has a detrimental
> effect on some benchmarks with the e1000 driver. So I tried
> disabling it with the argument "InterruptThrottleRate=0,0".
>
> This had the wanted effect in native linux, ie. full speed,
> 116 MB/s. However, in Xen0, disabling IC has the effect of
> severely reducing bandwidth down to around 1 MB/s.

I'd like to get to the bottom of this. A couple of people have reported odd performance anomalies with e1000's, but ours seem to work just fine.

Are you results measured from dom0 or from another domain? If the latter, what's the CPU assignment?

What happenes if you boot with 'igrnorebiostables' on the Xen command line (this forces uniprocessor)? This will use the legacy PIC rather than the IOAPIC.

The dramatic reduction in bandwidth you're seeing with coallescing on is probably our best chance of tracking down this bug (everyone else was seeing more subtle effects).

Ian

> Turning of IC has the effect that an interrupt is generated
> from the NIC for each received or sent MTU (1500 bytes). On a
> gigabit network this means that an interrupt may be
> generated, according to
> [http://www.pam2004.org/papers/265.pdf], every 12 us.
>
> I looked at #interrupts generated with and without IC, and
> they are ~ 1100 and ~ 400 respectively when sending a 2 MB
> message with ttcp.
>
> So I'm trying to explain what's happening. Is Xen slower in
> handling the interrupts than native? The CPU is an Intel P4 @
> 3400 MHz. Does it need more than 12 us to do a context switch
> when using Xen?
>
>
> Håvard
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by:
> Sybase ASE Linux Express Edition - download now for FREE
> LinuxWorld Reader's Choice Award Winner for best database on Linux.
> http://ads.osdn.com/?ad_idU88&alloc_id065&op=ick
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xen-devel
>
>

-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

Re: Interrupt Coalescence [ In reply to ]

havarbj at idi

Nov 12, 2004, 4:01 AM

Post #3 of 4 (284 views)

Permalink

On Fri, Nov 12, 2004 at 08:33:26AM -0000, Ian Pratt wrote:
> > I've found that Interrupt Coalescence (IC) has a detrimental
> > effect on some benchmarks with the e1000 driver. So I tried
> > disabling it with the argument "InterruptThrottleRate=0,0".
> >
> > This had the wanted effect in native linux, ie. full speed,
> > 116 MB/s. However, in Xen0, disabling IC has the effect of
> > severely reducing bandwidth down to around 1 MB/s.
>
> I'd like to get to the bottom of this. A couple of people have reported odd performance anomalies with e1000's, but ours seem to work just fine.
>
> Are you results measured from dom0 or from another domain? If the latter, what's the CPU assignment?
>

Single CPU using dom0

> What happenes if you boot with 'igrnorebiostables' on the Xen command line (this forces uniprocessor)? This will use the legacy PIC rather than the IOAPIC.
>

I've already found that 'ignorebiostables' solves the IOAPIC problem, giving equal performance in native and dom0.

Whether with or without 'ignorebiostables', disabling IC leads to dramatically reduced bandwidth in dom0 but not in native.

> The dramatic reduction in bandwidth you're seeing with coallescing on is probably our best chance of tracking down this bug (everyone else was seeing more subtle effects).
>
> Ian
>
> > Turning of IC has the effect that an interrupt is generated
> > from the NIC for each received or sent MTU (1500 bytes). On a
> > gigabit network this means that an interrupt may be
> > generated, according to
> > [http://www.pam2004.org/papers/265.pdf], every 12 us.
> >
> > I looked at #interrupts generated with and without IC, and
> > they are ~ 1100 and ~ 400 respectively when sending a 2 MB
> > message with ttcp.
> >
> > So I'm trying to explain what's happening. Is Xen slower in
> > handling the interrupts than native? The CPU is an Intel P4 @
> > 3400 MHz. Does it need more than 12 us to do a context switch
> > when using Xen?
> >
> >
> > Håvard
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by:
> > Sybase ASE Linux Express Edition - download now for FREE
> > LinuxWorld Reader's Choice Award Winner for best database on Linux.
> > http://ads.osdn.com/?ad_idU88&alloc_id065&op=ick
> > _______________________________________________
> > Xen-devel mailing list
> > Xen-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/xen-devel
> >
> >

-------------------------------------------------------
This SF.Net email is sponsored by:
Sybase ASE Linux Express Edition - download now for FREE
LinuxWorld Reader's Choice Award Winner for best database on Linux.
http://ads.osdn.com/?ad_idU88&alloc_id065&op=click
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel

RE: Interrupt Coalescence [ In reply to ]

m+Ian.Pratt at cl

Nov 13, 2004, 2:24 PM

Post #4 of 4 (281 views)

Permalink

> > What happenes if you boot with 'igrnorebiostables' on the
> Xen command line (this forces uniprocessor)? This will use
> the legacy PIC rather than the IOAPIC.
> >
> I've already found that 'ignorebiostables' solves the IOAPIC
> problem, giving equal performance in native and dom0.
>
> Whether with or without 'ignorebiostables', disabling IC
> leads to dramatically reduced bandwidth in dom0 but not in native.

I'm rather more interested in getting the IOAPIC working on your
platform than figuring out what's going on with the legacy PIC. Can you
say a bit about the CPU and chipset you're using?

Since it looks like there's a nasty interrupt problem on your system,
it's perhaps not surprising that disabling interrupt coallescing causes
problems. My guess is that you end up doing some nasty spinning iwithin
Xen.

Could you get a serial line on the system to enable you to readout the
Xen s/w performance counters? (hit 'p' to print, 'P' to reset)

Ian

-------------------------------------------------------
This SF.Net email is sponsored by: InterSystems CACHE
FREE OODBMS DOWNLOAD - A multidimensional database that combines
robust object and relational technologies, making it a perfect match
for Java, C++,COM, XML, ODBC and JDBC. www.intersystems.com/match8
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel