Mailing List Archive

xend crash/hang scenario
I inadvertently found a way to crash/hang xend and almost completely lock
oneself out of a domain. I was testing a new domain, and without thinking I
did 'cat /dev/urandom' (rather than head or tail, oopsie) to make sure it was
functioning. The resulting flood of output pegged xend at 99% cpu, where it
stayed for 10 minutes before I killed it off. The real issue came after a
restart, when I attempted to console back into the running domain. Either I
had not managed to ctrl+c the `cat` prior to hanging xend or the output
buffer was so full that it continued to spew garbage at me when I would
restart xend and console in (I'm guessing). I got back on the the console a
minute or so later by either managing to ctrl+c at the right time, or the
output buffer finally emptied.

This specific situation is not so scary since only people who didn't sleep
enough *cough* would be cat-ing /dev/urandom, but I found that the event is
not limited to something that silly. It seems that any large output of
binary to the console will hang xend. I can reproduce the exact situation
with "cat /usr/bin/vim" or any other decent-sized binary. Xend managed to
recover from cat-ing binaries up to about 1.5mb, but anything larger than 2mb
had xend using 99% cpu anywhere from 5 minutes to indefinitely (longer than I
could wait anyway). Memory usage during the hangs does not go up appreciably.
Strategic ctrl+c gets me back to console with relative ease, but it still
requires a xend restart (sometimes 2 or 3) before I can get back in.

In contrast to the binary output, even extremely large text files (500mb+)
appear to have no ill effect, even when cat'd in a while loop.

I have not tried outside of my setup so I don't know if it's reproducable, but
it is very concerning to me that any user in any domain could, with a typo,
crush the CPU and take out all management capability as well. I'll provide
any additional info if it will help.

Running Xen 2.0.3
Hardware: Dual Athlon XP 1900+ (1.6g) 512mb RAM
Domain0: Gentoo 2004.3 running 2.6.10-xen0 128mb RAM allocated CPU0
DomainU: Gentoo 2004.3 2.6.10-xenU / default config / 128mb RAM allocated CPU1

--
+--------------------------------+
Mike Culbertson
Senior Systems Administrator
DiD : 646-230-8752/(M)646-382-4779
Mail: mike@infoleak.com
+--------------------------------+


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
RE: xend crash/hang scenario [ In reply to ]
> It seems that any large
> output of
> binary to the console will hang xend. I can reproduce the
> exact situation
> with "cat /usr/bin/vim" or any other decent-sized binary.
> Xend managed to
> recover from cat-ing binaries up to about 1.5mb, but anything
> larger than 2mb
> had xend using 99% cpu anywhere from 5 minutes to
> indefinitely (longer than I
> could wait anyway).

Yep, fixing this has been on the todo list for a while.
I'm surprised its taken someone so long to spot it :-)

We need to limit the rate at which xend is prepared to take message off
a given control ring. There'll then be back pressure to the domain,
which can either choose to block or throw away messages.

> In contrast to the binary output, even extremely large text
> files (500mb+)
> appear to have no ill effect, even when cat'd in a while loop.

I think you're just getting lucky. At least I can't think why there'd be
a difference between binary and text, unless xend does some scanning of
the console messages I'm not aware of.

> I have not tried outside of my setup so I don't know if it's
> reproducable, but
> it is very concerning to me that any user in any domain
> could, with a typo,
> crush the CPU and take out all management capability as well.
> I'll provide
> any additional info if it will help.

Yep, this is serious and needs fixing. Volunteers?

Thanks,
Ian


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel
Re: xend crash/hang scenario [ In reply to ]
Hi,

On Thu, Jan 27, 2005 at 02:19:35AM -0800, Mike Culbertson wrote:
> with "cat /usr/bin/vim" or any other decent-sized binary. Xend managed to
> recover from cat-ing binaries up to about 1.5mb, but anything larger than 2mb
> had xend using 99% cpu anywhere from 5 minutes to indefinitely (longer than I
> could wait anyway). Memory usage during the hangs does not go up appreciably.
> Strategic ctrl+c gets me back to console with relative ease, but it still
> requires a xend restart (sometimes 2 or 3) before I can get back in.

For the record, I have seen the same thing too. Running a simple "yes"
command in a domain's console has the same effect.

Gabor


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xen-devel