Mailing List Archive

> I hope my current problems with Xen 2.0.5 are related to my own
> stupidity - otherwise I'm getting the impression that Xen is not ready
> for productional use at the current stage.

It's a project that's still evolving rapidly - particularly in terms of the
installation and management toolkit - however lots of people are using it in
production settings already. The community sites at xensource.com run in Xen
domains, so we do eat our own dogfood :-)

Performance is one of Xen's key focus points. Rest assured that the problems
you're seeing are not normal - hopefully we can get you up and running.

> Everything seems to work - all services are there and working so the
> system itself is working ok. But: when accessing resources like squirrel
> webmail or accessing the IMAP service the machine is very slow (compared
> to old setup). The load average climbs easily to values above 6, 12 was
> the highest value that I got when playing with a single client. Old
> system was never used in a way that a lot CPU power was needed, load avg
> stays there normally below 0.05-0.1 -- on the Xen setup the load never
> dropped below 2.5 (!!).

First check: Can you verify that IDE DMA is working on your system? If not,
you'll probably need to build a new kernel with the appropriate drivers.

If that doesn't help:
* what sort of workload are you imposing in your tests?
* do you know where the CPU time is going in the domain?
* does the output of top and vmstat look weird?
* anything strange in the dmesg output (try comparing to the dmesg from a
native boot)?

> Together with the still unsolved question about ballooning the memory
> this caused my impression that I shouldn't use Xen in production
> environment (see also my message from 06/28/2005 03:47 'Can't increase
> memory size during runtime' or 'problem to balloon memory beyond initial
> allocation' from Xuehai Zhang (07/07/2005 06:30). This is really sad
> because I really like Xen and WOULD like to use it...

The memory ballooning needs to be fixed. I can't remember what release the
reported breakage was in - it's probably an easy fix either way. The break
probably went unnoticed because people usually don't balloon domains above
their initial allocation.

HTH,
Mark

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hi Mark,

thanks for the reply!

> First check: Can you verify that IDE DMA is working on your system? If not,
> you'll probably need to build a new kernel with the appropriate drivers.

Yes, DMA is fine!

> If that doesn't help:

I'll come back to this and will some more test this afternoon. I will
post here...

> The memory ballooning needs to be fixed. I can't remember what release the
> reported breakage was in - it's probably an easy fix either way. The break
> probably went unnoticed because people usually don't balloon domains above
> their initial allocation.

That's great news! Not that it's broken but that it's a known problem ;)
I can live easily with the current and wait for a fix in the future...

Regards,
Patrick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

<snip>

> When I had a closer look to the mem line:
> Mem: 257488K total, 255368K used, 2120K free, 128K buffers
>
> ...it looked strange to me. I checked and found my config file for this
> domain looking like this:
>
> ...
> memory = 128
> extra = "mem=268435456"
> ...

Well spotted!

> This was left over from my tries to get the "xm balloon" working. I
> removed the extra line from the config file - and voila, it works... Is
> this config above wrong at all or is the strange behaviour I found
> related to the bugs with balloon feature..?

The config line looks OK at first glance but... I wonder what units the mem
parameter expects: if it expects KB then you've just caused it to allocate a
frame table for an absolutely *huge* physical memory, which would account for
all the real memory being allocated.

I can't guess any better without going through the code - it should hopefully
become apparent when someone fixes the ballooning problem (does anyone want
to volunteer for that, btw? It'd be useful if someone could post a patch.).

Cheers,
Mark

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Patrick,
Nice to know ballooning domains above their initial allocation is a known problem to fix. I also
forwarded my initial post here to xen devel list and Keir replied my email. You may find the
discussion at http://lists.xensource.com/archives/html/xen-devel/2005-07/msg00352.html.
Xuehai

Patrick Hess wrote:

>> The memory ballooning needs to be fixed. I can't remember what
>> release the reported breakage was in - it's probably an easy fix
>> either way. The break probably went unnoticed because people usually
>> don't balloon domains above their initial allocation.
>
>
> That's great news! Not that it's broken but that it's a known problem ;)
> I can live easily with the current and wait for a fix in the future...
>
> Regards,
> Patrick
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>


_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

Hello again,

I just had some time to check some more things...

> If that doesn't help:
> * what sort of workload are you imposing in your tests?

I just used thunderbird to access my IMAP folders and used a single
browser to access squirrel webmail (also using same IMAP).

> * do you know where the CPU time is going in the domain?

No idea :) An "xm list" on the dom0 didn't showed much seconds for the
"slow" machine as I expected when I saw this load avg.

> * does the output of top and vmstat look weird?

Yes :) See below...

> * anything strange in the dmesg output (try comparing to the dmesg from a
> native boot)?

No, that seems to be fine.
vmstat does not work:

root@pbone:~# vmstat
Unknown HZ value! (39) Assume 100.
procs memory swap io system
cpu
r b w swpd free buff cache si so bi bo in cs us
sy id
Segmentation fault

...and top looks like this after a minute running all services:

19:25:10 up 1 day, 6:10, 1 user, load average: 14.41, 5.82, 2.75
137 processes: 135 sleeping, 2 running, 0 zombie, 0 stopped
CPU states: 0.8% user, 99.2% system, 0.0% nice, 0.0% idle
Mem: 257488K total, 255368K used, 2120K free, 128K buffers
Swap: 0K total, 0K used, 0K free, 2440K cached
PID USER PRI NI SIZE RSS SHARE STAT %CPU %MEM TIME COMMAND
2567 list 18 0 3716 1004 72 D 0.3 0.3 0:00 python
2582 root 18 0 1824 400 144 R 0.3 0.1 0:00 top
2567 list 18 0 3664 1284 392 D 0.4 0.4 0:00 python
8 root 15 0 0 0 0 DW 0.2 0.0 0:00 kswapd0
2576 root 18 0 87092 5520 324 D 0.2 2.1 0:00 apache
2577 root 18 0 87092 5520 324 D 0.2 2.1 0:00 apache
2578 root 18 0 87092 5520 324 D 0.2 2.1 0:00 apache
2580 root 18 0 87092 5520 324 D 0.2 2.1 0:00 apache
2583 root 17 0 1760 656 524 D 0.2 0.2 0:00 cron


When I had a closer look to the mem line:
Mem: 257488K total, 255368K used, 2120K free, 128K buffers

...it looked strange to me. I checked and found my config file for this
domain looking like this:

...
memory = 128
extra = "mem=268435456"
...

This was left over from my tries to get the "xm balloon" working. I
removed the extra line from the config file - and voila, it works... Is
this config above wrong at all or is the strange behaviour I found
related to the bugs with balloon feature..?

Regards,
Patrick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Thu, 14 Jul 2005, Mark Williamson wrote:

> <snip>
>
> > When I had a closer look to the mem line:
> > Mem: 257488K total, 255368K used, 2120K free, 128K buffers
> >
> > ...it looked strange to me. I checked and found my config file for this
> > domain looking like this:
> >
> > ...
> > memory = 128
> > extra = "mem=268435456"
> > ...
>
> Well spotted!
>
> > This was left over from my tries to get the "xm balloon" working. I
> > removed the extra line from the config file - and voila, it works... Is
> > this config above wrong at all or is the strange behaviour I found
> > related to the bugs with balloon feature..?
>
> The config line looks OK at first glance but... I wonder what units the mem
> parameter expects: if it expects KB then you've just caused it to allocate a
> frame table for an absolutely *huge* physical memory, which would account for
> all the real memory being allocated.

My experiment shows changing "mem=268435456" to "mem=256m" doesn't
help with the ballooning above initial allocation problem.
(Linux BootPrompt-Howto
(http://en.tldp.org/HOWTO/BootPrompt-HOWTO-3.html#ss3.3)
suggests both "k" and "m" suffixes can be used for "mem=" argument).

BTW, i still think "mem=268435456" is a valid configuration to give memory
in bytes. Both "dmesg" and "free" outputs form the domain booted with this
configuration show the domain is booted with 256MB memory.


> I can't guess any better without going through the code - it should hopefully
> become apparent when someone fixes the ballooning problem (does anyone want
> to volunteer for that, btw? It'd be useful if someone could post a patch.).
>
> Cheers,
> Mark
>
> _______________________________________________
> Xen-users mailing list
> Xen-users@lists.xensource.com
> http://lists.xensource.com/xen-users
>
>

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

> My experiment shows changing "mem=268435456" to "mem=256m" doesn't
> help with the ballooning above initial allocation problem.
> (Linux BootPrompt-Howto
> (http://en.tldp.org/HOWTO/BootPrompt-HOWTO-3.html#ss3.3)
> suggests both "k" and "m" suffixes can be used for "mem=" argument).

Yep, that's the syntax I would expect for a kernel arg: I don't think this is
responsible for the ballooning problem though - that error must be somewhere
else in the code.

> BTW, i still think "mem=268435456" is a valid configuration to give memory
> in bytes. Both "dmesg" and "free" outputs form the domain booted with this
> configuration show the domain is booted with 256MB memory.

I meant the issue Patrick saw: he had the right amount of memory but none of
it was free. I was suspicious that it might all be taken up by kernel data
structures.

Did you see lots of memory allocated out of that 256M?

Cheers,
Mark

> > I can't guess any better without going through the code - it should
> > hopefully become apparent when someone fixes the ballooning problem (does
> > anyone want to volunteer for that, btw? It'd be useful if someone could
> > post a patch.).
> >
> > Cheers,
> > Mark
> >
> > _______________________________________________
> > Xen-users mailing list
> > Xen-users@lists.xensource.com
> > http://lists.xensource.com/xen-users

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On Fri, 15 Jul 2005, Mark Williamson wrote:

> > My experiment shows changing "mem=268435456" to "mem=256m" doesn't
> > help with the ballooning above initial allocation problem.
> > (Linux BootPrompt-Howto
> > (http://en.tldp.org/HOWTO/BootPrompt-HOWTO-3.html#ss3.3)
> > suggests both "k" and "m" suffixes can be used for "mem=" argument).
>
> Yep, that's the syntax I would expect for a kernel arg: I don't think this is
> responsible for the ballooning problem though - that error must be somewhere
> else in the code.
>
> > BTW, i still think "mem=268435456" is a valid configuration to give memory
> > in bytes. Both "dmesg" and "free" outputs form the domain booted with this
> > configuration show the domain is booted with 256MB memory.
>
> I meant the issue Patrick saw: he had the right amount of memory but none of
> it was free. I was suspicious that it might all be taken up by kernel data
> structures.
>
> Did you see lots of memory allocated out of that 256M?

The following is the memory allocation result (commands run from inside of
the domain).

$ dmesg | grep mem
Kernel command line:
ip=128.135.164.83:1.2.3.4:128.135.164.65:255.255.255.192::eth0:off
root=/dev/sda1 ro mem=256m maxmem=256m
vmalloc area: d0800000-fbff9000, maxmem 34000000
Memory: 125376k/262144k available (1630k kernel code, 5324k reserved, 440k
data, 112k init, 0k highmem)
xen_mem: Initialising balloon driver.
Freeing unused kernel memory: 112k freed

$ free
total used free shared buffers cached
Mem: 256932 147452 109480 0 3072 7920
-/+ buffers/cache: 136460 120472
Swap: 204792 0 204792



_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/14/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> <snip>
>
> > When I had a closer look to the mem line:
> > Mem: 257488K total, 255368K used, 2120K free, 128K buffers
> >
> > ...it looked strange to me. I checked and found my config file for this
> > domain looking like this:
> >
> > ...
> > memory = 128
> > extra = "mem=268435456"
> > ...
>
> Well spotted!
>
> > This was left over from my tries to get the "xm balloon" working. I
> > removed the extra line from the config file - and voila, it works... Is
> > this config above wrong at all or is the strange behaviour I found
> > related to the bugs with balloon feature..?
>
> The config line looks OK at first glance but... I wonder what units the mem
> parameter expects: if it expects KB then you've just caused it to allocate a
> frame table for an absolutely *huge* physical memory, which would account for
> all the real memory being allocated.
>

in domain config file, you can give memory parameters this way:
- <number>: memory unit is Byte
- <number>{k.K}: memory unit is Kilobyte
- <number>{m.M}: memory unit is Megabyte
- <number>{g.G}: memory unit is Gigabyte


> I can't guess any better without going through the code - it should hopefully
> become apparent when someone fixes the ballooning problem (does anyone want
> to volunteer for that, btw? It'd be useful if someone could post a patch.).

i will take a look, if nobody does it before me ;-)

cheers,
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/15/05, aq <aquynh@gmail.com> wrote:
> On 7/14/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > <snip>
> >
> > > When I had a closer look to the mem line:
> > > Mem: 257488K total, 255368K used, 2120K free, 128K buffers
> > >
> > > ...it looked strange to me. I checked and found my config file for this
> > > domain looking like this:
> > >
> > > ...
> > > memory = 128
> > > extra = "mem=268435456"
> > > ...
> >
> > Well spotted!
> >
> > > This was left over from my tries to get the "xm balloon" working. I
> > > removed the extra line from the config file - and voila, it works... Is
> > > this config above wrong at all or is the strange behaviour I found
> > > related to the bugs with balloon feature..?
> >
> > The config line looks OK at first glance but... I wonder what units the mem
> > parameter expects: if it expects KB then you've just caused it to allocate a
> > frame table for an absolutely *huge* physical memory, which would account for
> > all the real memory being allocated.
> >
>
> in domain config file, you can give memory parameters this way:
> - <number>: memory unit is Byte
> - <number>{k.K}: memory unit is Kilobyte
> - <number>{m.M}: memory unit is Megabyte
> - <number>{g.G}: memory unit is Gigabyte
>

oops, i took another look, and looks like the default memory unit is
Kilobyte, if you dont specify. if so, it is impossible to give the
maxmem memory in Byte unit, and the configuration above is probably
*invalid* because that amount of memory is way too much.


regards,
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/15/05, aq <aquynh@gmail.com> wrote:
> On 7/15/05, aq <aquynh@gmail.com> wrote:
> > On 7/14/05, Mark Williamson <mark.williamson@cl.cam.ac.uk> wrote:
> > > <snip>
> > >
> > > > When I had a closer look to the mem line:
> > > > Mem: 257488K total, 255368K used, 2120K free, 128K buffers
> > > >
> > > > ...it looked strange to me. I checked and found my config file for this
> > > > domain looking like this:
> > > >
> > > > ...
> > > > memory = 128
> > > > extra = "mem=268435456"
> > > > ...
> > >
> > > Well spotted!
> > >
> > > > This was left over from my tries to get the "xm balloon" working. I
> > > > removed the extra line from the config file - and voila, it works... Is
> > > > this config above wrong at all or is the strange behaviour I found
> > > > related to the bugs with balloon feature..?
> > >
> > > The config line looks OK at first glance but... I wonder what units the mem
> > > parameter expects: if it expects KB then you've just caused it to allocate a
> > > frame table for an absolutely *huge* physical memory, which would account for
> > > all the real memory being allocated.
> > >
> >
> > in domain config file, you can give memory parameters this way:
> > - <number>: memory unit is Byte
> > - <number>{k.K}: memory unit is Kilobyte
> > - <number>{m.M}: memory unit is Megabyte
> > - <number>{g.G}: memory unit is Gigabyte
> >
>
> oops, i took another look, and looks like the default memory unit is
> Kilobyte, if you dont specify. if so, it is impossible to give the
> maxmem memory in Byte unit, and the configuration above is probably
> *invalid* because that amount of memory is way too much.

hmm.. another oops :~p

the unit for programming interface is KB, but for *domain*
configuration, if you dont specify then the memory unit is MB. so you
should adjust your domain config file, and give it another shoot.


sorry for so much confuses ;-)
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

aq wrote:

> the unit for programming interface is KB, but for *domain*
> configuration, if you dont specify then the memory unit is MB. so you
> should adjust your domain config file, and give it another shoot.

In the config file I cannot use any k|K|m|M|... - only numeric values in
MB seem to be allowed here:

nd:/etc/xen# cat dev2.xcfg
kernel ="/boot/xen-linux-2.6.10nwxu"
memory = 64M
[...more stuff...]
nd:/etc/xen# xm create -c /etc/xen/dev2.xcfg
Using config file "/etc/xen/dev2.xcfg".
Traceback (most recent call last):
[...trace...]
File "/etc/xen/dev2.xcfg", line 2
memory = 64M
^
SyntaxError: invalid syntax


Regards,
Patrick

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users

On 7/15/05, Patrick Hess <ph@nwnet.de> wrote:
> aq wrote:
>
> > the unit for programming interface is KB, but for *domain*
> > configuration, if you dont specify then the memory unit is MB. so you
> > should adjust your domain config file, and give it another shoot.
>
> In the config file I cannot use any k|K|m|M|... - only numeric values in
> MB seem to be allowed here:
>
> nd:/etc/xen# cat dev2.xcfg
> kernel ="/boot/xen-linux-2.6.10nwxu"
> memory = 64M

no, dont do that. the domain config file is actually a python script,
and it will be executed by python interpreter. so "memory = 64M" is
not syntax-valid, and you will get an error like you saw.

there are some confuses: in domain config file, all memory unit is MB,
and you dont need to specify the memory unit. but if you config the
domain using xm interface (for example "xm maxmem DOM MEM" then you
have few choices when specifying MEM (ie you can use K/M/G suffix on
memory parameter)

so "memory = 64" is fine.

regards,
aq

_______________________________________________
Xen-users mailing list
Xen-users@lists.xensource.com
http://lists.xensource.com/xen-users