Mailing List Archive

RE: [Xen-devel] RE: Why my system only use 3.3G memory ?
It is certainly not a bug as such. It's a "Missing feature". Xen 2.0 does not support PAE (Page Address Extension). The PAE extension allows the processor to address memory beyond 4GB when accessing memory through the paged memory management unit (MMU or PMMU depending on who you talk to). This means that although the processor has a virtual limit of 4GB, it can map memory from beyond 4GB to map that into the virtual address space of the kernel. However, without PAE, the processor can only access 4GB and that's it, anything beyond that is outside it's reach.

The other part of the problem is that the BIOS maps PCI devices into the first 4GB of memory. Depending on how clever (or not) the BIOS is, it may eat more memory than it MUST. But there are several factors to take into account here. For example, some devices that require large address spaces will often also require that the device is evenly aligned to it's mapped size, so something that has a 64MB address space, should also be aligned to 64MB. This means that there may well be a hole of almost 64MB after this map. Some devices, such as graphics cards, can easily use up 256 or 512MB this way. [.Of course, if the BIOS is clever, it would use the remaining part for some other mappings, but it's not always that the BIOS is clever enough to do this].

So, your 700 or so MB that are "missing" is caused by the PCI mappings. Generally speaking, this space is "non-RAM", so any access to the unused areas in this space will just be lost.

You can work around this by running Xen 3.0, but the Xen 3.0 is yet to be released, so it's not suitable for production environments. It does, however, support both 64-bit and PAE addressing of the memory, depending on the type of processor you have.

Obviously, even with PAE, you will not be able to use ALL of the 4GB, because some of it will be mapped to the PCI hardware devices. But any "hole" between devices in the PCI map can be mapped in as RAM, so the kernel could possibly use up the gaps in the space with memory from above 4GB, rather than just "loosing" it.

I hope this clarifies the situation.

--
Mats

________________________________

From: xen-devel-bounces@lists.xensource.com [mailto:xen-devel-bounces@lists.xensource.com] On Behalf Of she11c0de she11c0de
Sent: 07 July 2005 11:00
To: m+Ian.Pratt@cl.cam.ac.uk; xen-users@lists.xensource.com
Cc: xen-devel@lists.xensource.com
Subject: [Xen-devel] RE: [Xen-users] Why my system only use 3.3G memory ?




I think it's maybe a bug, because if Xen 2.0 has 4G memory limit, even my Dell 2850 server has 6G memory , Xen 2.0 would use 4G memory on it, but it's just use 3327MB memory.

I have another server Dell 2650 with 4G memory dual Xeon 3.0G CPU, had install Redhat 9.0 and Xen 2.0, it's not have any problem, i has running 8 virtual server on it.

Here is my Dell2650 XenLinux system info.

[tongjian@XenServer43 11:59:19 ~ 350]$ xm info
system : Linux
host : XenServer43.testgroups.sina.com.cn
release : 2.6.9-xen0
version : #4 Fri Dec 31 19:09:27 CST 2004
machine : i686
cores : 2
hyperthreads_per_core : 2
cpu_mhz : 3052
memory : 3967
free_memory : 1757


[tongjian@XenServer43 17:35:18 ~ 351]$ xm dmesg | less
ERROR: cannot use unconfigured serial port COM1
__ __ ____ ___
\ \/ /___ _ __ |___ \ / _ \
\ // _ \ '_ \ __) || | | |
/ \ __/ | | | / __/ | |_| |
/_/\_\___|_| |_| |_____(_)___/

http://www.cl.cam.ac.uk/netos/xen
University of Cambridge Computer Laboratory

Xen version 2.0 (root@(none)) (gcc version 3.2.2 20030222 (Red Hat Linux 3.2.2-5)) Fri Dec 31 19:16:41 CST 2004
Latest ChangeSet: information unavailable

(XEN) Initialised 3967MB memory (1015776 pages) on a 3967MB machine
(XEN) Xen heap size is 10684KB
(XEN) CPU0: Before vendor init, caps: bfebfbff 00000000 00000000, vendor = 0
(XEN) CPU#0: Physical ID: 0, Logical ID: 0
(XEN) CPU caps: bfebfbff 00000000 00000000 00000000
(XEN) found SMP MP-table at 000fe710
(XEN) Memory Reservation 0xfe710, 4096 bytes
(XEN) Memory Reservation 0xf0000, 4096 bytes
(XEN) ACPI: RSDP (v000 DELL ) @ 0x000fdc40
(XEN) ACPI: RSDT (v001 DELL PE2650 0x00000001 MSFT 0x0100000a) @ 0x000fdc54
(XEN) ACPI: FADT (v001 DELL PE2650 0x00000001 MSFT 0x0100000a) @ 0x000fdc84
(XEN) ACPI: MADT (v001 DELL PE2650 0x00000001 MSFT 0x0100000a) @ 0x000fdcf8
(XEN) ACPI: SPCR (v001 DELL PE2650 0x00000001 MSFT 0x0100000a) @ 0x000fdd80
(XEN) ACPI: DSDT (v001 DELL PE2650 0x00000001 MSFT 0x0100000a) @ 0x00000000
(XEN) ACPI: Local APIC address 0xfee00000
(XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
(XEN) Processor #0 Pentium 4(tm) XEON(tm) APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
(XEN) Processor #6 Pentium 4(tm) XEON(tm) APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
(XEN) Processor #1 Pentium 4(tm) XEON(tm) APIC version 20
(XEN) ACPI: LAPIC (acpi_id[0x04] lapic_id[0x07] enabled)
(XEN) Processor #7 Pentium 4(tm) XEON(tm) APIC version 20











>From: "Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk>
>To: "she11c0de she11c0de" <she11c0de@hotmail.com>,<xen-users@lists.xensource.com>
>CC: <xen-devel@lists.xensource.com>,<ian.pratt@cl.cam.ac.uk>
>Subject: RE: [Xen-users] Why my system only use 3.3G memory ?
>Date: Thu, 7 Jul 2005 09:05:01 +0100
>
>(XEN) Physical RAM map:
>(XEN) 0000000000000000 - 00000000000a0000 (usable)
>(XEN) 0000000000100000 - 00000000cffc0000 (usable)
>(XEN) 00000000cffc0000 - 00000000cffcfc00 (ACPI data)
>(XEN) 00000000cffcfc00 - 00000000cffff000 (reserved)
>(XEN) 00000000e0000000 - 00000000fec90000 (reserved)
>(XEN) 00000000fed00000 - 00000000fed00400 (reserved)
>(XEN) 00000000fee00000 - 00000000fee10000 (reserved)
>(XEN) 00000000ffb00000 - 0000000100000000 (reserved)
>(XEN) System RAM: 3327MB (3407232kB)
>
>Your BIOS has created a whole at the top of the memory map into which the memory mapped PCI devices have been placed, effectively hiding the memory from use.
>
>What is slightly odd is that the whole is larger than I've seen before. I presume its the area between cffff000 and e0000000 which isn't listed in the e820 map (everything above cffc0000 is unusable).
>
>You'll have to wait until xen3 and use x86_32pae or x86_64 to use more memory.
>
>Ian
>
> > -----Original Message-----
> > From: xen-users-bounces@lists.xensource.com
> > [mailto:xen-users-bounces@lists.xensource.com] On Behalf Of
> > she11c0de she11c0de
> > Sent: 07 July 2005 06:50
> > To: xen-users@lists.xensource.com
> > Subject: [Xen-users] Why my system only use 3.3G memory ?
> >
> > Hi, all
> >
> > I have a server used to install Xen-2.0.6 on CentOS 4.1 i386,
> > hardware is DELL 2850, dual Xeon 3.2G CPU, 6G memory, use
> > myself custom compiled xen kernel(dom0 and domU), i know xen
> > 2.0 on i386 only use 4G memory, but i found on my server,
> > it's only use 3327M memory, whit's problem, who can help me?
> >
> > Here is some info.
> >
> > [root@CentOS4 ~]# xm info
> > system : Linux
> > host : CentOS4
> > release : 2.6.11.10-xen0
> > version : #3 Fri Jul 1 02:54:48 CST 2005
> > machine : i686
> > cores : 2
> > hyperthreads_per_core : 2
> > cpu_mhz : 3192
> > memory : 3327
> > free_memory : 2913
> >
> >
> > [root@CentOS4 ~]# xm dmesg | less
> > ERROR: cannot use unconfigured serial port COM1
> > __ __ ____ ___ __
> > \ \/ /___ _ __ |___ \ / _ \ / /_
> > \ // _ \ '_ \ __) || | | | '_ \
> > / \ __/ | | | / __/ | |_| | (_) |
> > /_/\_\___|_| |_| |_____(_)___(_)___/
> >
> > http://www.cl.cam.ac.uk/netos/xen
> > University of Cambridge Computer Laboratory
> >
> > Xen version 2.0.6 (tongjian@(none)) (gcc version 3.4.3
> > 20050227 (Red Hat 3.4.3-22.1)) Fri Jul 1 02:53:42 CST 2005
> > Latest ChangeSet: information unavailable
> >
> > (XEN) WARNING: Only the first 4GB of the physical memory map
> > can be accessed
> > (XEN) by Xen in 32-bit mode. Truncating the memory map...
> > (XEN) Physical RAM map:
> > (XEN) 0000000000000000 - 00000000000a0000 (usable)
> > (XEN) 0000000000100000 - 00000000cffc0000 (usable)
> > (XEN) 00000000cffc0000 - 00000000cffcfc00 (ACPI data)
> > (XEN) 00000000cffcfc00 - 00000000cffff000 (reserved)
> > (XEN) 00000000e0000000 - 00000000fec90000 (reserved)
> > (XEN) 00000000fed00000 - 00000000fed00400 (reserved)
> > (XEN) 00000000fee00000 - 00000000fee10000 (reserved)
> > (XEN) 00000000ffb00000 - 0000000100000000 (reserved)
> > (XEN) System RAM: 3327MB (3407232kB)
> > (XEN) Xen heap: 10MB (10680kB)
> > (XEN) CPU0: Before vendor init, caps: bfebfbff 20100000
> > 00000000, vendor = 0
> > (XEN) CPU#0: Physical ID: 0, Logical ID: 0
> > (XEN) CPU caps: bfebfbff 20100000 00000000 00000000
> > (XEN) found SMP MP-table at 000fe710
> > (XEN) ACPI: RSDP (v000 DELL
> > ) @ 0x000fd650
> > (XEN) ACPI: RSDT (v001 DELL PE BKC 0x00000001 MSFT
> > 0x0100000a) @ 0x000fd664
> > (XEN) ACPI: FADT (v001 DELL PE BKC 0x00000001 MSFT
> > 0x0100000a) @ 0x000fd6b0
> > (XEN) ACPI: MADT (v001 DELL PE BKC 0x00000001 MSFT
> > 0x0100000a) @ 0x000fd724
> > (XEN) ACPI: SPCR (v001 DELL PE BKC 0x00000001 MSFT
> > 0x0100000a) @ 0x000fd7cc
> > (XEN) ACPI: HPET (v001 DELL PE BKC 0x00000001 MSFT
> > 0x0100000a) @ 0x000fd81c
> > (XEN) ACPI: MCFG (v001 DELL PE BKC 0x00000001 MSFT
> > 0x0100000a) @ 0x000fd854
> > (XEN) ACPI: DSDT (v001 DELL PE BKC 0x00000001 MSFT
> > 0x0100000e) @ 0x00000000
> > (XEN) ACPI: Local APIC address 0xfee00000
> > (XEN) ACPI: LAPIC (acpi_id[0x01] lapic_id[0x00] enabled)
> > (XEN) Processor #0 Unknown CPU [15:4] APIC version 20
> > (XEN) ACPI: LAPIC (acpi_id[0x02] lapic_id[0x06] enabled)
> > (XEN) Processor #6 Unknown CPU [15:4] APIC version 20
> > (XEN) ACPI: LAPIC (acpi_id[0x03] lapic_id[0x01] enabled)
> > (XEN) Processor #1 Unknown CPU [15:4] APIC version 20
> > [root@CentOS4 ~]#
> >
> >
> >
> >
> >
> >
> > ________________________________
> >
> > 浣跨敤 MSN Messenger <http://g.msn.com/8HMBCN/2734??PS=47575> 涓庤仈鏈虹殑鏈嬪弸杩涜浜ゆ祦
> >


________________________________

使用 MSN Messenger <http://g.msn.com/8HMACN/2737??PS=47575> 与联机的朋友进行交流
Re: [Xen-devel] RE: Why my system only use 3.3G memory ? [ In reply to ]
On 8 Jul 2005, at 09:58, Petersson, Mats wrote:

> You can work around this by running Xen 3.0, but the Xen 3.0 is yet to
> be released, so it's not suitable for production environments. It
> does, however, support both 64-bit and PAE addressing of the memory,
> depending on the type of processor you have.
>
> Obviously, even with PAE, you will not be able to use ALL of the 4GB,
> because some of it will be mapped to the PCI hardware devices. But any
> "hole" between devices in the PCI map can be mapped in as RAM, so the
> kernel could possibly use up the gaps in the space with memory from
> above 4GB, rather than just "loosing" it.

The PAE support isn't fully baked yet anyway. We're waiting on fixed-up
PAE patches for XenLinux from Gerd Knorr. Should be happening 'real
soon now'. ;-)

-- Keir


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel
RE: [Xen-devel] RE: Why my system only use 3.3G memory ? [ In reply to ]
On Fri, 8 Jul 2005, Petersson, Mats wrote:

> The other part of the problem is that the BIOS maps PCI devices into the
> first 4GB of memory. Depending on how clever (or not) the BIOS is, it
> may eat more memory than it MUST. But there are several factors to take
> into account here. For example, some devices that require large address
> spaces will often also require that the device is evenly aligned to it's
> mapped size, so something that has a 64MB address space, should also be
> aligned to 64MB.

This is actually a PCI requirement. The address has to be aligned on its
granularity. Really silly.

>This means that there may well be a hole of almost 64MB
> after this map. Some devices, such as graphics cards, can easily use up
> 256 or 512MB this way. [.Of course, if the BIOS is clever, it would use
> the remaining part for some other mappings, but it's not always that the
> BIOS is clever enough to do this].

BIOSes are generally dumb as dirt on this score. LinuxBIOS is pretty smart
about it, however.

> Obviously, even with PAE, you will not be able to use ALL of the 4GB,
> because some of it will be mapped to the PCI hardware devices. But any
> "hole" between devices in the PCI map can be mapped in as RAM, so the
> kernel could possibly use up the gaps in the space with memory from
> above 4GB, rather than just "loosing" it.

it's more complex than that. Some chipsets allow remapping the ram that is
"lost" to pci mappings up above 4 GB. There's no end of weird tricks in
this space.

ron

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel