Mailing List Archive

(+ Stefano)

On 25/08/2020 13:04, Charles Chiou wrote:
> Hello,

Hello,

> I'm having problem with virtual console when booting up 32 bit guests. I'm using qemu virt machine, running xen 64 bit, DOM0 Linux is 64 bit. The 64 bit guests that I've tried (Linux, an RTOS) have no issues, but when I tried to run 32 bit OS (tried linux, FreeRTOS from Galois, a bare-metal app), I get "Invalid MFN 0x33a08" or similar messages when making HVC calls such as:
>
> mov r0, #0
> mov r1, #35
> ldr r2, =banner
> mov r12, #18
> hvc #0xea1<----
>
> and get this at the HVC call to XEN:
>
> (XEN) p2m.c:1919: d8v0: Invalid MFN 0x33a08
>
> Not sure if there's misconfiguration somewhere, or compile options, etc. that caused this... Any pointer is greatly appreciated!
>
> Below is output of similar problem when booting up a 32 bit linux kernel:

Looking at the code, this seems like an issue when trying to translate a
guest virtual address to a machine address.

A few questions:
- Which QEMU version are you using?
- What's your Linux configuration? Are you using LPAE or short page
tables?

I am also not sure what's the state of virtualization support in QEMU
for 32-bit guest. I have CCed Stefano who may have a better idea.

Best regards,

> / # xl -v create -c /share/misc/linux32.cfg
> Parsing config from /share/misc/linux32.cfg
> libxl: info: libxl_create.c:122:libxl__domain_build_info_setdefault: qemu-xen is unavailable, using qemu-xen-traditional instead: No such file or directory
> libxl: detail: libxl_create.c:623:libxl__domain_make: passthrough: disabled
> domainbuilder: detail: xc_dom_allocate: cmdline="rw root=/dev/ram rdinit=/sbin/init earlyprintk=serial,ttyAMA0", features=""
> domainbuilder: detail: xc_dom_kernel_file: filename="/share/linux-domu-32bit/arch/arm/boot/zImage"
> domainbuilder: detail: xc_dom_malloc_filemap : 5904 kB
> domainbuilder: detail: xc_dom_module_file: filename="/share/dom0-rootfs.img.gz"
> domainbuilder: detail: xc_dom_malloc_filemap : 13553 kB
> domainbuilder: detail: xc_dom_boot_xen_init: ver 4.15, caps xen-3.0-aarch64 xen-3.0-armv7l
> domainbuilder: detail: xc_dom_rambase_init: RAM starts at 40000
> domainbuilder: detail: xc_dom_parse_image: called
> domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
> domainbuilder: detail: loader probe failed
> domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64) loader ...
> domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an arm64 Image
> domainbuilder: detail: loader probe failed
> domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32) loader ...
> domainbuilder: detail: loader probe OK
> domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
> domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l: 0x40008000 -> 0x405cc200
> domainbuilder: detail: xc_dom_devicetree_mem: called
> domainbuilder: detail: xc_dom_mem_init: mem 128 MB, pages 0x8000 pages, 4k each
> domainbuilder: detail: xc_dom_mem_init: 0x8000 pages
> domainbuilder: detail: xc_dom_boot_mem_init: called
> domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
> domainbuilder: detail: populate_guest_memory: populating RAM @ 0000000040000000-0000000048000000 (128MB)
> domainbuilder: detail: populate_one_size: populated 0x40/0x40 entries with shift 9
> domainbuilder: detail: meminit: placing boot modules at 0x472c2000
> domainbuilder: detail: meminit: ramdisk: 0x472c2000 -> 0x47fff000
> domainbuilder: detail: meminit: devicetree: 0x47fff000 -> 0x48000000
> domainbuilder: detail: xc_dom_build_image: called
> domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x40008+0x5c5 at 0xffff93aea000
> domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x40008000 -> 0x405cd000 (pfn 0x40008 + 0x5c5 pages)
> domainbuilder: detail: xc_dom_load_zimage_kernel: called
> domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 0x40008000-0x405cd000
> domainbuilder: detail: xc_dom_load_zimage_kernel: copy 6046208 bytes from blob 0xffff94dec000 to dst 0xffff93aea000
> domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x472c2+0xd3d at 0xffff92dad000
> domainbuilder: detail: xc_dom_alloc_segment: module0 : 0x472c2000 -> 0x47fff000 (pfn 0x472c2 + 0xd3d pages)
> domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x47fff+0x1 at 0xffff95847000
> domainbuilder: detail: xc_dom_alloc_segment: devicetree : 0x47fff000 -> 0x48000000 (pfn 0x47fff + 0x1 pages)
> domainbuilder: detail: alloc_magic_pages: called
> domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x48000000
> domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0
> domainbuilder: detail: xc_dom_boot_image: called
> domainbuilder: detail: bootearly: doing nothing
> domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-aarch64
> domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-armv7l <= matches
> domainbuilder: detail: start_info_arm: called
> domainbuilder: detail: domain builder memory footprint
> domainbuilder: detail: allocated
> domainbuilder: detail: malloc : 117 kB
> domainbuilder: detail: anon mmap : 0 bytes
> domainbuilder: detail: mapped
> domainbuilder: detail: file mmap : 19457 kB
> domainbuilder: detail: domU mmap : 19468 kB
> domainbuilder: detail: vcpu_arm32: called
> domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x40008000
> domainbuilder: detail: compat_gnttab_hvm_seed: d7: pfn=0x38000
> domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[0] -> d0 0x39000
> domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[1] -> d0 0x39001
> domainbuilder: detail: xc_dom_release: called
> (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> (XEN) d7v0: vGICR: SGI: unhandled word write 0x000000ffffffff to ICACTIVER0
> (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> (XEN) p2m.c:1919: d7v0: Invalid MFN 0x39281
>
> --
> Charles Chiou
> NOTE: This email (including attachments) contain Ambarella Proprietary and/or Confidential Information and is intended solely for the use of the individual(s) to whom it is addressed. Any unauthorized review, use, disclosure, distribute, copy, or print is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.
>

--
Julien Grall

> (+ Stefano)
>
> On 25/08/2020 13:04, Charles Chiou wrote:
> > Hello,
>
> Hello,
>
> > I'm having problem with virtual console when booting up 32 bit guests. I'm
> using qemu virt machine, running xen 64 bit, DOM0 Linux is 64 bit. The 64 bit
> guests that I've tried (Linux, an RTOS) have no issues, but when I tried to run
> 32 bit OS (tried linux, FreeRTOS from Galois, a bare-metal app), I get "Invalid
> MFN 0x33a08" or similar messages when making HVC calls such as:
> >
> > mov r0, #0
> > mov r1, #35
> > ldr r2, =banner
> > mov r12, #18
> > hvc #0xea1<----
> >
> > and get this at the HVC call to XEN:
> >
> > (XEN) p2m.c:1919: d8v0: Invalid MFN 0x33a08
> >
> > Not sure if there's misconfiguration somewhere, or compile options, etc.
> that caused this... Any pointer is greatly appreciated!
> >
> > Below is output of similar problem when booting up a 32 bit linux kernel:
>
> Looking at the code, this seems like an issue when trying to translate a guest
> virtual address to a machine address.
>
> A few questions:
> - Which QEMU version are you using?
> - What's your Linux configuration? Are you using LPAE or short page tables?

I am using default ubuntu package on bionic:

$ qemu-system-aarch64 --version
QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29)
Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers

I just tried the stable-4.14 branch and this "Invalid MFN 0x..." doesn't happen. It was on the master branch where I encountered this problem.
However, I don't seem to get anything to call guest_printk() after making hypervisor_console_io call even for 64-bit guest in this version (worked on master). This is a separate question: could that be a XEN compile option problem as I see "debug=n" from xen register dump:
(XEN) ----[ Xen-4.14.1-pre arm64 debug=n Not tainted ]----

I'm ignoring 32-bit Linux for now and trying to boot bare metal apps that calls hypervisor_console_io() at the very beginning of the boot-up. As I'm trying to port embedded RTOS apps (both 32 and 64 bits) to run as guests.

Thank you!


> I am also not sure what's the state of virtualization support in QEMU for 32-
> bit guest. I have CCed Stefano who may have a better idea.
>
> Best regards,
>
> > / # xl -v create -c /share/misc/linux32.cfg Parsing config from
> > /share/misc/linux32.cfg
> > libxl: info: libxl_create.c:122:libxl__domain_build_info_setdefault:
> > qemu-xen is unavailable, using qemu-xen-traditional instead: No such
> > file or directory
> > libxl: detail: libxl_create.c:623:libxl__domain_make: passthrough:
> > disabled
> > domainbuilder: detail: xc_dom_allocate: cmdline="rw root=/dev/ram
> rdinit=/sbin/init earlyprintk=serial,ttyAMA0", features=""
> > domainbuilder: detail: xc_dom_kernel_file: filename="/share/linux-domu-
> 32bit/arch/arm/boot/zImage"
> > domainbuilder: detail: xc_dom_malloc_filemap : 5904 kB
> > domainbuilder: detail: xc_dom_module_file: filename="/share/dom0-
> rootfs.img.gz"
> > domainbuilder: detail: xc_dom_malloc_filemap : 13553 kB
> > domainbuilder: detail: xc_dom_boot_xen_init: ver 4.15, caps
> > xen-3.0-aarch64 xen-3.0-armv7l
> > domainbuilder: detail: xc_dom_rambase_init: RAM starts at 40000
> > domainbuilder: detail: xc_dom_parse_image: called
> > domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary
> loader ...
> > domainbuilder: detail: loader probe failed
> > domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64)
> loader ...
> > domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an
> > arm64 Image
> > domainbuilder: detail: loader probe failed
> > domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32)
> loader ...
> > domainbuilder: detail: loader probe OK
> > domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
> > domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l:
> > 0x40008000 -> 0x405cc200
> > domainbuilder: detail: xc_dom_devicetree_mem: called
> > domainbuilder: detail: xc_dom_mem_init: mem 128 MB, pages 0x8000
> > pages, 4k each
> > domainbuilder: detail: xc_dom_mem_init: 0x8000 pages
> > domainbuilder: detail: xc_dom_boot_mem_init: called
> > domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
> > domainbuilder: detail: populate_guest_memory: populating RAM @
> > 0000000040000000-0000000048000000 (128MB)
> > domainbuilder: detail: populate_one_size: populated 0x40/0x40 entries
> > with shift 9
> > domainbuilder: detail: meminit: placing boot modules at 0x472c2000
> > domainbuilder: detail: meminit: ramdisk: 0x472c2000 -> 0x47fff000
> > domainbuilder: detail: meminit: devicetree: 0x47fff000 -> 0x48000000
> > domainbuilder: detail: xc_dom_build_image: called
> > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> 0x40008+0x5c5 at 0xffff93aea000
> > domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x40008000 ->
> 0x405cd000 (pfn 0x40008 + 0x5c5 pages)
> > domainbuilder: detail: xc_dom_load_zimage_kernel: called
> > domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg
> > 0x40008000-0x405cd000
> > domainbuilder: detail: xc_dom_load_zimage_kernel: copy 6046208 bytes
> > from blob 0xffff94dec000 to dst 0xffff93aea000
> > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> 0x472c2+0xd3d at 0xffff92dad000
> > domainbuilder: detail: xc_dom_alloc_segment: module0 : 0x472c2000 -
> > 0x47fff000 (pfn 0x472c2 + 0xd3d pages)
> > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> 0x47fff+0x1 at 0xffff95847000
> > domainbuilder: detail: xc_dom_alloc_segment: devicetree : 0x47fff000 ->
> 0x48000000 (pfn 0x47fff + 0x1 pages)
> > domainbuilder: detail: alloc_magic_pages: called
> > domainbuilder: detail: xc_dom_build_image : virt_alloc_end :
> > 0x48000000
> > domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0
> > domainbuilder: detail: xc_dom_boot_image: called
> > domainbuilder: detail: bootearly: doing nothing
> > domainbuilder: detail: xc_dom_compat_check: supported guest type:
> > xen-3.0-aarch64
> > domainbuilder: detail: xc_dom_compat_check: supported guest type:
> > xen-3.0-armv7l <= matches
> > domainbuilder: detail: start_info_arm: called
> > domainbuilder: detail: domain builder memory footprint
> > domainbuilder: detail: allocated
> > domainbuilder: detail: malloc : 117 kB
> > domainbuilder: detail: anon mmap : 0 bytes
> > domainbuilder: detail: mapped
> > domainbuilder: detail: file mmap : 19457 kB
> > domainbuilder: detail: domU mmap : 19468 kB
> > domainbuilder: detail: vcpu_arm32: called
> > domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x40008000
> > domainbuilder: detail: compat_gnttab_hvm_seed: d7: pfn=0x38000
> > domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[0] -> d0
> > 0x39000
> > domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[1] -> d0
> > 0x39001
> > domainbuilder: detail: xc_dom_release: called
> > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> > (XEN) d7v0: vGICR: SGI: unhandled word write 0x000000ffffffff to
> > ICACTIVER0
> > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x39281
NOTE: This email (including attachments) contain Ambarella Proprietary and/or Confidential Information and is intended solely for the use of the individual(s) to whom it is addressed. Any unauthorized review, use, disclosure, distribute, copy, or print is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.

On Tue, 25 Aug 2020, Charles Chiou wrote:
> > (+ Stefano)
> >
> > On 25/08/2020 13:04, Charles Chiou wrote:
> > > Hello,
> >
> > Hello,
> >
> > > I'm having problem with virtual console when booting up 32 bit guests. I'm
> > using qemu virt machine, running xen 64 bit, DOM0 Linux is 64 bit. The 64 bit
> > guests that I've tried (Linux, an RTOS) have no issues, but when I tried to run
> > 32 bit OS (tried linux, FreeRTOS from Galois, a bare-metal app), I get "Invalid
> > MFN 0x33a08" or similar messages when making HVC calls such as:
> > >
> > > mov r0, #0
> > > mov r1, #35
> > > ldr r2, =banner
> > > mov r12, #18
> > > hvc #0xea1<----
> > >
> > > and get this at the HVC call to XEN:
> > >
> > > (XEN) p2m.c:1919: d8v0: Invalid MFN 0x33a08
> > >
> > > Not sure if there's misconfiguration somewhere, or compile options, etc.
> > that caused this... Any pointer is greatly appreciated!
> > >
> > > Below is output of similar problem when booting up a 32 bit linux kernel:
> >
> > Looking at the code, this seems like an issue when trying to translate a guest
> > virtual address to a machine address.
> >
> > A few questions:
> > - Which QEMU version are you using?
> > - What's your Linux configuration? Are you using LPAE or short page tables?
>
> I am using default ubuntu package on bionic:
>
> $ qemu-system-aarch64 --version
> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29)
> Copyright (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
>
> I just tried the stable-4.14 branch and this "Invalid MFN 0x..." doesn't happen. It was on the master branch where I encountered this problem.
> However, I don't seem to get anything to call guest_printk() after making hypervisor_console_io call even for 64-bit guest in this version (worked on master). This is a separate question: could that be a XEN compile option problem as I see "debug=n" from xen register dump:
> (XEN) ----[ Xen-4.14.1-pre arm64 debug=n Not tainted ]----

The call chain is:

xen/drivers/char/console.c:do_console_io
xen/drivers/char/console.c:guest_console_write
xen/drivers/char/console.c:guest_printk

You can enable debug through kconfig by doing "make menuconfig" under
xen/


> I'm ignoring 32-bit Linux for now and trying to boot bare metal apps that calls hypervisor_console_io() at the very beginning of the boot-up. As I'm trying to port embedded RTOS apps (both 32 and 64 bits) to run as guests.

Excellent. When doing that kind of work, I find the debug hypercalls very useful, see:

xen/arch/arm/traps.c:do_debug_trap

Once you enable DEBUG in the build, you can do

hvc 0xfffd

In the guest for instance to print the program counter.



> Thank you!
>
>
> > I am also not sure what's the state of virtualization support in QEMU for 32-
> > bit guest. I have CCed Stefano who may have a better idea.
> >
> > Best regards,
> >
> > > / # xl -v create -c /share/misc/linux32.cfg Parsing config from
> > > /share/misc/linux32.cfg
> > > libxl: info: libxl_create.c:122:libxl__domain_build_info_setdefault:
> > > qemu-xen is unavailable, using qemu-xen-traditional instead: No such
> > > file or directory
> > > libxl: detail: libxl_create.c:623:libxl__domain_make: passthrough:
> > > disabled
> > > domainbuilder: detail: xc_dom_allocate: cmdline="rw root=/dev/ram
> > rdinit=/sbin/init earlyprintk=serial,ttyAMA0", features=""
> > > domainbuilder: detail: xc_dom_kernel_file: filename="/share/linux-domu-
> > 32bit/arch/arm/boot/zImage"
> > > domainbuilder: detail: xc_dom_malloc_filemap : 5904 kB
> > > domainbuilder: detail: xc_dom_module_file: filename="/share/dom0-
> > rootfs.img.gz"
> > > domainbuilder: detail: xc_dom_malloc_filemap : 13553 kB
> > > domainbuilder: detail: xc_dom_boot_xen_init: ver 4.15, caps
> > > xen-3.0-aarch64 xen-3.0-armv7l
> > > domainbuilder: detail: xc_dom_rambase_init: RAM starts at 40000
> > > domainbuilder: detail: xc_dom_parse_image: called
> > > domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary
> > loader ...
> > > domainbuilder: detail: loader probe failed
> > > domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64)
> > loader ...
> > > domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an
> > > arm64 Image
> > > domainbuilder: detail: loader probe failed
> > > domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32)
> > loader ...
> > > domainbuilder: detail: loader probe OK
> > > domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
> > > domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l:
> > > 0x40008000 -> 0x405cc200
> > > domainbuilder: detail: xc_dom_devicetree_mem: called
> > > domainbuilder: detail: xc_dom_mem_init: mem 128 MB, pages 0x8000
> > > pages, 4k each
> > > domainbuilder: detail: xc_dom_mem_init: 0x8000 pages
> > > domainbuilder: detail: xc_dom_boot_mem_init: called
> > > domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
> > > domainbuilder: detail: populate_guest_memory: populating RAM @
> > > 0000000040000000-0000000048000000 (128MB)
> > > domainbuilder: detail: populate_one_size: populated 0x40/0x40 entries
> > > with shift 9
> > > domainbuilder: detail: meminit: placing boot modules at 0x472c2000
> > > domainbuilder: detail: meminit: ramdisk: 0x472c2000 -> 0x47fff000
> > > domainbuilder: detail: meminit: devicetree: 0x47fff000 -> 0x48000000
> > > domainbuilder: detail: xc_dom_build_image: called
> > > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> > 0x40008+0x5c5 at 0xffff93aea000
> > > domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x40008000 ->
> > 0x405cd000 (pfn 0x40008 + 0x5c5 pages)
> > > domainbuilder: detail: xc_dom_load_zimage_kernel: called
> > > domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg
> > > 0x40008000-0x405cd000
> > > domainbuilder: detail: xc_dom_load_zimage_kernel: copy 6046208 bytes
> > > from blob 0xffff94dec000 to dst 0xffff93aea000
> > > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> > 0x472c2+0xd3d at 0xffff92dad000
> > > domainbuilder: detail: xc_dom_alloc_segment: module0 : 0x472c2000 -
> > > 0x47fff000 (pfn 0x472c2 + 0xd3d pages)
> > > domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn
> > 0x47fff+0x1 at 0xffff95847000
> > > domainbuilder: detail: xc_dom_alloc_segment: devicetree : 0x47fff000 ->
> > 0x48000000 (pfn 0x47fff + 0x1 pages)
> > > domainbuilder: detail: alloc_magic_pages: called
> > > domainbuilder: detail: xc_dom_build_image : virt_alloc_end :
> > > 0x48000000
> > > domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0
> > > domainbuilder: detail: xc_dom_boot_image: called
> > > domainbuilder: detail: bootearly: doing nothing
> > > domainbuilder: detail: xc_dom_compat_check: supported guest type:
> > > xen-3.0-aarch64
> > > domainbuilder: detail: xc_dom_compat_check: supported guest type:
> > > xen-3.0-armv7l <= matches
> > > domainbuilder: detail: start_info_arm: called
> > > domainbuilder: detail: domain builder memory footprint
> > > domainbuilder: detail: allocated
> > > domainbuilder: detail: malloc : 117 kB
> > > domainbuilder: detail: anon mmap : 0 bytes
> > > domainbuilder: detail: mapped
> > > domainbuilder: detail: file mmap : 19457 kB
> > > domainbuilder: detail: domU mmap : 19468 kB
> > > domainbuilder: detail: vcpu_arm32: called
> > > domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x40008000
> > > domainbuilder: detail: compat_gnttab_hvm_seed: d7: pfn=0x38000
> > > domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[0] -> d0
> > > 0x39000
> > > domainbuilder: detail: xc_dom_set_gnttab_entry: d7 gnt[1] -> d0
> > > 0x39001
> > > domainbuilder: detail: xc_dom_release: called
> > > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> > > (XEN) d7v0: vGICR: SGI: unhandled word write 0x000000ffffffff to
> > > ICACTIVER0
> > > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x3b401
> > > (XEN) p2m.c:1919: d7v0: Invalid MFN 0x39281
> NOTE: This email (including attachments) contain Ambarella Proprietary and/or Confidential Information and is intended solely for the use of the individual(s) to whom it is addressed. Any unauthorized review, use, disclosure, distribute, copy, or print is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.
>

> > > Looking at the code, this seems like an issue when trying to
> > > translate a guest virtual address to a machine address.
> > >
> > > A few questions:
> > > - Which QEMU version are you using?
> > > - What's your Linux configuration? Are you using LPAE or short page
> tables?
> >
> > I am using default ubuntu package on bionic:
> >
> > $ qemu-system-aarch64 --version
> > QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29) Copyright
> > (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> >
> > I just tried the stable-4.14 branch and this "Invalid MFN 0x..." doesn't
> happen. It was on the master branch where I encountered this problem.
> > However, I don't seem to get anything to call guest_printk() after making
> hypervisor_console_io call even for 64-bit guest in this version (worked on
> master). This is a separate question: could that be a XEN compile option
> problem as I see "debug=n" from xen register dump:
> > (XEN) ----[ Xen-4.14.1-pre arm64 debug=n Not tainted ]----
>
> The call chain is:
>
> xen/drivers/char/console.c:do_console_io
> xen/drivers/char/console.c:guest_console_write
> xen/drivers/char/console.c:guest_printk
>
> You can enable debug through kconfig by doing "make menuconfig" under
> xen/

Thank you for pointing this out! After I enabled the verbose debug messages, the hvc #0xea1 call is now taking effect. I'm seeing the previous problem of "invalid MFN" after that. It seems that the pointer 0x40000058 had been mapped to MFN 0x2be08. How do I debug or where locate the problem? I'm very new to XEN so it's no obvious yet to me from reading the source code how after guest_printk(), xen translate the addresses.

This is the output of my guest boot up:
/ # xl -v create -c /share/misc/bm.cfg
Parsing config from /share/misc/bm.cfg
libxl: info: libxl_create.c:122:libxl__domain_build_info_setdefault: qemu-xen is unavailable, using qemu-xen-traditional instead: No such file or directory
libxl: detail: libxl_create.c:623:libxl__domain_make: passthrough: disabled
domainbuilder: detail: xc_dom_allocate: cmdline="", features=""
domainbuilder: detail: xc_dom_kernel_file: filename="/share/misc/bm.img"
domainbuilder: detail: xc_dom_boot_xen_init: ver 4.14, caps xen-3.0-aarch64 xen-3.0-armv7l
domainbuilder: detail: xc_dom_rambase_init: RAM starts at 40000
domainbuilder: detail: xc_dom_parse_image: called
domainbuilder: detail: xc_dom_find_loader: trying multiboot-binary loader ...
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM64) loader ...
domainbuilder: detail: xc_dom_probe_zimage64_kernel: kernel is not an arm64 Image
domainbuilder: detail: loader probe failed
domainbuilder: detail: xc_dom_find_loader: trying Linux zImage (ARM32) loader ...
domainbuilder: detail: loader probe OK
domainbuilder: detail: xc_dom_parse_zimage32_kernel: called
domainbuilder: detail: xc_dom_parse_zimage32_kernel: xen-3.0-armv7l: 0x40008000 -> 0x4000807c
domainbuilder: detail: xc_dom_devicetree_mem: called
domainbuilder: detail: xc_dom_mem_init: mem 128 MB, pages 0x8000 pages, 4k each
domainbuilder: detail: xc_dom_mem_init: 0x8000 pages
domainbuilder: detail: xc_dom_boot_mem_init: called
domainbuilder: detail: set_mode: guest xen-3.0-armv7l, address size 32
domainbuilder: detail: populate_guest_memory: populating RAM @ 0000000040000000-0000000048000000 (128MB)
domainbuilder: detail: populate_one_size: populated 0x40/0x40 entries with shift 9
domainbuilder: detail: meminit: placing boot modules at 0x47fff000
domainbuilder: detail: meminit: devicetree: 0x47fff000 -> 0x48000000
domainbuilder: detail: xc_dom_build_image: called
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x40008+0x1 at 0xffff8c229000
domainbuilder: detail: xc_dom_alloc_segment: kernel : 0x40008000 -> 0x40009000 (pfn 0x40008 + 0x1 pages)
domainbuilder: detail: xc_dom_load_zimage_kernel: called
domainbuilder: detail: xc_dom_load_zimage_kernel: kernel seg 0x40008000-0x40009000
domainbuilder: detail: xc_dom_load_zimage_kernel: copy 124 bytes from blob 0xffff8c6c0000 to dst 0xffff8c229000
domainbuilder: detail: xc_dom_pfn_to_ptr_retcount: domU mapping: pfn 0x47fff+0x1 at 0xffff8c228000
domainbuilder: detail: xc_dom_alloc_segment: devicetree : 0x47fff000 -> 0x48000000 (pfn 0x47fff + 0x1 pages)
domainbuilder: detail: alloc_magic_pages: called
domainbuilder: detail: xc_dom_build_image : virt_alloc_end : 0x48000000
domainbuilder: detail: xc_dom_build_image : virt_pgtab_end : 0x0
domainbuilder: detail: xc_dom_boot_image: called
domainbuilder: detail: bootearly: doing nothing
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-aarch64
domainbuilder: detail: xc_dom_compat_check: supported guest type: xen-3.0-armv7l <= matches
domainbuilder: detail: start_info_arm: called
domainbuilder: detail: domain builder memory footprint
domainbuilder: detail: allocated
domainbuilder: detail: malloc : 2848 bytes
domainbuilder: detail: anon mmap : 0 bytes
domainbuilder: detail: mapped
domainbuilder: detail: file mmap : 124 bytes
domainbuilder: detail: domU mmap : 8192 bytes
domainbuilder: detail: vcpu_arm32: called
domainbuilder: detail: Initial state CPSR 0x1d3 PC 0x40008000
domainbuilder: detail: compat_gnttab_hvm_seed: d5: pfn=0x38000
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[0] -> d0 0x39000
domainbuilder: detail: xc_dom_set_gnttab_entry: d5 gnt[1] -> d0 0x39001
domainbuilder: detail: xc_dom_release: called
(XEN) p2m.c:1919: d5v0: Invalid MFN 0x2be08

The 32-bit guest is very simple, it calls hvc very early in and uses physical address:

$ arm-none-eabi-objdump -d bm.elf

bm.elf: file format elf32-littlearm


Disassembly of section .text:

40008000 <_stext>:
40008000: 13100a4d tstne r0, #315392 ; 0x4d000
40008004: 13100a4d tstne r0, #315392 ; 0x4d000
40008008: 13100a4d tstne r0, #315392 ; 0x4d000
4000800c: 13100a4d tstne r0, #315392 ; 0x4d000
40008010: 13100a4d tstne r0, #315392 ; 0x4d000
40008014: 13100a4d tstne r0, #315392 ; 0x4d000
40008018: 13100a4d tstne r0, #315392 ; 0x4d000
4000801c: e320f000 nop {0}
40008020: ea000006 b 40008040 <reset>
40008024: 016f2818 .word 0x016f2818
40008028: 00000000 .word 0x00000000
4000802c: 0079c200 .word 0x0079c200
40008030: 04030201 .word 0x04030201
40008034: e320f000 nop {0}
40008038: e320f000 nop {0}
4000803c: e320f000 nop {0}

40008040 <reset>:
40008040: e3a00000 mov r0, #0
40008044: e3a01023 mov r1, #35 ; 0x23
40008048: e28f2008 add r2, pc, #8
4000804c: e3a0c012 mov ip, #18
40008050: e140ea71 hvc 3745 ; 0xea1
40008054: eafffffe b 40008054 <reset+0x14>

40008058 <banner>:
40008058: 65726854 .word 0x65726854
4000805c: 2f586461 .word 0x2f586461
40008060: 204e4558 .word 0x204e4558
40008064: 746e6f43 .word 0x746e6f43
40008068: 656e6961 .word 0x656e6961
4000806c: 6f422072 .word 0x6f422072
40008070: 6e69746f .word 0x6e69746f
40008074: 70752067 .word 0x70752067
40008078: 000a0d21 .word 0x000a0d21




Thank you!

>
> > I'm ignoring 32-bit Linux for now and trying to boot bare metal apps that
> calls hypervisor_console_io() at the very beginning of the boot-up. As I'm
> trying to port embedded RTOS apps (both 32 and 64 bits) to run as guests.
>
> Excellent. When doing that kind of work, I find the debug hypercalls very
> useful, see:
>
> xen/arch/arm/traps.c:do_debug_trap
>
> Once you enable DEBUG in the build, you can do
>
> hvc 0xfffd
>
> In the guest for instance to print the program counter.
NOTE: This email (including attachments) contain Ambarella Proprietary and/or Confidential Information and is intended solely for the use of the individual(s) to whom it is addressed. Any unauthorized review, use, disclosure, distribute, copy, or print is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.

On Wed, 26 Aug 2020, Charles Chiou wrote:
> > > > Looking at the code, this seems like an issue when trying to
> > > > translate a guest virtual address to a machine address.
> > > >
> > > > A few questions:
> > > > - Which QEMU version are you using?
> > > > - What's your Linux configuration? Are you using LPAE or short page
> > tables?
> > >
> > > I am using default ubuntu package on bionic:
> > >
> > > $ qemu-system-aarch64 --version
> > > QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29) Copyright
> > > (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> > >
> > > I just tried the stable-4.14 branch and this "Invalid MFN 0x..." doesn't
> > happen. It was on the master branch where I encountered this problem.
> > > However, I don't seem to get anything to call guest_printk() after making
> > hypervisor_console_io call even for 64-bit guest in this version (worked on
> > master). This is a separate question: could that be a XEN compile option
> > problem as I see "debug=n" from xen register dump:
> > > (XEN) ----[ Xen-4.14.1-pre arm64 debug=n Not tainted ]----
> >
> > The call chain is:
> >
> > xen/drivers/char/console.c:do_console_io
> > xen/drivers/char/console.c:guest_console_write
> > xen/drivers/char/console.c:guest_printk
> >
> > You can enable debug through kconfig by doing "make menuconfig" under
> > xen/
>
> Thank you for pointing this out! After I enabled the verbose debug messages, the hvc #0xea1 call is now taking effect.

Great!


>I'm seeing the previous problem of "invalid MFN" after that. It seems that the pointer 0x40000058 had been mapped to MFN 0x2be08. How do I debug or where locate the problem? I'm very new to XEN so it's no obvious yet to me from reading the source code how after guest_printk(), xen translate the addresses.

The baremetal application code and the disassembly look correct. I
wonder if the issue is that the baremetal application is passing a guest
physical address when Xen expects a guest virtual address.

The string gets copied by:

xen/drivers/char/console.c:guest_console_write

calling:

copy_from_guest -> copy_from_guest_offset -> raw_copy_from_guest

eventually it goes to:

xen/arch/arm/guestcopy.c:copy_guest
xen/arch/arm/guestcopy.c:translate_get_page


Looking at the code, translate_get_page is called with linear=true
write=false. linear=true causes translate_get_page to call
get_page_from_gva. I wonder if it is possible that get_page_from_gva is
not doing the right thing here.

As a test, maybe you could hack guest_console_write to call a modified
version of raw_copy_from_guest that uses guest physical addresses
instead.



diff --git a/xen/arch/arm/guestcopy.c b/xen/arch/arm/guestcopy.c
index 7a0f3e9d5f..106a95e33f 100644
--- a/xen/arch/arm/guestcopy.c
+++ b/xen/arch/arm/guestcopy.c
@@ -130,6 +130,11 @@ unsigned long raw_copy_from_guest(void *to, const void __user *from, unsigned le
COPY_from_guest | COPY_linear);
}

+unsigned long raw_copy_from_guest_special(void *to, const void __user *from, unsigned len)
+{
+ return copy_guest(to, (uint64_t)from, len, GPA_INFO(current->domain),
+ COPY_from_guest);
+}
unsigned long copy_to_guest_phys_flush_dcache(struct domain *d,
paddr_t gpa,
void *buf,
diff --git a/xen/drivers/char/console.c b/xen/drivers/char/console.c
index 861ad53a8f..2dd6187aa5 100644
--- a/xen/drivers/char/console.c
+++ b/xen/drivers/char/console.c
@@ -590,6 +590,7 @@ static inline void xen_console_write_debug_port(const char *buf, size_t len)
}
#endif

+extern unsigned long raw_copy_from_guest_special(void *to, const void __user *from, unsigned len);
static long guest_console_write(XEN_GUEST_HANDLE_PARAM(char) buffer,
unsigned int count)
{
@@ -604,9 +605,18 @@ static long guest_console_write(XEN_GUEST_HANDLE_PARAM(char) buffer,
__HYPERVISOR_console_io, "iih",
CONSOLEIO_write, count, buffer);

- kcount = min((size_t)count, sizeof(kbuf) - 1);
- if ( copy_from_guest(kbuf, buffer, kcount) )
- return -EFAULT;
+ if ( current->domain->domain_id > 0 )
+ {
+ kcount = min((size_t)count, sizeof(kbuf) - 1);
+ if ( raw_copy_from_guest_special(kbuf, buffer.p, kcount) )
+ return -EFAULT;
+ }
+ else
+ {
+ kcount = min((size_t)count, sizeof(kbuf) - 1);
+ if ( copy_from_guest(kbuf, buffer, kcount) )
+ return -EFAULT;
+ }

if ( is_hardware_domain(cd) )
{

Hi,

On 26/08/2020 18:20, Stefano Stabellini wrote:
> On Wed, 26 Aug 2020, Charles Chiou wrote:
>>>>> Looking at the code, this seems like an issue when trying to
>>>>> translate a guest virtual address to a machine address.
>>>>>
>>>>> A few questions:
>>>>> - Which QEMU version are you using?
>>>>> - What's your Linux configuration? Are you using LPAE or short page
>>> tables?
>>>>
>>>> I am using default ubuntu package on bionic:
>>>>
>>>> $ qemu-system-aarch64 --version
>>>> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29) Copyright
>>>> (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
>>>>
>>>> I just tried the stable-4.14 branch and this "Invalid MFN 0x..." doesn't
>>> happen. It was on the master branch where I encountered this problem.
>>>> However, I don't seem to get anything to call guest_printk() after making
>>> hypervisor_console_io call even for 64-bit guest in this version (worked on
>>> master). This is a separate question: could that be a XEN compile option
>>> problem as I see "debug=n" from xen register dump:
>>>> (XEN) ----[ Xen-4.14.1-pre arm64 debug=n Not tainted ]----
>>>
>>> The call chain is:
>>>
>>> xen/drivers/char/console.c:do_console_io
>>> xen/drivers/char/console.c:guest_console_write
>>> xen/drivers/char/console.c:guest_printk
>>>
>>> You can enable debug through kconfig by doing "make menuconfig" under
>>> xen/
>>
>> Thank you for pointing this out! After I enabled the verbose debug messages, the hvc #0xea1 call is now taking effect.
>
> Great!
>
>
>> I'm seeing the previous problem of "invalid MFN" after that. It seems that the pointer 0x40000058 had been mapped to MFN 0x2be08. How do I debug or where locate the problem? I'm very new to XEN so it's no obvious yet to me from reading the source code how after guest_printk(), xen translate the addresses.
>
> The baremetal application code and the disassembly look correct. I
> wonder if the issue is that the baremetal application is passing a guest
> physical address when Xen expects a guest virtual address.

Bear in mind that issuing an hypercall with MMU disabled is usually
risky because of memory attribute mismatch between Xen and the Guest.

Although, you would only see data corruption and not "invalid MFN".

>
> The string gets copied by:
>
> xen/drivers/char/console.c:guest_console_write
>
> calling:
>
> copy_from_guest -> copy_from_guest_offset -> raw_copy_from_guest
>
> eventually it goes to:
>
> xen/arch/arm/guestcopy.c:copy_guest
> xen/arch/arm/guestcopy.c:translate_get_page
>
>
> Looking at the code, translate_get_page is called with linear=true
> write=false. linear=true causes translate_get_page to call
> get_page_from_gva. I wonder if it is possible that get_page_from_gva is
> not doing the right thing here.

When the MMU is turned off, VA == PA. The AT instruction is able to deal
with such situation. However...

>>>> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29) Copyright
>>>> (c) 2003-2017 Fabrice Bellard and the QEMU Project developers


... this is fairly an old version of QEMU. I remember some issues with
the implementation of the AT instruction in QEMU. Looking at the commit
logs, it seems there are a few fixes that is not part of 2.11.1.

I would highly recomend to use a more recent version of QEMU in order to
narrow down the problem.

Cheers,

--
Julien Grall

> >>>>> Looking at the code, this seems like an issue when trying to
> >>>>> translate a guest virtual address to a machine address.
> >>>>>
> >>>>> A few questions:
> >>>>> - Which QEMU version are you using?
> >>>>> - What's your Linux configuration? Are you using LPAE or
> >>>>> short page
> >>> tables?
> >>>>
> >>>> I am using default ubuntu package on bionic:
> >>>>
> >>>> $ qemu-system-aarch64 --version
> >>>> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29)
> >>>> Copyright
> >>>> (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> >>>>
> >>>> I just tried the stable-4.14 branch and this "Invalid MFN 0x..."
> >>>> doesn't
> >>> happen. It was on the master branch where I encountered this problem.
> >>>> However, I don't seem to get anything to call guest_printk() after
> >>>> making
> >>> hypervisor_console_io call even for 64-bit guest in this version
> >>> (worked on master). This is a separate question: could that be a XEN
> >>> compile option problem as I see "debug=n" from xen register dump:
> >>>> (XEN) ----[ Xen-4.14.1-pre arm64 debug=n Not tainted ]----
> >>>
> >>> The call chain is:
> >>>
> >>> xen/drivers/char/console.c:do_console_io
> >>> xen/drivers/char/console.c:guest_console_write
> >>> xen/drivers/char/console.c:guest_printk
> >>>
> >>> You can enable debug through kconfig by doing "make menuconfig"
> >>> under xen/
> >>
> >> Thank you for pointing this out! After I enabled the verbose debug
> messages, the hvc #0xea1 call is now taking effect.
> >
> > Great!
> >
> >
> >> I'm seeing the previous problem of "invalid MFN" after that. It seems that
> the pointer 0x40000058 had been mapped to MFN 0x2be08. How do I debug
> or where locate the problem? I'm very new to XEN so it's no obvious yet to
> me from reading the source code how after guest_printk(), xen translate the
> addresses.
> >
> > The baremetal application code and the disassembly look correct. I
> > wonder if the issue is that the baremetal application is passing a
> > guest physical address when Xen expects a guest virtual address.
>
> Bear in mind that issuing an hypercall with MMU disabled is usually risky
> because of memory attribute mismatch between Xen and the Guest.
>
> Although, you would only see data corruption and not "invalid MFN".
>
> >
> > The string gets copied by:
> >
> > xen/drivers/char/console.c:guest_console_write
> >
> > calling:
> >
> > copy_from_guest -> copy_from_guest_offset -> raw_copy_from_guest
> >
> > eventually it goes to:
> >
> > xen/arch/arm/guestcopy.c:copy_guest
> > xen/arch/arm/guestcopy.c:translate_get_page
> >
> >
> > Looking at the code, translate_get_page is called with linear=true
> > write=false. linear=true causes translate_get_page to call
> > get_page_from_gva. I wonder if it is possible that get_page_from_gva
> > is not doing the right thing here.
>
> When the MMU is turned off, VA == PA. The AT instruction is able to deal
> with such situation. However...

Hi Stefano, thank you for the pointers. It was very helpful to get started in tracing the internals of xen.

> >>>> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29)
> Copyright >>>> (c) 2003-2017 Fabrice Bellard and the QEMU Project
> developers
>
>
> ... this is fairly an old version of QEMU. I remember some issues with the
> implementation of the AT instruction in QEMU. Looking at the commit logs, it
> seems there are a few fixes that is not part of 2.11.1.
>
> I would highly recomend to use a more recent version of QEMU in order to
> narrow down the problem.

Hi Julien, switching qemu to v3.1.1.1 solved the problem! (Tried 5.x but ran into other problems, but those are something else). The address translation now seems ok now and gvirt_to_maddr() is returning the right values after executing "at s12e1w, %0", and guest console output is working. Thank you!

--
Charles Chiou
NOTE: This email (including attachments) contain Ambarella Proprietary and/or Confidential Information and is intended solely for the use of the individual(s) to whom it is addressed. Any unauthorized review, use, disclosure, distribute, copy, or print is prohibited. If you are not an intended recipient, please contact the sender by reply email and destroy all copies of the original message. Thank you.

On Thu, 27 Aug 2020, Charles Chiou wrote:
> > >>>>> Looking at the code, this seems like an issue when trying to
> > >>>>> translate a guest virtual address to a machine address.
> > >>>>>
> > >>>>> A few questions:
> > >>>>> - Which QEMU version are you using?
> > >>>>> - What's your Linux configuration? Are you using LPAE or
> > >>>>> short page
> > >>> tables?
> > >>>>
> > >>>> I am using default ubuntu package on bionic:
> > >>>>
> > >>>> $ qemu-system-aarch64 --version
> > >>>> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29)
> > >>>> Copyright
> > >>>> (c) 2003-2017 Fabrice Bellard and the QEMU Project developers
> > >>>>
> > >>>> I just tried the stable-4.14 branch and this "Invalid MFN 0x..."
> > >>>> doesn't
> > >>> happen. It was on the master branch where I encountered this problem.
> > >>>> However, I don't seem to get anything to call guest_printk() after
> > >>>> making
> > >>> hypervisor_console_io call even for 64-bit guest in this version
> > >>> (worked on master). This is a separate question: could that be a XEN
> > >>> compile option problem as I see "debug=n" from xen register dump:
> > >>>> (XEN) ----[ Xen-4.14.1-pre arm64 debug=n Not tainted ]----
> > >>>
> > >>> The call chain is:
> > >>>
> > >>> xen/drivers/char/console.c:do_console_io
> > >>> xen/drivers/char/console.c:guest_console_write
> > >>> xen/drivers/char/console.c:guest_printk
> > >>>
> > >>> You can enable debug through kconfig by doing "make menuconfig"
> > >>> under xen/
> > >>
> > >> Thank you for pointing this out! After I enabled the verbose debug
> > messages, the hvc #0xea1 call is now taking effect.
> > >
> > > Great!
> > >
> > >
> > >> I'm seeing the previous problem of "invalid MFN" after that. It seems that
> > the pointer 0x40000058 had been mapped to MFN 0x2be08. How do I debug
> > or where locate the problem? I'm very new to XEN so it's no obvious yet to
> > me from reading the source code how after guest_printk(), xen translate the
> > addresses.
> > >
> > > The baremetal application code and the disassembly look correct. I
> > > wonder if the issue is that the baremetal application is passing a
> > > guest physical address when Xen expects a guest virtual address.
> >
> > Bear in mind that issuing an hypercall with MMU disabled is usually risky
> > because of memory attribute mismatch between Xen and the Guest.
> >
> > Although, you would only see data corruption and not "invalid MFN".
> >
> > >
> > > The string gets copied by:
> > >
> > > xen/drivers/char/console.c:guest_console_write
> > >
> > > calling:
> > >
> > > copy_from_guest -> copy_from_guest_offset -> raw_copy_from_guest
> > >
> > > eventually it goes to:
> > >
> > > xen/arch/arm/guestcopy.c:copy_guest
> > > xen/arch/arm/guestcopy.c:translate_get_page
> > >
> > >
> > > Looking at the code, translate_get_page is called with linear=true
> > > write=false. linear=true causes translate_get_page to call
> > > get_page_from_gva. I wonder if it is possible that get_page_from_gva
> > > is not doing the right thing here.
> >
> > When the MMU is turned off, VA == PA. The AT instruction is able to deal
> > with such situation. However...
>
> Hi Stefano, thank you for the pointers. It was very helpful to get started in tracing the internals of xen.

You are welcome :-)


> > >>>> QEMU emulator version 2.11.1(Debian 1:2.11+dfsg-1ubuntu7.29)
> > Copyright >>>> (c) 2003-2017 Fabrice Bellard and the QEMU Project
> > developers
> >
> >
> > ... this is fairly an old version of QEMU. I remember some issues with the
> > implementation of the AT instruction in QEMU. Looking at the commit logs, it
> > seems there are a few fixes that is not part of 2.11.1.
> >
> > I would highly recomend to use a more recent version of QEMU in order to
> > narrow down the problem.
>
> Hi Julien, switching qemu to v3.1.1.1 solved the problem! (Tried 5.x but ran into other problems, but those are something else). The address translation now seems ok now and gvirt_to_maddr() is returning the right values after executing "at s12e1w, %0", and guest console output is working. Thank you!

Great to hear!