Mailing List Archive: kmap_atomic issue with SLES11SP1 32bit XEN driver code

kmap_atomic issue with SLES11SP1 32bit XEN driver code

Dec 7, 2011, 1:47 PM

Post #1 of 12 (525 views)

kernel: Linux 360 2.6.32.12-0.7-xen #1 SMP 2010-05-20 11:14:20 +0200
i686 i686 i386 GNU/Linux

I am having issues with some code that uses kmap_atomic(). I am getting:

BUG: unable to handle kernel paging request at 23a1a000 (which is the
address returned from kmap_atomic().)

The same code works when running on non-XEN 32bit kernels so I am
wondering why this does not work under
XEN kernels. Is there a different approach that I need to take for 32bit
XEN kernels?

I really only need to do this code segment if the memory address is a
high memory address. Is there a MACRO or function
that can help me determine this?

Here is a code snippet:

void *linux_vaddr = NULL; /* kmapped temporary
virtual address */
int linux_page_offset = 0; /* offset in page */
int count = 0; /* bytes left to
transfer */
int left = byte_count; /* number of bytes left
to transfer */
int memcpysize = 0; /* current size to
transfer */
struct page *linux_page = NULL; /* calculated page */
int kmap_flags = 0;

linux_page = __pfn_to_page(physical_address >> PAGE_SHIFT);
linux_page_offset = (physical_address &
0x0000000000000FFFULL);
memcpysize = min((PAGE_SIZE - linux_page_offset), left);

kmap_flags = KM_USER0;

linux_vaddr = kmap_atomic(linux_page, kmap_flags) +
linux_page_offset;

printk("%s: called kmap_atomic, "
"calling memcpy linux_vaddr = 0x%x virt_address
= 0x%x count = 0x%x\n",
__func__, linux_vaddr, virt_address, count);
/*
* Either need to copy to a kmapped destination
* or a kmapped source.
*/
if (type == 0) // Write to s/g element, dest virtual
addr was known.
memcpy((void *)virt_address+count, (void
*)linux_vaddr, memcpysize);
else // Source virt. address was known.
memcpy((void *)linux_vaddr, (void
*)virt_address+count, memcpysize);

printk("OS_Linux32_Xfer_SG_Page: calling kunmap_atomic, "
"calling memcpy linux_vaddr = 0x%lx\n",
linux_vaddr);

kunmap_atomic(linux_vaddr, kmap_flags);

--
Don Brace
SPSN Linux Development
Hewlett-Packard Company

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

JBeulich at suse

Dec 12, 2011, 7:16 AM

Post #2 of 12 (528 views)

Permalink

>>> On 07.12.11 at 22:47, dbrace <dab@hp.com> wrote:
> kernel: Linux 360 2.6.32.12-0.7-xen #1 SMP 2010-05-20 11:14:20 +0200
> i686 i686 i386 GNU/Linux

Pretty outdated.

> I am having issues with some code that uses kmap_atomic(). I am getting:
>
> BUG: unable to handle kernel paging request at 23a1a000 (which is the
> address returned from kmap_atomic().)

For a valid high memory page this is surely not being returned by
kmap_atomic(), so we have to assume that what you started with is
the address of a low memory one.

> The same code works when running on non-XEN 32bit kernels so I am
> wondering why this does not work under
> XEN kernels. Is there a different approach that I need to take for 32bit
> XEN kernels?

No, but ...

> I really only need to do this code segment if the memory address is a
> high memory address. Is there a MACRO or function
> that can help me determine this?

PageHighMem().

> Here is a code snippet:
>
> void *linux_vaddr = NULL; /* kmapped temporary
> virtual address */
> int linux_page_offset = 0; /* offset in page */
> int count = 0; /* bytes left to
> transfer */
> int left = byte_count; /* number of bytes left
> to transfer */
> int memcpysize = 0; /* current size to
> transfer */
> struct page *linux_page = NULL; /* calculated page */
> int kmap_flags = 0;
>
> linux_page = __pfn_to_page(physical_address >> PAGE_SHIFT);

... without knowing where you got physical_address from it's impossible
to tell whether your problem is that under Xen physical and machine
(sometimes called "bus") addresses being distinct (end hence you're
possibly lacking a translation between the two).

Jan

> linux_page_offset = (physical_address &
> 0x0000000000000FFFULL);
> memcpysize = min((PAGE_SIZE - linux_page_offset), left);
>
> kmap_flags = KM_USER0;
>
> linux_vaddr = kmap_atomic(linux_page, kmap_flags) +
> linux_page_offset;
>
> printk("%s: called kmap_atomic, "
> "calling memcpy linux_vaddr = 0x%x virt_address
> = 0x%x count = 0x%x\n",
> __func__, linux_vaddr, virt_address, count);
> /*
> * Either need to copy to a kmapped destination
> * or a kmapped source.
> */
> if (type == 0) // Write to s/g element, dest virtual
> addr was known.
> memcpy((void *)virt_address+count, (void
> *)linux_vaddr, memcpysize);
> else // Source virt. address was known.
> memcpy((void *)linux_vaddr, (void
> *)virt_address+count, memcpysize);
>
> printk("OS_Linux32_Xfer_SG_Page: calling kunmap_atomic, "
> "calling memcpy linux_vaddr = 0x%lx\n",
> linux_vaddr);
>
> kunmap_atomic(linux_vaddr, kmap_flags);
>
>
> --
> Don Brace
> SPSN Linux Development
> Hewlett-Packard Company
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

konrad at darnok

Dec 14, 2011, 1:34 PM

Post #3 of 12 (519 views)

Permalink

On Mon, Dec 12, 2011 at 03:16:24PM +0000, Jan Beulich wrote:
> >>> On 07.12.11 at 22:47, dbrace <dab@hp.com> wrote:
> > kernel: Linux 360 2.6.32.12-0.7-xen #1 SMP 2010-05-20 11:14:20 +0200
> > i686 i686 i386 GNU/Linux
>
> Pretty outdated.
>
> > I am having issues with some code that uses kmap_atomic(). I am getting:
> >
> > BUG: unable to handle kernel paging request at 23a1a000 (which is the
> > address returned from kmap_atomic().)
>
> For a valid high memory page this is surely not being returned by
> kmap_atomic(), so we have to assume that what you started with is
> the address of a low memory one.
>
> > The same code works when running on non-XEN 32bit kernels so I am
> > wondering why this does not work under
> > XEN kernels. Is there a different approach that I need to take for 32bit
> > XEN kernels?
>
> No, but ...
>
> > I really only need to do this code segment if the memory address is a
> > high memory address. Is there a MACRO or function
> > that can help me determine this?
>
> PageHighMem().
>
> > Here is a code snippet:
> >
> > void *linux_vaddr = NULL; /* kmapped temporary
> > virtual address */
> > int linux_page_offset = 0; /* offset in page */
> > int count = 0; /* bytes left to
> > transfer */
> > int left = byte_count; /* number of bytes left
> > to transfer */
> > int memcpysize = 0; /* current size to
> > transfer */
> > struct page *linux_page = NULL; /* calculated page */
> > int kmap_flags = 0;
> >
> > linux_page = __pfn_to_page(physical_address >> PAGE_SHIFT);
>
> ... without knowing where you got physical_address from it's impossible
> to tell whether your problem is that under Xen physical and machine
> (sometimes called "bus") addresses being distinct (end hence you're
> possibly lacking a translation between the two).

I don't know much about the SLES kernels, but if it uses lazy page
updating it might be hitting:

2cd1c8d x86/paravirt: PTE updates in k(un)map_atomic need to be
synchronous, regardless of lazy_mmu mode

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

dab at hp

Dec 14, 2011, 4:47 PM

Post #4 of 12 (532 views)

Permalink

>>... without knowing where you got physical_address from it's impossible
>>to tell whether your problem is that under Xen physical and machine
>>(sometimes called "bus") addresses being distinct (end hence you're
??possibly lacking a translation between the two).

Is there a way to tell from the address is this is happening?

On 12/12/2011 09:16 AM, Jan Beulich wrote:
>>>> On 07.12.11 at 22:47, dbrace<dab@hp.com> wrote:
>> kernel: Linux 360 2.6.32.12-0.7-xen #1 SMP 2010-05-20 11:14:20 +0200
>> i686 i686 i386 GNU/Linux
> Pretty outdated.
>
>> I am having issues with some code that uses kmap_atomic(). I am getting:
>>
>> BUG: unable to handle kernel paging request at 23a1a000 (which is the
>> address returned from kmap_atomic().)
> For a valid high memory page this is surely not being returned by
> kmap_atomic(), so we have to assume that what you started with is
> the address of a low memory one.
>
>> The same code works when running on non-XEN 32bit kernels so I am
>> wondering why this does not work under
>> XEN kernels. Is there a different approach that I need to take for 32bit
>> XEN kernels?
> No, but ...
>
>> I really only need to do this code segment if the memory address is a
>> high memory address. Is there a MACRO or function
>> that can help me determine this?
> PageHighMem().
>
>> Here is a code snippet:
>>
>> void *linux_vaddr = NULL; /* kmapped temporary
>> virtual address */
>> int linux_page_offset = 0; /* offset in page */
>> int count = 0; /* bytes left to
>> transfer */
>> int left = byte_count; /* number of bytes left
>> to transfer */
>> int memcpysize = 0; /* current size to
>> transfer */
>> struct page *linux_page = NULL; /* calculated page */
>> int kmap_flags = 0;
>>
>> linux_page = __pfn_to_page(physical_address>> PAGE_SHIFT);
> ... without knowing where you got physical_address from it's impossible
> to tell whether your problem is that under Xen physical and machine
> (sometimes called "bus") addresses being distinct (end hence you're
> possibly lacking a translation between the two).
>
> Jan
>
>> linux_page_offset = (physical_address&
>> 0x0000000000000FFFULL);
>> memcpysize = min((PAGE_SIZE - linux_page_offset), left);
>>
>> kmap_flags = KM_USER0;
>>
>> linux_vaddr = kmap_atomic(linux_page, kmap_flags) +
>> linux_page_offset;
>>
>> printk("%s: called kmap_atomic, "
>> "calling memcpy linux_vaddr = 0x%x virt_address
>> = 0x%x count = 0x%x\n",
>> __func__, linux_vaddr, virt_address, count);
>> /*
>> * Either need to copy to a kmapped destination
>> * or a kmapped source.
>> */
>> if (type == 0) // Write to s/g element, dest virtual
>> addr was known.
>> memcpy((void *)virt_address+count, (void
>> *)linux_vaddr, memcpysize);
>> else // Source virt. address was known.
>> memcpy((void *)linux_vaddr, (void
>> *)virt_address+count, memcpysize);
>>
>> printk("OS_Linux32_Xfer_SG_Page: calling kunmap_atomic, "
>> "calling memcpy linux_vaddr = 0x%lx\n",
>> linux_vaddr);
>>
>> kunmap_atomic(linux_vaddr, kmap_flags);
>>
>>
>> --
>> Don Brace
>> SPSN Linux Development
>> Hewlett-Packard Company
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xensource.com
>> http://lists.xensource.com/xen-devel
>
>

--
Don Brace
SPSN Linux Development
Hewlett-Packard Company

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

JBeulich at suse

Dec 15, 2011, 12:43 AM

Post #5 of 12 (522 views)

Permalink

>>> On 14.12.11 at 22:34, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> I don't know much about the SLES kernels, but if it uses lazy page
> updating it might be hitting:
>
> 2cd1c8d x86/paravirt: PTE updates in k(un)map_atomic need to be
> synchronous, regardless of lazy_mmu mode

We use lazy mode, but only in non-IRQ (and theoretically
preemption-enabled; theoretically because PREEMPT isn't a
permitted config option in the forward ported kernels) context.
That is, we'd be susceptible to this problem only if within a
lazy-mode enabled section k{,un}map_atomic were used
synchronously (which I'd consider a coding mistake).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

JBeulich at suse

Dec 15, 2011, 12:45 AM

Post #6 of 12 (506 views)

Permalink

>>> On 15.12.11 at 01:47, dbrace <dab@hp.com> wrote:
>> >... without knowing where you got physical_address from it's impossible
>>>to tell whether your problem is that under Xen physical and machine
>>>(sometimes called "bus") addresses being distinct (end hence you're
> ??possibly lacking a translation between the two).
>
>
> Is there a way to tell from the address is this is happening?

It's likely, but as said in my earlier reply - "without knowing where you
got physical_address from it's impossible to tell".

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

dab at hp

Dec 15, 2011, 8:15 AM

Post #7 of 12 (512 views)

Permalink

I do not understand what this means:
" That is, we'd be susceptible to this problem only if within a
lazy-mode enabled section k{,un}map_atomic were used
synchronously (which I'd consider a coding mistake)."

Could you elaborate on this?

Here is what I am trying to do.
I am in a LLD I have bus addresses that are DMA-able that I would normally
DMA to except this driver is emulating hardware in software and so it does
not need to DMA is needs to transfer the data with the CPU.

On non-XEN 32bit kernels I use kmap_atomic() and it handles both highmem addresses
and non highmem addresses with no issues.

When I am running 32bit XEN kernels I run into issues like this:
"BUG: unable to handle kernel paging request at 9822bf40"

-----Original Message-----
From: Jan Beulich [mailto:JBeulich@suse.com]
Sent: Thursday, December 15, 2011 2:44 AM
To: Konrad Rzeszutek Wilk
Cc: Brace, Don; xen-devel@lists.xensource.com
Subject: Re: [Xen-devel] kmap_atomic issue with SLES11SP1 32bit XEN driver code

>>> On 14.12.11 at 22:34, Konrad Rzeszutek Wilk <konrad@darnok.org> wrote:
> I don't know much about the SLES kernels, but if it uses lazy page
> updating it might be hitting:
>
> 2cd1c8d x86/paravirt: PTE updates in k(un)map_atomic need to be
> synchronous, regardless of lazy_mmu mode

We use lazy mode, but only in non-IRQ (and theoretically
preemption-enabled; theoretically because PREEMPT isn't a
permitted config option in the forward ported kernels) context.
That is, we'd be susceptible to this problem only if within a
lazy-mode enabled section k{,un}map_atomic were used
synchronously (which I'd consider a coding mistake).

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

JBeulich at suse

Dec 15, 2011, 8:37 AM

Post #8 of 12 (512 views)

Permalink

>>> On 15.12.11 at 17:15, "Brace, Don" <dab@hp.com> wrote:
> I do not understand what this means:
> " That is, we'd be susceptible to this problem only if within a
> lazy-mode enabled section k{,un}map_atomic were used
> synchronously (which I'd consider a coding mistake)."
>
> Could you elaborate on this?

Not sure what additional explanation is needed. kmap_atomic() is
intended to be used in atomic context, and if you use it elsewhere
that's a mis-use.

> Here is what I am trying to do.
> I am in a LLD I have bus addresses that are DMA-able that I would normally
> DMA to except this driver is emulating hardware in software and so it does
> not need to DMA is needs to transfer the data with the CPU.

So this makes clear that what I told you about a possibly missing
translation is likely the culprit. Quoting from your original mail

linux_page = __pfn_to_page(physical_address >> PAGE_SHIFT);

I have to assume that physical_address really isn't a physical address,
but a bus one. Hence you first need to translate it to a physical one
before passing it to __pfn_to_page().

> On non-XEN 32bit kernels I use kmap_atomic() and it handles both highmem
> addresses and non highmem addresses with no issues.

And so does it on Xen.

> When I am running 32bit XEN kernels I run into issues like this:
> "BUG: unable to handle kernel paging request at 9822bf40"

Because you pass in a random struct page *.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

dab at hp

Dec 15, 2011, 3:19 PM

Post #9 of 12 (510 views)

Permalink

Can you give me some URLs to documentation that can help me understand
my issue?

Is sounds like I cannot go back to the correct virtual address after I
map it for DMA.

On 12/15/2011 10:37 AM, Jan Beulich wrote:
>>>> On 15.12.11 at 17:15, "Brace, Don"<dab@hp.com> wrote:
>> I do not understand what this means:
>> " That is, we'd be susceptible to this problem only if within a
>> lazy-mode enabled section k{,un}map_atomic were used
>> synchronously (which I'd consider a coding mistake)."
>>
>> Could you elaborate on this?
> Not sure what additional explanation is needed. kmap_atomic() is
> intended to be used in atomic context, and if you use it elsewhere
> that's a mis-use.
>
>> Here is what I am trying to do.
>> I am in a LLD I have bus addresses that are DMA-able that I would normally
>> DMA to except this driver is emulating hardware in software and so it does
>> not need to DMA is needs to transfer the data with the CPU.
> So this makes clear that what I told you about a possibly missing
> translation is likely the culprit. Quoting from your original mail
>
> linux_page = __pfn_to_page(physical_address>> PAGE_SHIFT);
>
> I have to assume that physical_address really isn't a physical address,
> but a bus one. Hence you first need to translate it to a physical one
> before passing it to __pfn_to_page().
>
>> On non-XEN 32bit kernels I use kmap_atomic() and it handles both highmem
>> addresses and non highmem addresses with no issues.
> And so does it on Xen.
>
>> When I am running 32bit XEN kernels I run into issues like this:
>> "BUG: unable to handle kernel paging request at 9822bf40"
> Because you pass in a random struct page *.
>
> Jan
>

--
Don Brace
SPSN Linux Development
Hewlett-Packard Company

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

JBeulich at suse

Dec 16, 2011, 12:51 AM

Post #10 of 12 (510 views)

Permalink

>>> On 16.12.11 at 00:19, dbrace <dab@hp.com> wrote:
> Can you give me some URLs to documentation that can help me understand
> my issue?

I'm not aware of any.

> Is sounds like I cannot go back to the correct virtual address after I
> map it for DMA.

Is it really a problem to insert a proper bus_to_phys() (or mfn_to_pfn()
if you are willing to tolerate Xen-specific code) translation?

Jan

> On 12/15/2011 10:37 AM, Jan Beulich wrote:
>>>>> On 15.12.11 at 17:15, "Brace, Don"<dab@hp.com> wrote:
>>> I do not understand what this means:
>>> " That is, we'd be susceptible to this problem only if within a
>>> lazy-mode enabled section k{,un}map_atomic were used
>>> synchronously (which I'd consider a coding mistake)."
>>>
>>> Could you elaborate on this?
>> Not sure what additional explanation is needed. kmap_atomic() is
>> intended to be used in atomic context, and if you use it elsewhere
>> that's a mis-use.
>>
>>> Here is what I am trying to do.
>>> I am in a LLD I have bus addresses that are DMA-able that I would normally
>>> DMA to except this driver is emulating hardware in software and so it does
>>> not need to DMA is needs to transfer the data with the CPU.
>> So this makes clear that what I told you about a possibly missing
>> translation is likely the culprit. Quoting from your original mail
>>
>> linux_page = __pfn_to_page(physical_address>> PAGE_SHIFT);
>>
>> I have to assume that physical_address really isn't a physical address,
>> but a bus one. Hence you first need to translate it to a physical one
>> before passing it to __pfn_to_page().
>>
>>> On non-XEN 32bit kernels I use kmap_atomic() and it handles both highmem
>>> addresses and non highmem addresses with no issues.
>> And so does it on Xen.
>>
>>> When I am running 32bit XEN kernels I run into issues like this:
>>> "BUG: unable to handle kernel paging request at 9822bf40"
>> Because you pass in a random struct page *.
>>
>> Jan
>>
>
> --
> Don Brace
> SPSN Linux Development
> Hewlett-Packard Company

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

konrad at darnok

Dec 16, 2011, 7:16 AM

Post #11 of 12 (512 views)

Permalink

On Thu, Dec 15, 2011 at 05:19:32PM -0600, dbrace wrote:
> Can you give me some URLs to documentation that can help me understand
> my issue?
>
> Is sounds like I cannot go back to the correct virtual address after I
> map it for DMA.

Well yes. That assumption that virt_to_phys == phys_to_virt is not valid
anymore. The obvious solution to your problem is to carry a _both_
values - the bus address (the one you get from the DMA API), and the
virtual address (the one you get from alloc_page). That is how all the
drivers do it.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: kmap_atomic issue with SLES11SP1 32bit XEN driver code [ In reply to ]

JBeulich at suse

Dec 18, 2011, 11:52 PM

Post #12 of 12 (508 views)

Permalink

>>> On 15.12.11 at 01:47, dbrace <dab@hp.com> wrote:
>> >... without knowing where you got physical_address from it's impossible
>>>to tell whether your problem is that under Xen physical and machine
>>>(sometimes called "bus") addresses being distinct (end hence you're
> ??possibly lacking a translation between the two).
>
>
> Is there a way to tell from the address is this is happening?

Not really - these are distinct address spaces, and hence generally
any address can be valid in both. Seeing the initial memory map(s)
one may be able to disambiguate a few regions, but that's only
those where the machine memory really represents MMIO of some
sort.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel