Mailing List Archive: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

[PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN

liyihang9 at huawei

Mar 26, 2024, 5:43 AM

Post #1 of 11 (55 views)

This series [1] reducing the kmalloc() minimum alignment on arm64 to 8
(from 128). In libsas, this will cause SMP requests to be 8-byte-aligned
through kmalloc() allocation. However, for the hisi_sas hardware, all
commands address must be 16-byte-aligned. Otherwise, the commands fail to
be executed.

ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request.

Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1]
Signed-off-by: Yihang Li <liyihang9@huawei.com>
---
Changes since v1:
- Directly modify alloc_smp_req() instead of using handler callback.
---
drivers/scsi/libsas/sas_expander.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
index a2204674b680..941abc7298df 100644
--- a/drivers/scsi/libsas/sas_expander.c
+++ b/drivers/scsi/libsas/sas_expander.c
@@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size,

static inline void *alloc_smp_req(int size)
{
- u8 *p = kzalloc(size, GFP_KERNEL);
+ u8 *p;
+
+ size = ALIGN(size, ARCH_DMA_MINALIGN);
+ p = kzalloc(size, GFP_KERNEL);
if (p)
p[0] = SMP_REQUEST;
return p;
--
2.33.0

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

dlemoal at kernel

Mar 26, 2024, 6:14 AM

Post #2 of 11 (55 views)

On 3/26/24 21:43, Yihang Li wrote:
> This series [1] reducing the kmalloc() minimum alignment on arm64 to 8
> (from 128). In libsas, this will cause SMP requests to be 8-byte-aligned
> through kmalloc() allocation. However, for the hisi_sas hardware, all
> commands address must be 16-byte-aligned. Otherwise, the commands fail to
> be executed.
>
> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
> operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request.
>
> Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1]
> Signed-off-by: Yihang Li <liyihang9@huawei.com>
> ---
> Changes since v1:
> - Directly modify alloc_smp_req() instead of using handler callback.
> ---
> drivers/scsi/libsas/sas_expander.c | 5 ++++-
> 1 file changed, 4 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
> index a2204674b680..941abc7298df 100644
> --- a/drivers/scsi/libsas/sas_expander.c
> +++ b/drivers/scsi/libsas/sas_expander.c
> @@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size,
>
> static inline void *alloc_smp_req(int size)
> {
> - u8 *p = kzalloc(size, GFP_KERNEL);
> + u8 *p;
> +
> + size = ALIGN(size, ARCH_DMA_MINALIGN);
> + p = kzalloc(size, GFP_KERNEL);

Nit: why not:

p = kzalloc(ALIGN(size, ARCH_DMA_MINALIGN), GFP_KERNEL);

> if (p)
> p[0] = SMP_REQUEST;
> return p;

Otherwise looks OK to me.

John,

Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be
GFP_NOIO... Is this ever called in the IO path or error recovery ?

--
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

john.g.garry at oracle

Mar 26, 2024, 6:32 AM

Post #3 of 11 (55 views)

On 26/03/2024 13:14, Damien Le Moal wrote:
> On 3/26/24 21:43, Yihang Li wrote:
>> This series [1] reducing the kmalloc() minimum alignment on arm64 to 8
>> (from 128). In libsas, this will cause SMP requests to be 8-byte-aligned
>> through kmalloc() allocation. However, for the hisi_sas hardware, all
>> commands address must be 16-byte-aligned.
> Otherwise, the commands fail to
>> be executed.
>>
>> ARCH_DMA_MINALIGN represents the minimum (static) alignment for safe DMA
>> operations, so use ARCH_DMA_MINALIGN as the alignment for SMP request.
>>
>> Link: https://lkml.kernel.org/r/20230612153201.554742-1-catalin.marinas@arm.com [1]
>> Signed-off-by: Yihang Li <liyihang9@huawei.com>
>> ---
>> Changes since v1:
>> - Directly modify alloc_smp_req() instead of using handler callback.
>> ---
>> drivers/scsi/libsas/sas_expander.c | 5 ++++-
>> 1 file changed, 4 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c
>> index a2204674b680..941abc7298df 100644
>> --- a/drivers/scsi/libsas/sas_expander.c
>> +++ b/drivers/scsi/libsas/sas_expander.c
>> @@ -135,7 +135,10 @@ static int smp_execute_task(struct domain_device *dev, void *req, int req_size,
>>
>> static inline void *alloc_smp_req(int size)
>> {
>> - u8 *p = kzalloc(size, GFP_KERNEL);
>> + u8 *p;
>> +
>> + size = ALIGN(size, ARCH_DMA_MINALIGN);

If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and
not 16B as minimum alignment?

Or are we really talking about an arch requirement?

>> + p = kzalloc(size, GFP_KERNEL);
>
> Nit: why not:
>
> p = kzalloc(ALIGN(size, ARCH_DMA_MINALIGN), GFP_KERNEL);
>
>> if (p)
>> p[0] = SMP_REQUEST;
>> return p;
>
> Otherwise looks OK to me.
>
> John,
>
> Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be
> GFP_NOIO... Is this ever called in the IO path or error recovery ?
>

These should not be called in the IO path - as they are management
functions. But I am quite confident that they can be called in SCSI
error handling (for libsas).

Thanks,
John

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

dlemoal at kernel

Mar 26, 2024, 6:40 AM

Post #4 of 11 (55 views)

On 3/26/24 22:32, John Garry wrote:
>> John,
>>
>> Unrelated to this patch, but I wonder if the GFP_KERNEL used here shouldn't be
>> GFP_NOIO... Is this ever called in the IO path or error recovery ?
>>
>
> These should not be called in the IO path - as they are management
> functions. But I am quite confident that they can be called in SCSI
> error handling (for libsas).

So it sounds like GFP_NOIO would be a lot safer...

>
> Thanks,
> John

--
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

hch at infradead

Mar 27, 2024, 11:35 PM

Post #5 of 11 (53 views)

On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote:
> > > + u8 *p;
> > > +
> > > + size = ALIGN(size, ARCH_DMA_MINALIGN);
>
>
> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not
> 16B as minimum alignment?
>
> Or are we really talking about an arch requirement?

One thing is that we should never allocate unaligned memory for
anything DMA mapped, or data will be corrupted by non-coherent DMA.
So ARCH_DMA_MINALIGN needs to be here. If specific hardware has
further requirements we'll need to communicated it through a field
or op vector.

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

liyihang9 at huawei

Mar 27, 2024, 11:59 PM

Post #6 of 11 (53 views)

On 2024/3/28 14:35, Christoph Hellwig wrote:
> On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote:
>>>> + u8 *p;
>>>> +
>>>> + size = ALIGN(size, ARCH_DMA_MINALIGN);
>>
>>
>> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not
>> 16B as minimum alignment?
>>
>> Or are we really talking about an arch requirement?
>
> One thing is that we should never allocate unaligned memory for
> anything DMA mapped, or data will be corrupted by non-coherent DMA.
> So ARCH_DMA_MINALIGN needs to be here. If specific hardware has
> further requirements we'll need to communicated it through a field
> or op vector.

Got it. Looks like it's still going to be aligned to ARCH_DMA_MINALIGN.

Thanks,
Yihang

>
>
> .
>

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

dlemoal at kernel

Mar 28, 2024, 12:23 AM

Post #7 of 11 (51 views)

On 3/28/24 15:59, Yihang Li wrote:
>
>
> On 2024/3/28 14:35, Christoph Hellwig wrote:
>> On Tue, Mar 26, 2024 at 01:32:09PM +0000, John Garry wrote:
>>>>> + u8 *p;
>>>>> +
>>>>> + size = ALIGN(size, ARCH_DMA_MINALIGN);
>>>
>>>
>>> If this is a hisi_sas requirement, then why use ARCH_DMA_MINALIGN and not
>>> 16B as minimum alignment?
>>>
>>> Or are we really talking about an arch requirement?
>>
>> One thing is that we should never allocate unaligned memory for
>> anything DMA mapped, or data will be corrupted by non-coherent DMA.
>> So ARCH_DMA_MINALIGN needs to be here. If specific hardware has
>> further requirements we'll need to communicated it through a field
>> or op vector.
>
> Got it. Looks like it's still going to be aligned to ARCH_DMA_MINALIGN.

But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN
down to 8B but hisi driver needs at least 16 ?

So in the end, you need something like:

size = ALIGN(size, max(16, ARCH_DMA_MINALIGN));

no ?

And define a macro for the "16" value to document it.
Something like:

#define SAS_SMP_REQ_ALIGN 16

Not sure about the name... Naming is hard :)

>
> Thanks,
> Yihang
>
>>
>>
>> .
>>

--
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

hch at infradead

Mar 28, 2024, 12:29 AM

Post #8 of 11 (51 views)

On Thu, Mar 28, 2024 at 04:23:22PM +0900, Damien Le Moal wrote:
> But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN
> down to 8B but hisi driver needs at least 16 ?
>
> So in the end, you need something like:
>
> size = ALIGN(size, max(16, ARCH_DMA_MINALIGN));
>
> no ?

I don't think we ever have an 8 byte dma minalign. With 8-byte
aligned addresses dma_mapping_error could run into problems.

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

dlemoal at kernel

Mar 28, 2024, 12:36 AM

Post #9 of 11 (51 views)

On 3/28/24 16:29, Christoph Hellwig wrote:
> On Thu, Mar 28, 2024 at 04:23:22PM +0900, Damien Le Moal wrote:
>> But I thought that the original issue was that some arch have ARCH_DMA_MINALIGN
>> down to 8B but hisi driver needs at least 16 ?
>>
>> So in the end, you need something like:
>>
>> size = ALIGN(size, max(16, ARCH_DMA_MINALIGN));
>>
>> no ?
>
> I don't think we ever have an 8 byte dma minalign. With 8-byte
> aligned addresses dma_mapping_error could run into problems.

My bad: it is kmalloc() that can return something aligned to 8B...
So "size = ALIGN(size, ARCH_DMA_MINALIGN);" is the right thing to do.

--
Damien Le Moal
Western Digital Research

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

hch at infradead

Mar 28, 2024, 12:45 AM

Post #10 of 11 (51 views)

On Thu, Mar 28, 2024 at 04:36:12PM +0900, Damien Le Moal wrote:
> My bad: it is kmalloc() that can return something aligned to 8B...

Yes, that's new on arm64, and possibly soon riscv.

Re: [PATCH v2] scsi: libsas: Allocation SMP request is aligned to ARCH_DMA_MINALIGN [ In reply to ]

liyihang9 at huawei

Mar 28, 2024, 1:00 AM

Post #11 of 11 (51 views)

On 2024/3/28 15:45, Christoph Hellwig wrote:
> On Thu, Mar 28, 2024 at 04:36:12PM +0900, Damien Le Moal wrote:
>> My bad: it is kmalloc() that can return something aligned to 8B...
>
> Yes, that's new on arm64, and possibly soon riscv.

Thanks for the discussion, I will still aligned to ARCH_DMA_MINALIGN in v4.