Mailing List Archive

On Sun, Sep 25, 2011 at 04:45:39PM +0800, MaoXiaoyun wrote:
>
>
> Hi:
>
> We met an ext4 BUG_ON in extents.c:1716 which crash kernel flush thread, and result in disk unvailiable.
>
> BUG details refer to: http://www.gossamer-threads.com/lists/xen/devel/217091?do=post_view_threaded
>
> Attached is the fix, verified in our env.

So.. you are asking for this upstream git commit to be back-ported to 2.6.32, right?

>
> Without this patch, more than 3 servers hit BUG_ON in our hundreds of servers every day.
>
>
> many thanks.



_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

----------------------------------------
> Date: Mon, 26 Sep 2011 10:28:08 -0400
> From: konrad.wilk@oracle.com
> To: tinnycloud@hotmail.com
> CC: linux-ext4@vger.kernel.org; xen-devel@lists.xensource.com; tytso@mit.edu; jack@suse.cz
> Subject: Re: [patch 1/1] ext4-fix-dirty-extent-when-origin-leaf-extent-reac.patch
>
> On Sun, Sep 25, 2011 at 04:45:39PM +0800, MaoXiaoyun wrote:
> >
> >
> > Hi:
> >
> > We met an ext4 BUG_ON in extents.c:1716 which crash kernel flush thread, and result in disk unvailiable.
> >
> > BUG details refer to: http://www.gossamer-threads.com/lists/xen/devel/217091?do=post_view_threaded
> >
> > Attached is the fix, verified in our env.
>
> So.. you are asking for this upstream git commit to be back-ported to 2.6.32, right?
>

The patch is for 2.6.39. It can be patched on 2.6.32 too.
Thanks.

> >
> > Without this patch, more than 3 servers hit BUG_ON in our hundreds of servers every day.
> >
> >
> > many thanks.
>
>

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

>>> On 27.09.11 at 04:22, MaoXiaoyun <tinnycloud@hotmail.com> wrote:

>
>
> ----------------------------------------
>> Date: Mon, 26 Sep 2011 10:28:08 -0400
>> From: konrad.wilk@oracle.com
>> To: tinnycloud@hotmail.com
>> CC: linux-ext4@vger.kernel.org; xen-devel@lists.xensource.com; tytso@mit.edu;
> jack@suse.cz
>> Subject: Re: [patch 1/1] ext4-fix-dirty-extent-when-origin-leaf-extent-reac.patch
>>
>> On Sun, Sep 25, 2011 at 04:45:39PM +0800, MaoXiaoyun wrote:
>> >
>> >
>> > Hi:
>> >
>> > We met an ext4 BUG_ON in extents.c:1716 which crash kernel flush thread,
> and result in disk unvailiable.
>> >
>> > BUG details refer to:
> http://www.gossamer-threads.com/lists/xen/devel/217091?do=post_view_threaded
>> >
>> > Attached is the fix, verified in our env.
>>
>> So.. you are asking for this upstream git commit to be back-ported to 2.6.32,
> right?
>>
>
> The patch is for 2.6.39. It can be patched on 2.6.32 too.
> Thanks.

So why don't you suggest applying this to the stable tree maintainers
instead? xen-devel really isn't the right forum for this sort of bug fixes,
particularly when the underlying kernel.org tree is still being maintained.

Jan

>> >
>> > Without this patch, more than 3 servers hit BUG_ON in our hundreds of
> servers every day.
>> >
>> >
>> > many thanks.
>>
>>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel




_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/27/2011 05:09 PM, Jan Beulich wrote:
>>>> On 27.09.11 at 04:22, MaoXiaoyun <tinnycloud@hotmail.com> wrote:
>
>>
>>
>> ----------------------------------------
>>> Date: Mon, 26 Sep 2011 10:28:08 -0400
>>> From: konrad.wilk@oracle.com
>>> To: tinnycloud@hotmail.com
>>> CC: linux-ext4@vger.kernel.org; xen-devel@lists.xensource.com; tytso@mit.edu;
>> jack@suse.cz
>>> Subject: Re: [patch 1/1] ext4-fix-dirty-extent-when-origin-leaf-extent-reac.patch
>>>
>>> On Sun, Sep 25, 2011 at 04:45:39PM +0800, MaoXiaoyun wrote:
>>>>
>>>>
>>>> Hi:
>>>>
>>>> We met an ext4 BUG_ON in extents.c:1716 which crash kernel flush thread,
>> and result in disk unvailiable.
>>>>
>>>> BUG details refer to:
>> http://www.gossamer-threads.com/lists/xen/devel/217091?do=post_view_threaded
>>>>
>>>> Attached is the fix, verified in our env.
>>>
>>> So.. you are asking for this upstream git commit to be back-ported to 2.6.32,
>> right?
>>>
>>
>> The patch is for 2.6.39. It can be patched on 2.6.32 too.
>> Thanks.
>
> So why don't you suggest applying this to the stable tree maintainers
> instead? xen-devel really isn't the right forum for this sort of bug fixes,
> particularly when the underlying kernel.org tree is still being maintained.
AFAIK, the upstream linux kernel doesn't have this problem because this
part of codes have been refactored. So I am not sure whether Greg KH
will accept it or not.

btw, I don't think the fix is appropriate. One of my colleague is
working out another patch to resolve this(I will ask him to post the
patch when it is ready). And we will contact Redhat for considering
merging it to the enterprise kernel.

Thanks
Tao

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Mon, Sep 26, 2011 at 10:28:08AM -0400, Konrad Rzeszutek Wilk wrote:
> >
> > Attached is the fix, verified in our env.
>
> So.. you are asking for this upstream git commit to be back-ported
> to 2.6.32, right?

I'm curious --- is there a good reason why Xen users are using an
upstream 2.6.32 kernel? If they are using a distro kernel, fine, but
then the distro kernel should be providing the support. But at this
point, 2.6.32 is so positively *ancient* that, I'm personally not
interesting in providing free, unpaid distro support for users who
aren't willing to either (a) pay $$$ and get a supported distro
kernel, or (b) use a much more modern kernel. At this point, Guest
and Host Xen support is available in 3.0 kernels, so there's really no
excuse, right?

- Ted

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 27/09/2011 21:35, Ted Ts'o wrote:
> On Mon, Sep 26, 2011 at 10:28:08AM -0400, Konrad Rzeszutek Wilk wrote:
>>>
>>> Attached is the fix, verified in our env.
>>
>> So.. you are asking for this upstream git commit to be back-ported
>> to 2.6.32, right?
>
> I'm curious --- is there a good reason why Xen users are using an
> upstream 2.6.32 kernel? If they are using a distro kernel, fine, but
> then the distro kernel should be providing the support. But at this
> point, 2.6.32 is so positively *ancient* that, I'm personally not
> interesting in providing free, unpaid distro support for users who
> aren't willing to either (a) pay $$$ and get a supported distro
> kernel, or (b) use a much more modern kernel. At this point, Guest
> and Host Xen support is available in 3.0 kernels, so there's really no
> excuse, right?
>
> - Ted
>

In my case, for Dom0 I use the 2.6.32 kernel from my distrib, because it's stable.
I have stability problems with the kernel 3.0 on Dom0, and as I haven't physical access neither kvm or serial port, I don't know what to report... It just hang randomly, without any line in logs.

Does the Xen support on 3.0 kernels should be considered stable ?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

----------------------------------------
> Date: Tue, 27 Sep 2011 15:35:23 -0400
> From: tytso@mit.edu
> To: konrad.wilk@oracle.com
> CC: tinnycloud@hotmail.com; linux-ext4@vger.kernel.org; xen-devel@lists.xensource.com; jack@suse.cz
> Subject: Re: [patch 1/1] ext4-fix-dirty-extent-when-origin-leaf-extent-reac.patch
>
> On Mon, Sep 26, 2011 at 10:28:08AM -0400, Konrad Rzeszutek Wilk wrote:
> > >
> > > Attached is the fix, verified in our env.
> >
> > So.. you are asking for this upstream git commit to be back-ported
> > to 2.6.32, right?
>
> I'm curious --- is there a good reason why Xen users are using an
> upstream 2.6.32 kernel? If they are using a distro kernel, fine, but
> then the distro kernel should be providing the support. But at this
> point, 2.6.32 is so positively *ancient* that, I'm personally not
> interesting in providing free, unpaid distro support for users who
> aren't willing to either (a) pay $$$ and get a supported distro
> kernel, or (b) use a much more modern kernel. At this point, Guest
> and Host Xen support is available in 3.0 kernels, so there's really no
> excuse, right?

Mmm...

We first met this bug at pvops kernel(jeremy's tree, 2.6.32.36).

We failed to find any related fix from google, so we debug the bug ourself.
Fortunately, we located root cause and thought some other xen users might
have this problem as well, that's why we sent out the fix to Xen-devel.

We go through the code from 2.6.32 - 2.6.39, this bug exists.
People who use *ancient* kernel need this.

Thanks.

> - Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Sep 28, 2011 at 01:41:30AM +0200, Olivier B. wrote:
> On 27/09/2011 21:35, Ted Ts'o wrote:
> >On Mon, Sep 26, 2011 at 10:28:08AM -0400, Konrad Rzeszutek Wilk wrote:
> >>>
> >>> Attached is the fix, verified in our env.
> >>
> >>So.. you are asking for this upstream git commit to be back-ported
> >>to 2.6.32, right?
> >
> >I'm curious --- is there a good reason why Xen users are using an
> >upstream 2.6.32 kernel? If they are using a distro kernel, fine, but
> >then the distro kernel should be providing the support. But at this
> >point, 2.6.32 is so positively *ancient* that, I'm personally not
> >interesting in providing free, unpaid distro support for users who
> >aren't willing to either (a) pay $$$ and get a supported distro
> >kernel, or (b) use a much more modern kernel. At this point, Guest
> >and Host Xen support is available in 3.0 kernels, so there's really no
> >excuse, right?
> >
> > - Ted
> >
>
> In my case, for Dom0 I use the 2.6.32 kernel from my distrib, because it's stable.
> I have stability problems with the kernel 3.0 on Dom0, and as I haven't physical access neither kvm or serial port, I don't know what to report... It just hang randomly, without any line in logs.

No physical access? No IPMI? No SOL?
>
> Does the Xen support on 3.0 kernels should be considered stable ?

Yes - it should be considered stable (3.0.4 at least, with one particular
patch that is going in 3.0.5: https://lkml.org/lkml/2011/9/2/114).

Please help me find whatever is causing your crash.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On 09/27/2011 12:35 PM, Ted Ts'o wrote:
> On Mon, Sep 26, 2011 at 10:28:08AM -0400, Konrad Rzeszutek Wilk wrote:
>>>
>>> Attached is the fix, verified in our env.
>> So.. you are asking for this upstream git commit to be back-ported
>> to 2.6.32, right?
> I'm curious --- is there a good reason why Xen users are using an
> upstream 2.6.32 kernel? If they are using a distro kernel, fine, but
> then the distro kernel should be providing the support. But at this
> point, 2.6.32 is so positively *ancient* that, I'm personally not
> interesting in providing free, unpaid distro support for users who
> aren't willing to either (a) pay $$$ and get a supported distro
> kernel, or (b) use a much more modern kernel. At this point, Guest
> and Host Xen support is available in 3.0 kernels, so there's really no
> excuse, right?

The 2.6.32.x-based kernel has been the preferred "stable" kernel for Xen
users for a while, and it is still considered to be more stable and
functional than what's upstream (obviously we're trying to fix that).
Also, because many current distros don't support Xen dom0, it has been
an ad-hoc distro kernel.

Since kernel.org 2.6.32 is still considered to be a maintained
long-term-stable kernel, I keep the xen.git version up-to-date with
stable-2.6.32 bugfixes and occasional separate Xen-specific fixes. But
I'd really prefer to avoid having any non-Xen private changes in that
tree, in favour of getting everything from upstream stable.

Do you not consider it worth continuing support of the 2.6.32 stable
tree with respect to ext4?

J

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

On Wed, Sep 28, 2011 at 11:41:11AM -0700, Jeremy Fitzhardinge wrote:
> Since kernel.org 2.6.32 is still considered to be a maintained
> long-term-stable kernel, I keep the xen.git version up-to-date with
> stable-2.6.32 bugfixes and occasional separate Xen-specific fixes. But
> I'd really prefer to avoid having any non-Xen private changes in that
> tree, in favour of getting everything from upstream stable.
>
> Do you not consider it worth continuing support of the 2.6.32 stable
> tree with respect to ext4?

I just don't have the *time* to maintain backports of ext4 fixes to
2.6.32. There have been so many bug fixes to ext4, and some of them
depend on changes in the quota subsystem, so trying to back port them
all would be hellish, and not something I'm willing to do on a
volunteer basis.

I'm busy enough with silly things like trying to help with the
kernel.org getting back on-line, that channelling my stay-really-late
hours to support users who are too cheap to pay distro support fees is
not really a way that I would choose to spend my personal time.

If someone would like to volunteer to be unpaid distro support, that's
great. It's worth it as long as I get to volunteer somebody else's
time. :-)

- Ted


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel