Mailing List Archive

On 2000-11-21T09:41:06,
Joseph Mack <mack.joseph@epa.gov> said:

> RedHat has created a situation which makes them money and stops
> others from working on LVS. Any problem from a person
> with RedHat - can't compile, can't find modules, doesn't work,
> I have to read through and think about. I then have to make a reasonable
> guess as to whether this is a RedHat induced problem or a real problem.

I wouldn't have any problem if the default response to a bugreport from a
person using a distribution kernel would be "talk to your vendor or repatch
your own kernel from ftp.kernel.org", if it appears to be related to that.

That is a reasonable stance to take. The kernel people already do that.

> you don't agree that this is a large load and has erased any good will
> I had towards RedHat?

Well, I can't argue with that obviously, you will know better how you feel
than I do.

> This is why patches that change between kernel releases should be
> treated differently. LVS has been releasing about 2 versions for
> each kernel release for 2 yrs now.

It is _impossible_ to treat that that differently. They ship as part of the
distribution kernel, if you need an update to that before the next official
release, you are on your own.

This is how distributions have always handled this. (The SuSE support database
has hints how to rebuild kernels which work with the distribution, but support
itself is voided unless you have a special support contract)

That makes it all the more important to make LVS part of the shipped kernel,
so people can receive official support from the vendors and the vendors have
to be able to reproduce the problem and certify the kernel as a whole in the
first place.

> > If the distribution shipped reiserfs included, and they tried to patch
> > a newer release of reiserfs into that, that will break too.
> Does reiserfs have full releases between kernels?

Yes.

> If so what do the people who answer questions in their own time about
> reiserfs think about including it in the distribution?

There usually isn't a problem with that. Frankly, I am quite amazed why the
inclusion of LVS into RH's default kernel has created such a load - it
shouldn't be the case.

> I didn't know that individual distributions have that many patches.
> I've assumed that if the standard kernel is good enough for Linus and for
> me then it is good enough for most of us.

This isn't quite true. Distributions all have additional bugfixes, features
(for large machines for one), filesystems (reiserfs, ext3...) which haven't
been accepted into Linus' kernel when the distribution was build.

> How many of these patches are from projects that have production releases
> between kernels.
> How many of them are small (<10 lines) bug fixes?

I don't have the numbers for these.

> > so we have to include it in our kernel.
>
> could you put the patches in a contrib directory and have the patch applied
> as part of the build?

Well, thats what happens. It is applied as part of the build. You _could_
download the src.rpm, swap patches, and rebuild the kernel package. However,
this will take quite some effort (ie merging the patches et al), and also most
kernel RPMs are setup to build much more than just a single kernel.

So they could just as well go ahead and rebuild the kernel with the patches
they need from kernel.org anyway.

> > Having LVS included will work fine for most people
>
> it's the other people that I'm concerned about.

There will always be people who can't get it right. The LVS install file
should tell people

"Hey, if you are getting an error message like this, or if your
distribution already includes LVS, don't try applying the patch on top
of it.

First, try checking with your Linux vendor whether an updated kernel
package with full support is already available.

If you think this revision of the LVS patches provides significant
improvement over the previous ones, or fixes a critical bug for you,
and the vendor doesn't yet provide an updated kernel, please ask them
to do so.

If you need a newer revision than what your distribution supplies,
please grab a clean kernel tree from kernel.org and start from that.
Make sure to apply all other patches you need! "

Do you think this would help?

> LVS has been production level for 2 yrs now, there is no sudden change in
> the quality, reliability or functionality of the LVS code just because it
> has gone to 1.0.0.

Exactly. So there _isn't_ a pressing reason why people who are unable to
should upgrade.

> Rather than SuSE easing the lives of its purchasers, I'd rather SuSE
> first minimised its impact on people developing the software that SuSE uses
> and who are doing it in their spare time.

We do have to balance both. And I don't see a conflict.

But you are essentially asking for vendors not to include any patch on top of
the plain kernel, which just will not work.

> there has only been 1 or 2 of these in the last 2 yrs. This is not a
> significant factor in deciding how to use LVS.

If there isn't a signifcant reason why people should upgrade, they can either
continue to use the distribution supplied kernel - which will most likely work
fine, I have systems in production for almost 1-2 years now without having
needed an upgrade - or go through fetching a kernel upgrade from kernel.org.

Sincerely,
Lars Marowsky-Brée <lmb@suse.de>
Development HA

--
Perfection is our goal, excellence will be tolerated. -- J. Yahl

> On 2000-11-21T09:41:06,
> Joseph Mack <mack.joseph@epa.gov> said:
>
> > RedHat has created a situation which makes them money and stops
> > others from working on LVS. Any problem from a person
> > with RedHat - can't compile, can't find modules, doesn't work,
> > I have to read through and think about. I then have to make a reasonable
> > guess as to whether this is a RedHat induced problem or a real problem.
>
> I wouldn't have any problem if the default response to a bugreport from a
> person using a distribution kernel would be "talk to your vendor or repatch
> your own kernel from ftp.kernel.org", if it appears to be related to that.
>
> That is a reasonable stance to take. The kernel people already do that.

I think this *is* the way to go. Unfortunately, there seems to be a lot
more traffic here than on the Piranha list (which I also follow).

[snip]

> > This is why patches that change between kernel releases should be
> > treated differently. LVS has been releasing about 2 versions for
> > each kernel release for 2 yrs now.
>
> It is _impossible_ to treat that that differently. They ship as part of the
> distribution kernel, if you need an update to that before the next official
> release, you are on your own.
>
> This is how distributions have always handled this. (The SuSE support database
> has hints how to rebuild kernels which work with the distribution, but support
> itself is voided unless you have a special support contract)
>
> That makes it all the more important to make LVS part of the shipped kernel,
> so people can receive official support from the vendors and the vendors have
> to be able to reproduce the problem and certify the kernel as a whole in the
> first place.

I think having LVS included in the kernel *is* the way to go. There are
a LOT of things that require patches to the kernel (VPN support via
PoPToP or FreeSWAN, or updates to things that are included in the
kernel already). I understand Linus wanting to maintain control over
the kernel and not wanting to include non-production quality code in
the kernel. I also understand wanting to keep the kernel lean, clean
and mean. On the other hand, there are a number of things that users
are going to really want/need/demand (VPN, journaling filesystem, USB,
Firewire, Load Balancing Layer 3 switch), and if Linus doesn't include it
in the official kernel, then the distributions are going to meet demand
and put it in the kernel via patches. The whole point of having a
"distribution" is to put this stuff together so the end-users, many of
whom aren't experts, don't have to. Make sure all the required pieces
are there, but then it is on the distributor to support *THAT* configuration.

The problems this leads to, of course, is a proliferation of configurations,
none quite the same. This is a *SERIOUS* problem, but not fatal. It does
make generic support much more difficult though, as Joseph has pointed
out.

> > > If the distribution shipped reiserfs included, and they tried to patch
> > > a newer release of reiserfs into that, that will break too.
> > Does reiserfs have full releases between kernels?
>
> Yes.
>
> > If so what do the people who answer questions in their own time about
> > reiserfs think about including it in the distribution?
>
> There usually isn't a problem with that. Frankly, I am quite amazed why the
> inclusion of LVS into RH's default kernel has created such a load - it
> shouldn't be the case.

I wonder what will happen if Redhat included reiserfs in their distributions?
No offense to Suse, but at least in the US, Redhat is used far more widely
than Suse. In particular, what if the version included with Redhat is
different than that included with Suse, which is different than the
"current stable" release?

> > I didn't know that individual distributions have that many patches.
> > I've assumed that if the standard kernel is good enough for Linus and for
> > me then it is good enough for most of us.
>
> This isn't quite true. Distributions all have additional bugfixes, features
> (for large machines for one), filesystems (reiserfs, ext3...) which haven't
> been accepted into Linus' kernel when the distribution was build.

Exactly, and most end users don't want to have to track this stuff
down, and apply it themselves, often from a tarball instead of a
a package. Many users have no idea what a tar is, what a Makefile
is, or know that "cd <wherever>; ./configure; make install" is a
good first stab at it, etc. This doesn't even bring up the difficulty
of adding multiple (often incompatible) patches to a kernel, compiling
it, ending up with a system that won't boot, and figuring out how to
recover from that. Most users have no interest in even learning how
to do all that. Yet they require the functionality. For example,
a small business might want to have a director that is also a firewall
and VPN gateway (both IPsec and PPTP). I can tell you from bitter
experience this is not an easy thing to construct; *I* wish there
was a distribution that included LVS, IPsec and PPTP VPNS, and a
journaling filesystem, out of the box. I might end up creating
one of my own.

[snip]

> > > so we have to include it in our kernel.
> >
> > could you put the patches in a contrib directory and have the patch applied
> > as part of the build?
>
> Well, thats what happens. It is applied as part of the build. You _could_
> download the src.rpm, swap patches, and rebuild the kernel package. However,
> this will take quite some effort (ie merging the patches et al), and also most
> kernel RPMs are setup to build much more than just a single kernel.
>
> So they could just as well go ahead and rebuild the kernel with the patches
> they need from kernel.org anyway.

This is my experience. Actually, I found it easier to just start with
kernel.org and apply patches than to figure out what all the source RPMs
were, etc (with Redhat). There is not a huge difference between the methods;
I have done both.

> > > Having LVS included will work fine for most people
> >
> > it's the other people that I'm concerned about.
>
> There will always be people who can't get it right. The LVS install file
> should tell people
>
> "Hey, if you are getting an error message like this, or if your
> distribution already includes LVS, don't try applying the patch on top
> of it.
>
> First, try checking with your Linux vendor whether an updated kernel
> package with full support is already available.
>
> If you think this revision of the LVS patches provides significant
> improvement over the previous ones, or fixes a critical bug for you,
> and the vendor doesn't yet provide an updated kernel, please ask them
> to do so.
>
> If you need a newer revision than what your distribution supplies,
> please grab a clean kernel tree from kernel.org and start from that.
> Make sure to apply all other patches you need! "
>
> Do you think this would help?

I certainly do. It is a *LOT* of work to include error checking and
useful error messages, and boring to boot, but it is often also quite
helpful. If I ever get a break from my real job, maybe I could work
on the install process for LVS. I certainly have a lot of experience
in implementing "Jumpstart for Dummies" (obviously a reference taken
from the book series, and only meant to imply that my job is to make
it easier for those who know less about it than me, which is why they
pay me to do it in the first place - many of these folks are NOT dummies).

> > LVS has been production level for 2 yrs now, there is no sudden change in
> > the quality, reliability or functionality of the LVS code just because it
> > has gone to 1.0.0.
>
> Exactly. So there _isn't_ a pressing reason why people who are unable to
> should upgrade.

I concur. Many folks in the "real world" of production continue to use
Oracle 7.3.X on Solaris 2.5.1 or 2.6 even though the far superior Oracle
8i and Solaris 7 and 8 are available.

> > Rather than SuSE easing the lives of its purchasers, I'd rather SuSE
> > first minimised its impact on people developing the software that SuSE uses
> > and who are doing it in their spare time.
>
> We do have to balance both. And I don't see a conflict.
>
> But you are essentially asking for vendors not to include any patch on top of
> the plain kernel, which just will not work.

I agree. If Suse or Redhat don't include the extra patches, then nobody
will use their distributions - they will find somebody who does. I
would, and I probably know a lot more than 90+% of the users of Redhat and
Suse products.

> > there has only been 1 or 2 of these in the last 2 yrs. This is not a
> > significant factor in deciding how to use LVS.
>
> If there isn't a signifcant reason why people should upgrade, they can either
> continue to use the distribution supplied kernel - which will most likely work
> fine, I have systems in production for almost 1-2 years now without having
> needed an upgrade - or go through fetching a kernel upgrade from kernel.org.

Exactly. This does mean supporting multiple revisions of the code, but
once you reach a certain critical mass of users, this just can't be helped.

It also means that Suse and Redhat need to be on the ball about supplying
"must have" updates that fix serious problems.

--
John Cronin

On 2000-11-21T10:43:45,
John Cronin <jsc3@havoc.gtf.org> said:

> The problems this leads to, of course, is a proliferation of configurations,
> none quite the same. This is a *SERIOUS* problem, but not fatal.

This problem gets worse (at least for the distributions) if the users had even
more choices of kernels to run.

> It does make generic support much more difficult though, as Joseph has
> pointed out.

I don't think it is quite that bad.

If the distribution ships LVS in the kernel + corressponding ipvsadm, how is
this support different from supporting someone who patched it in themselves?

Seriously, how often have we seen a question which was traced down to a kernel
bug?

> I wonder what will happen if Redhat included reiserfs in their distributions?
> No offense to Suse, but at least in the US, Redhat is used far more widely
> than Suse. In particular, what if the version included with Redhat is
> different than that included with Suse, which is different than the
> "current stable" release?

I understand this issue.

Yes, having multiple vendors ship different (with regard to features) versions
of the same code _is_ annoying, but it can barely be avoided.

This has already happened with regard to the RAID code for one.

But at least with regard to LVS, the versions aren't _that_ much different so
far, and you can quite easily tell from the LVS version number.

In Europe, it is exactly the other way round BTW ;-)

> I certainly do. It is a *LOT* of work to include error checking and
> useful error messages, and boring to boot, but it is often also quite
> helpful. If I ever get a break from my real job, maybe I could work
> on the install process for LVS.

With Linux 2.4, LVS is "just another netfilter module", so this will become
a lot easier. Just replace the netfilter module.

> I agree. If Suse or Redhat don't include the extra patches, then nobody
> will use their distributions - they will find somebody who does.

Or worse, people will not consider using Linux at all as their OS, because
they can't get the needed features out of the box.

> Exactly. This does mean supporting multiple revisions of the code, but
> once you reach a certain critical mass of users, this just can't be helped.

Most problems relate to general issues with understanding how LVS works. The
kernel patch used for that isn't quite a problem for that.

> It also means that Suse and Redhat need to be on the ball about supplying
> "must have" updates that fix serious problems.

Definetely. If it is a must have update, we have to put it out. We do the same
all the time, although mostly only security relevant updates qualify as
putting out an update between releases.

(Unless the feature is really impressive)

Sincerely,
Lars Marowsky-Brée <lmb@suse.de>
Development HA

--
Perfection is our goal, excellence will be tolerated. -- J. Yahl

Hi,

Lars Marowsky-Bree wrote:
>
> Seriously, how often have we seen a question which was traced down to a kernel
> bug?

only twice ;)

> With Linux 2.4, LVS is "just another netfilter module", so this will become
> a lot easier. Just replace the netfilter module.

Just don't forget the dummy user to tell that he shouldn't load the
ip_conntrack module toghether with ipvs since I'm positive that SuSE
will ship a preconfigured kernel with all possible modules compiled
in. [ no offense to aa ]

> > I agree. If Suse or Redhat don't include the extra patches, then nobody
> > will use their distributions - they will find somebody who does.
>
> Or worse, people will not consider using Linux at all as their OS, because
> they can't get the needed features out of the box.

<idiot mode> This is ok for me. We have Windows for the dummy users </idiot

> Most problems relate to general issues with understanding how LVS works. The
> kernel patch used for that isn't quite a problem for that.

Redhat made the error of including the patch too early. It just was not
ready for productionary use. But the other way 'round if distros would
not include bleeding edge development patches, they would have nothing
to sell ;)

Regards,
Roberto Nibali, ratz

--
mailto: `echo NrOatSz@tPacA.cMh | sed 's/[NOSPAM]//g'`

On Tue, 21 Nov 2000, Lars Marowsky-Bree wrote:

> There usually isn't a problem with that. Frankly, I am quite amazed why the
> inclusion of LVS into RH's default kernel has created such a load - it
> shouldn't be the case.

you would hope not, but it would seem that people with problems using LVS
with RH come straight to us. Given that it's happened once, I'd like to
plan for it not to happen the same way when you do it.

> > > Having LVS included will work fine for most people
> >
> > it's the other people that I'm concerned about.
>
> There will always be people who can't get it right. The LVS install file
> should tell people
>
> "Hey, if you are getting an error message like this, or if your
> distribution already includes LVS, don't try applying the patch on top
> of it.

Given that you're going to do what you're going to do, we can do this.

> Do you think this would help?

yes, that should work. We can at least try it.

> Exactly. So there _isn't_ a pressing reason why people who are unable to
> should upgrade.

I guess not. If I'm running some 2.0.36 machines here, I should be able to
handle people who elect to run on ipvs which is one revision back

> > Rather than SuSE easing the lives of its purchasers, I'd rather SuSE
> > first minimised its impact on people developing the software that SuSE uses
> > and who are doing it in their spare time.
>
> We do have to balance both. And I don't see a conflict.
>
> But you are essentially asking for vendors not to include any patch on top of
> the plain kernel, which just will not work.

no I'm not, you can re-read my mail if you want to see what I said.

Joe
--
Joseph Mack mack@ncifcrf.gov