Mailing List Archive

On Wed, 28 Mar 2007 07:58:59 -0400
Chris Gianelloni <wolf31o2@gentoo.org> wrote:

> Please, everyone, go back and read the actual *facts* that were
> discovered using copies of *our* repositories before going around
> using data from outside sources.

Alec Warner's test results are here, of course:

http://www.gentoo.org/proj/en/infrastructure/cvs-migration.xml

FI on gentoo-x86 we're doing about 10,000 commits a month (from 100 to
500 commits a day), according to my #gentoo-commits logs. (Assuming
the SVN revision is a 32-bit number, it'll take about 1000 years to
saturate).

Personally I'm a fan of SVN over CVS, but that's from a client
perspective not the server. It would be interesting to find out why
SVN consumes double the bandwidth to checkout a full tree. It would
also be interesting to find out why the server disk usage is 4x that of
CVS (and what difference the choice of back-end makes).

FWIW the things I like in SVN, in order of importance to me are:
1) ability to do diffs off-line
2) maintains history when copying, moving etc (e.g. 'svn log' of CPV-r2
traces the history back through the point at which it was copied
from CPV-r1)
3) command line is largely the same as CVS (which avoids confusion
when moving between CVS and SVN repositories)

Alec - any chance you could flesh out exactly what tests you did? I
would have expected that the update-diff-commit cycle that we (well,
repoman) typically do would be more efficient on SVN than CVS, in
terms of the amount of data transferred between the client and server
(svn client sends diffs, cvs client sends whole files, and the diff
operation in the repoman cycle would be local in svn).

--
Kevin F. Quinn

--
Kevin F. Quinn

On Wed, 2007-03-28 at 15:23 +0200, Kevin F. Quinn wrote:
> SVN consumes double the bandwidth to checkout a full tree. It would
> also be interesting to find out why the server disk usage is 4x that of
> CVS (and what difference the choice of back-end makes).

I would bet that the file-system also makes some difference, along with
the back-end choices. Unless Alec took this into consideration.

--
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Kevin F. Quinn wrote:
> On Wed, 28 Mar 2007 07:58:59 -0400
> Chris Gianelloni <wolf31o2@gentoo.org> wrote:
>
>> Please, everyone, go back and read the actual *facts* that were
>> discovered using copies of *our* repositories before going around
>> using data from outside sources.
>
> Alec Warner's test results are here, of course:
>
> http://www.gentoo.org/proj/en/infrastructure/cvs-migration.xml

I've looked at this just now and in the past and at the last thread in which this was discussed
(http://marc.info/?t=116855132300001&r=1&w=2) and
http://dev.gentoo.org/~antarus/projects/soc/glep-0052.txt and there doesn't seem to be any hard data
which can be used to base an informed decision on. Things like git not supporting partial syncs were
brought up as being too painful for non-broadband users and disagreed with
(http://marc.info/?l=gentoo-dev&m=116918412629635&w=2). You can find some more issues like this in
the entire thread. Furthermore only git and svn (svk) seem to have been investigated and it is
unclear which versions were used. If you believe, like me, that non-distributed SCMs are broken,
then this leaves only git (and svk but "It is certainly not the optimal distributed VCS solution.").

These were the reasons I decided to look and see what other infos could be had on the internet. Of
course it is hard to come up with good measures of performance and I've certainly found very little
hard data.

So in light of all that I don't think it is wasteful to restart this discussion.

Of course not everyone is yet convinced that non-distributed SCMs are broken, so perhaps it would be
good if I ask the following question _here_ instead of privately.


Chris,
if I am to continue my plan of producing frequent releases of minimal amd64 install cd, then it
would probably help if I can use some versioning control and you might be interested in having easy
access to any changes I make. How can we achieve both? I believe the stuff I'm interested in is in
some CVS repository. As I see it I have the following options:

1) get commit access to the repository and start a branch in there. Merging may not be too hard, I
don't really know. However CVS commit access is not something that is given lightly. It would vice
versa also mean that you would have commit access to my stuff, which I might not like.

2) file bugs with patches attached. But maybe you just want to forget about releases until 2007.1
comes along once 2007.0 is finished.

3) fork the code or convert the repository into a repo of my own. Even if I choose to use the same
kind of repo (CVS in this case), then how hard will merging be? Again, this goes both ways.

I hope I missed something here, but of the three the third looks the most appealing and likely with
me forking into darcs probably. I don't think this issue would be here if the code were in a
distributed SCM, but maybe by the time 2007.1 is due I will have amassed enough interesting changes
that it is easier for you to then just clone my distributed repo ;P.


So can we please discuss what distributed SCM is best for the tree or likely to be in the future and
what hard data obtained with what tests should be gathered to rank SCMs and what feature differences
there are and how much we should care about them?

Marijn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGDi9op/VmCx0OL2wRArS/AKDGGC74l6xMFStjt3wS6PcOlTj/9wCdGwuR
8evRaXm3V8G7WWfUaC9luNM=
=XPYE
-----END PGP SIGNATURE-----
--
gentoo-dev@gentoo.org mailing list

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Kevin F. Quinn wrote:
>> On Wed, 28 Mar 2007 07:58:59 -0400
>> Chris Gianelloni <wolf31o2@gentoo.org> wrote:
>>
>>> Please, everyone, go back and read the actual *facts* that were
>>> discovered using copies of *our* repositories before going around
>>> using data from outside sources.
>>
>> Alec Warner's test results are here, of course:
>>
>> http://www.gentoo.org/proj/en/infrastructure/cvs-migration.xml
>
> So can we please discuss what distributed SCM is best for the tree or
> likely to be in the future and
> what hard data obtained with what tests should be gathered to rank SCMs
> and what feature differences
> there are and how much we should care about them?
>
> Marijn

I think the main point of this discussion is you can discuss all you like
but no one is going to switch because of a discussion. If you want to
convert the tree to something else; get two boxes, do the conversion, get
some data and then present it. If darcs has reasonable benifits over the
current system then bonus; you would have data to back up any other points
you have in a discussion to switch. But without that data you have
nothing. Our tree is not a 'normal' dataset for many of these systems.

Here are some stats from the 6th of last April.

Total CVS Files: 234672
Total CVS Revisions: 1309603
CVS Repos Size in KB: 783590
First Revision Date: Fri Jul 28 00:35:42 2000
Last Revision Date: Thu Apr 6 01:02:36 2006

We do around 10k revisions a month; so add another 120000 revisions
(roughly) to that count to get to this year.

-Alec

--
gentoo-dev@gentoo.org mailing list

On Sat, 2007-03-31 at 11:52 +0200, Marijn Schouten (hkBst) wrote:
> So in light of all that I don't think it is wasteful to restart this discussion.

I do.

Want to bring it back up? Go perform some tests and report back with
some data if you feel prior efforts weren't done properly or
reproducible. My *entire* point was that *discussion* of this issue is
worthless compared to numbers and data. I see no need to hear 300+
people tell everyone else their *opinion* on what they *think* is
better. Seeing some actual data, though, should be definitely
encouraged.

> 1) get commit access to the repository and start a branch in there. Merging may not be too hard, I
> don't really know. However CVS commit access is not something that is given lightly. It would vice
> versa also mean that you would have commit access to my stuff, which I might not like.

Release Engineering has its own space for officially-released and
in-progress media. Your stuff wouldn't really fall under that category,
so there's no need for you to have commit access to the repository.
Release Engineering holds final say on all changes and they need to go
through Release Engineering for testing before they will be accepted.
This is how all of our changes work and it's worked pretty well for us.

> 2) file bugs with patches attached. But maybe you just want to forget about releases until 2007.1
> comes along once 2007.0 is finished.

You are correct. Release Engineering is not concerned with the interim
state of the tree, and therefore has no need for constantly updating the
spec files. We focus our attention on the release snapshots and make
changes based on said snapshot, not on the current state of the tree.
We all have better things we would like to do than constantly update
spec files. Now, if you're talking about working with Release
Engineering during release times, that would be grand, but anything
off-cycle wouldn't be of much interest for us. Also, remember that
there's really no point in refreshing media on a constant basis. I
could see refreshing it after a new kernel version goes stable, and
that's about it. Anything else seems like a terrible waste of time and
resources for minimal gain. In most cases, your CD wouldn't change at
all from day to day. For QA purposes, I run builds year-round. I just
don't release them because I don't test them thoroughly.

> 3) fork the code or convert the repository into a repo of my own. Even if I choose to use the same
> kind of repo (CVS in this case), then how hard will merging be? Again, this goes both ways.

You don't merge. You would file a bug with the changes and we would
either accept or deny the changes on a case-by-case basis.

> I hope I missed something here, but of the three the third looks the most appealing and likely with
> me forking into darcs probably. I don't think this issue would be here if the code were in a
> distributed SCM, but maybe by the time 2007.1 is due I will have amassed enough interesting changes
> that it is easier for you to then just clone my distributed repo ;P.

Again, it is *very* unlikely. If you look at the changes in the minimal
CD set between releases, you will see that it is extremely minimal, if
changes are even made. We simply don't change things that work. The
only changes that really go in are bug fixes. Are you saying that
you're going to be tracking all of the release bugs and fixing only
those? Are you planning on adding features? The former would be
helpful to Release Engineering, the latter would not be nearly as useful
to us unless they were absolutely compelling.

> So can we please discuss what distributed SCM is best for the tree or likely to be in the future and
> what hard data obtained with what tests should be gathered to rank SCMs and what feature differences
> there are and how much we should care about them?

I don't get why you discuss a distributed SCM, then proceed to talk
about minimal CD + releases stuff which has nothing to do with the main
tree. We keep our spec files in CVS, but it isn't the same repository
as the tree. If you're talking about changing the tree, then yes, you
should be filing bugs and getting them fixed in the tree.

I'm honestly not sure what exactly it is that you're trying to
accomplish. Some additional explanation would be great.

Again, if you want to see the tree converted to something else, you need
to show compelling reasons and data *why* it should be done. Discussing
it doesn't really show those things and lends itself to giving only
beliefs, political or personal, about given SCM software. I honestly
don't care what anybody *thinks* about any particular SCM. I am
interested in the facts and numbers. I don't have much preference
myself other than that I already know CVS/SVN. If we were to make a
change, even to SVN, I'd like to see some well-thought-out reasons why
and some numbers to back it up.

--
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris Gianelloni wrote:
> On Sat, 2007-03-31 at 11:52 +0200, Marijn Schouten (hkBst) wrote:
>> So in light of all that I don't think it is wasteful to restart this discussion.
>
> I do.
>
> Want to bring it back up? Go perform some tests and report back with
> some data if you feel prior efforts weren't done properly or
> reproducible. My *entire* point was that *discussion* of this issue is
> worthless compared to numbers and data. I see no need to hear 300+
> people tell everyone else their *opinion* on what they *think* is
> better. Seeing some actual data, though, should be definitely
> encouraged.
> Again, if you want to see the tree converted to something else, you need
> to show compelling reasons and data *why* it should be done. Discussing
> it doesn't really show those things and lends itself to giving only
> beliefs, political or personal, about given SCM software. I honestly
> don't care what anybody *thinks* about any particular SCM. I am
> interested in the facts and numbers. I don't have much preference
> myself other than that I already know CVS/SVN. If we were to make a
> change, even to SVN, I'd like to see some well-thought-out reasons why
> and some numbers to back it up.

I just don't think it is obvious what tests should be performed. Furthermore the difference between
the different systems is not just performance, but also features. So we need to discuss what
standards any candidate SCM should measure up to.

I thought the shortcomings in features of CVS in comparison with SVN were understood. Given in turn
SVN's shortcomings in comparison to distributed SCMs and the abundance and maturity of them it seems
to me that the only decision to be made is what to switch to.

> I don't get why you discuss a distributed SCM, then proceed to talk
> about minimal CD + releases stuff which has nothing to do with the main
> tree.

Just an example to demonstrate how non-distributed SCM impose artificial restrictions. You wanted to
be convinced, right? I realize the specifics of the example, specifically the expected small extent
of divergence, make this a bad example in practice. But think about the theory.
But let me try again. Suppose you are developing an ebuild or are cooperating in developing an
ebuild or set of ebuilds with eclasses such as happens now in overlays. Such overlays could just be
branches in the same repository with easy merging between branches which preserves history. All with
one tool.
It would also empower people who don't have push access to the tree or to a specific overlay or to
any overlay, by making it possible for them to do everything people with push access do except
pushing, instead of also making it very hard to use the same SCM.

- From some discussion on irc I learned that lack of tree and history slicing are two concerns of
git's readyness. I hope to do some tests on the tree slicing soon.
I also learned that darcs does not support enough architectures, most importantly mips. Therefore
I'd like to know what architectures need to be supported by a candidate SCM.

Marijn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.3 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGEo81p/VmCx0OL2wRApQxAKCh+ZB64BnDId+ZLPDh2k3xxIoQFgCgsLTJ
pFc/u9hEFshBUAIhXlvGgLk=
=j+xm
-----END PGP SIGNATURE-----
--
gentoo-dev@gentoo.org mailing list

On Tue, 03 Apr 2007 19:30:29 +0200
"Marijn Schouten (hkBst)" <hkBst@gentoo.org> wrote:

> Therefore I'd like to
> know what architectures need to be supported by a candidate SCM.

Oh that's an easy one.
All arches that Gentoo supports.

Also it needs to support FreeBSD :)

Thanks

Roy
--
gentoo-dev@gentoo.org mailing list

On Tue, 2007-04-03 at 19:30 +0200, Marijn Schouten (hkBst) wrote:
> I just don't think it is obvious what tests should be performed. Furthermore the difference between
> the different systems is not just performance, but also features. So we need to discuss what
> standards any candidate SCM should measure up to.

No, we really don't.

First off, let's look at things we know we need. This is pretty much
the CVS feature set. Next, look at things we want. Does any SCM
provide things we want? Now, I'm not going to reiterate all the junk
people have said they want, since it's all archived for prosperity.
Next, start comparing the things we *require* and the things we *want*
in each SCM. Some good metrics people have already been using are
server-side disk space, client-side disk space, bandwidth, time for
checkout, time for commit, time for update...

Also, remember that the needs of the few definitely doesn't outweigh the
needs of the many. If 99.9% of the developer pool are doing only
checkout/update/commit cycles, then having a 50% drop in performance or
a 700% increase in disk usage only to gain features that don't affect
the 99.9% make a migration no longer worth it. This is what I mean by
using numbers to back up your ideas.

> I thought the shortcomings in features of CVS in comparison with SVN were understood. Given in turn
> SVN's shortcomings in comparison to distributed SCMs and the abundance and maturity of them it seems
> to me that the only decision to be made is what to switch to.

What shortcomings, exactly? This is something that you have to
quantify.

CVS does $x
CVS does not do $y

I simply have not seen much of anything that would be useful to a large
enough section of our developer pool to be worth the problems of a
migration. About the only thing I see is "svn cp" to preserve history.
I see lots of reasons for it in non-tree repositories, but little in the
tree, which already has branches and tags disabled, among other things.

> > I don't get why you discuss a distributed SCM, then proceed to talk
> > about minimal CD + releases stuff which has nothing to do with the main
> > tree.
>
> Just an example to demonstrate how non-distributed SCM impose artificial restrictions. You wanted to
> be convinced, right? I realize the specifics of the example, specifically the expected small extent
> of divergence, make this a bad example in practice. But think about the theory.

OK. You weren't able to successfully demonstrate anything to me, then.
I saw nothing in your mail that showed me why what you described would
be a problem, especially considering the examples you used.

> But let me try again. Suppose you are developing an ebuild or are cooperating in developing an
> ebuild or set of ebuilds with eclasses such as happens now in overlays. Such overlays could just be
> branches in the same repository with easy merging between branches which preserves history. All with
> one tool.

I guess I've just never had the need to do anything of the sort. I'm
perfectly capable of using revision bumps and other methodologies
already in use in the main tree in my overlays. Why do we need two sets
of practices? Why do we need to modify the main tree to fit the model
of the much smaller and less utilized overlays?

> It would also empower people who don't have push access to the tree or to a specific overlay or to
> any overlay, by making it possible for them to do everything people with push access do except
> pushing, instead of also making it very hard to use the same SCM.

Like what?

Qualify your statements. I don't use other SCM software, like many of
our developers/users. If you're going to try to tell me that I can't do
something I don't want to do, or don't even know is possible, you won't
convince me without compelling examples.

My point is that instead of discussing all of this yet again, you get
together some features you think are required and why, as well as some
performance metrics, as I stated above, and try approaching this from a
more technical front and less of an emotional one. Like I said, I don't
care which SCM you like. You shouldn't care which one I like. There's
no way we could ever please everyone, so why even bother to switch?

> - From some discussion on irc I learned that lack of tree and history slicing are two concerns of
> git's readyness. I hope to do some tests on the tree slicing soon.

Excellent.

This was something that wasn't available before, so if you're wanting to
test it with a newer git that does this well, then that is something we
can look at as something that has changed.

> I also learned that darcs does not support enough architectures, most importantly mips. Therefore
> I'd like to know what architectures need to be supported by a candidate SCM.

Ideally, all of them. I would consider dropping support for an
architecture we support currently a strong reason to never consider that
SCM. If I cannot commit from the machine I'm doing a KEYWORD request
on, the SCM fails IMO.

--
Chris Gianelloni
Release Engineering Strategic Lead
Alpha/AMD64/x86 Architecture Teams
Games Developer/Council Member/Foundation Trustee
Gentoo Foundation

Chris Gianelloni <wolf31o2@gentoo.org> posted
1175626466.8202.56.camel@inertia.twi-31o2.org, excerpted below, on Tue,
03 Apr 2007 14:54:26 -0400:

> Now, I'm not going to reiterate all the junk people have said they want,
> since it's all archived for prosperity.

Now /that/ was worth the read. Interesting eggcorn[1] there. =8^)

FWIW, Google lists 27 hits for "archived for prosperity", 11,400 hits for
"archived for posterity".

You've made my day, thanks! =8^)

[1] http://en.wiktionary.org/wiki/eggcorn

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

--
gentoo-dev@gentoo.org mailing list

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

There seemed to be still some doubt as to the benefits of distributed SCMs.
Fortunately Linus himself, at a talk at Google a few weeks back, explained the
benefits of being distributed and said fun and interesting things about git
and other source code managers. Yay for video cameras.

slashdot [ http://developers.slashdot.org/article.pl?sid=07/06/03/004214 ]

youtube [ http://www.youtube.com/watch?v=4XpnKHJAok8 ]

youtube non-flash (direct link to video file):
[ http://chi-v131.chi.youtube.com/get_video?video_id =4XpnKHJAok8 ]

independent non-flash:
[ http://www.meebey.net/temp/Tech Talk: Linus Torvalds on git.avi ]

happy watching,

Marijn
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGYvO8p/VmCx0OL2wRAp26AKCLy9iGQWIgFQ0ZuAGyknGLWS2QfACgg/mM
sKc9jojM1cPW80b7kOY8/Jg=
=rsjM
-----END PGP SIGNATURE-----
--
gentoo-dev@gentoo.org mailing list

On Sun, 03 Jun 2007 19:00:44 +0200
"Marijn Schouten (hkBst)" <hkBst@gentoo.org> wrote:
> There seemed to be still some doubt as to the benefits of distributed
> SCMs. Fortunately Linus himself, at a talk at Google a few weeks
> back, explained the benefits of being distributed and said fun and
> interesting things about git and other source code managers. Yay for
> video cameras.

Linus is biased towards the Linux development model, which is highly
atypical.

--
Ciaran McCreesh

Ciaran McCreesh wrote:
> Linus is biased towards the Linux development model, which is highly
> atypical.

Better said as "git is pretty good for developing linux", would it fit
our needs?

The only problem I see is getting a good documentation about using git
sourcemage[1] made something nice even if I'm not sure how much up to date.


[1] http://wiki.sourcemage.org/Git_Guide


--

Luca Barbato

Gentoo/linux Gentoo/PPC
http://dev.gentoo.org/~lu_zero
--
gentoo-dev@gentoo.org mailing list

On 6/3/07, Luca Barbato <lu_zero@gentoo.org> wrote:
> Ciaran McCreesh wrote:
> > Linus is biased towards the Linux development model, which is highly
> > atypical.
>
> Better said as "git is pretty good for developing linux", would it fit
> our needs?
>
> The only problem I see is getting a good documentation about using git
> sourcemage[1] made something nice even if I'm not sure how much up to date.
>
>
> [1] http://wiki.sourcemage.org/Git_Guide

How about Git User's Manual [2]?

[2] http://www.kernel.org/pub/software/scm/git/docs/user-manual.html
>
> --
>
> Luca Barbato
>
> Gentoo/linux Gentoo/PPC
> http://dev.gentoo.org/~lu_zero
> --
> gentoo-dev@gentoo.org mailing list
>
>


--
Duy
--
gentoo-dev@gentoo.org mailing list

O/H Marijn Schouten (hkBst) έγραψε:
> There seemed to be still some doubt as to the benefits of distributed
> SCMs.
> Fortunately Linus himself, at a talk at Google a few weeks back,
> explained the
> benefits of being distributed and said fun and interesting things
> about git
> and other source code managers. Yay for video cameras.
>
> slashdot [ http://developers.slashdot.org/article.pl?sid=07/06/03/004214 ]
>
> youtube [ http://www.youtube.com/watch?v=4XpnKHJAok8 ]
>
> youtube non-flash (direct link to video file):
> [ http://chi-v131.chi.youtube.com/get_video?video_id =4XpnKHJAok8 ]
>
> independent non-flash:
> [ http://www.meebey.net/temp/Tech Talk: Linus Torvalds on git.avi ]
>
> happy watching,
>
> Marijn
very interesting talk...
but i think linus is too biased against other scms...
--
gentoo-dev@gentoo.org mailing list

On Mon, Jun 04, 2007 at 03:30:37AM +0300, Stratos Psomadakis wrote:
> but i think linus is too biased against other scms...

He is biased against technical choices done in other SCMs, which is
not exactly the main thing. Specifically, from what I can see, he
hates:

- centralization (cvs, svn...)

- file ids (cvs, svn, hg, ...)

- hiding what is really going on on the pretext that "it's easier"
(lots of interfaces out there, or people who want commit -a as
default for git)

- lack of ways to be sure we're talking about the same code in the end
(quilt)

- anything slow (almost every other scm out there)

I very probably have missed some. It's obvious why git doesn't have
properties Linus hates.

OG.
--
gentoo-dev@gentoo.org mailing list