Mailing List Archive

On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> The TL;DR is that a crack team of infra-folks[0] have been putting together
> demos of CI services and things like gitlab / gitea / gerrit and so on.
>
> Some of these come in combined (e.g. gitlab offers repo hosting, code
> review / pull reqs, CI services, and deploy services.) Some of these are
> piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> repo-hosting but CI is separate (e.g. drone.)
>
> On the infra-side, I think we are pretty happy with repo-hosting (gitolite)
> and repo-serving (gitweb). We are missing a CI piece and a pull-request
> piece. Most of the users using PRs use either a gitlab or github mirror.
>
> I think the value of CI is pretty obvious to me (and I see tons of use
> cases in Infra.) We could easily build CI into our current repository
> solution (e.g. gitolite.) However gitolite doesn't really support PRs in a
> uniform way and so CI is mostly for submitted code; similar to the existing
> ::gentoo repo CI offered by mgorny.
>
> If we build a code review solution (like gitea / gerrit) would people use
> it? Would you use it if you couldn't merge (because the code review
> solution can't gpg sign your commits or merges) so a tool like the existing
> pram tool would be needed to merge?
>

Does GitLab count? Gerrit is just PITA. I think we had some concerns
about Gitea, so I'd like to test it before deciding. GitLab OTOH works
just fine for a lot of projects, and seems the next best thing after
GitHub.

--
Best regards,
Micha? Górny

On Tue, 26 May 2020 20:24:56 -0700
Alec Warner <antarus@gentoo.org> wrote:

> The TL;DR is that a crack team of infra-folks[0] have been putting
> together demos of CI services and things like gitlab / gitea / gerrit
> and so on.
>
> Some of these come in combined (e.g. gitlab offers repo hosting, code
> review / pull reqs, CI services, and deploy services.) Some of these
> are piecemeal (e.g. gerrit has code review, zuul has CI) and gitea
> offers repo-hosting but CI is separate (e.g. drone.)
>
> On the infra-side, I think we are pretty happy with repo-hosting
> (gitolite) and repo-serving (gitweb). We are missing a CI piece and a
> pull-request piece. Most of the users using PRs use either a gitlab
> or github mirror.
>
> I think the value of CI is pretty obvious to me (and I see tons of use
> cases in Infra.) We could easily build CI into our current repository
> solution (e.g. gitolite.) However gitolite doesn't really support PRs
> in a uniform way and so CI is mostly for submitted code; similar to
> the existing ::gentoo repo CI offered by mgorny.
>
> If we build a code review solution (like gitea / gerrit) would people
> use it? Would you use it if you couldn't merge (because the code
> review solution can't gpg sign your commits or merges) so a tool like
> the existing pram tool would be needed to merge?
>
> -A
>
> [0] Mostly arzano, if I'm honest. I am just the point-y haired
> manager in this effort.

There are several CI systems that could be used. As for CI, my
experience has been with buildbot. It would be fairly straightforward
to integrate with the current gitolite/gitweb hosting.
It is also extremely flexible in its capabilities, although can be
difficult to master. It has a good web interface, but it can even be
run via it's API's using curses, python, bash,.... There is a
buildbot-travis plugin which is capable of running existing .travis file
configurations. It also extends its capabilities to include custom
buildbot step definitions. I have not packaged it yet for gentoo, but
it is on my todo list. One of buildbot's capabilities is the ability to
run tests/builds on multiple workers (different arches or whatever)
both in parallel or series. It could be made to integrate with our
bugzilla using the python client (pybugs was it?).

But that still leaves a PR/code review option.

On Tue, May 26, 2020, 23:08 Micha? Górny <mgorny@gentoo.org> wrote:

> On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> > The TL;DR is that a crack team of infra-folks[0] have been putting
> together
> > demos of CI services and things like gitlab / gitea / gerrit and so on.
> >
> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> > review / pull reqs, CI services, and deploy services.) Some of these are
> > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> > repo-hosting but CI is separate (e.g. drone.)
> >
> > On the infra-side, I think we are pretty happy with repo-hosting
> (gitolite)
> > and repo-serving (gitweb). We are missing a CI piece and a pull-request
> > piece. Most of the users using PRs use either a gitlab or github mirror.
> >
> > I think the value of CI is pretty obvious to me (and I see tons of use
> > cases in Infra.) We could easily build CI into our current repository
> > solution (e.g. gitolite.) However gitolite doesn't really support PRs in
> a
> > uniform way and so CI is mostly for submitted code; similar to the
> existing
> > ::gentoo repo CI offered by mgorny.
> >
> > If we build a code review solution (like gitea / gerrit) would people use
> > it? Would you use it if you couldn't merge (because the code review
> > solution can't gpg sign your commits or merges) so a tool like the
> existing
> > pram tool would be needed to merge?
> >
>
> Does GitLab count? Gerrit is just PITA. I think we had some concerns
> about Gitea, so I'd like to test it before deciding. GitLab OTOH works
> just fine for a lot of projects, and seems the next best thing after
> GitHub


Gitlab does count (we deployed and tested an onprem version.) I think there
are some major issues with it though.
- Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI approved
and many of the features we need are EE only and are not available in CE.
- Complex: Gitlab is a giant piece of software with maybe 8-12 components
(unicorn, postgres, redis, memcache, sidekiq, puma, workhouse, gitaly,
grafana, sshd,nginx, prometheus ..the list goes on)
- I think gitlab ships with more features than we will use (CD, docker
registry, issues / bugs, wiki, analytics, snippets, milestones, repo
hosting, repo browsing, ... Again the list goes on.) I don't play to
migrate away from bugs.gentoo.org nor wiki.gentoo.org, nor gitolite. I
think if we did; then gitlab would be a more compelling option because it
is a one-stop-shop solution for those use cases.

My understanding of gitea is that it works great for not-::gentoo, but
::gentoo and gitea don't work well and it would require work upstream to
fix; other large repos seemed to work OK in gitea (based on our test
deployment and conversations with gitea upstream.)

Gerrit is widely used for large projects and I'm not worried for ::gentoo
and we have deployed gerrit and it seems to work fine. Gerrit doesn't have
CI (we would need to deploy something) and it uses gitweb for repository
browsing (which we use today.)

-A


>
> --
> Best regards,
> Micha? Górny
>
>

On Wed, 2020-05-27 at 01:14 -0700, Alec Warner wrote:
> On Tue, May 26, 2020, 23:08 Micha? Górny <mgorny@gentoo.org> wrote:
>
> > On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> > > The TL;DR is that a crack team of infra-folks[0] have been putting
> > together
> > > demos of CI services and things like gitlab / gitea / gerrit and so on.
> > >
> > > Some of these come in combined (e.g. gitlab offers repo hosting, code
> > > review / pull reqs, CI services, and deploy services.) Some of these are
> > > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> > > repo-hosting but CI is separate (e.g. drone.)
> > >
> > > On the infra-side, I think we are pretty happy with repo-hosting
> > (gitolite)
> > > and repo-serving (gitweb). We are missing a CI piece and a pull-request
> > > piece. Most of the users using PRs use either a gitlab or github mirror.
> > >
> > > I think the value of CI is pretty obvious to me (and I see tons of use
> > > cases in Infra.) We could easily build CI into our current repository
> > > solution (e.g. gitolite.) However gitolite doesn't really support PRs in
> > a
> > > uniform way and so CI is mostly for submitted code; similar to the
> > existing
> > > ::gentoo repo CI offered by mgorny.
> > >
> > > If we build a code review solution (like gitea / gerrit) would people use
> > > it? Would you use it if you couldn't merge (because the code review
> > > solution can't gpg sign your commits or merges) so a tool like the
> > existing
> > > pram tool would be needed to merge?
> > >
> >
> > Does GitLab count? Gerrit is just PITA. I think we had some concerns
> > about Gitea, so I'd like to test it before deciding. GitLab OTOH works
> > just fine for a lot of projects, and seems the next best thing after
> > GitHub
>
> Gitlab does count (we deployed and tested an onprem version.) I think there
> are some major issues with it though.
> - Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI approved
> and many of the features we need are EE only and are not available in CE.

What are these features, and why do you believe we need them?

> - Complex: Gitlab is a giant piece of software with maybe 8-12 components
> (unicorn, postgres, redis, memcache, sidekiq, puma, workhouse, gitaly,
> grafana, sshd,nginx, prometheus ..the list goes on)

Is gitea any better?

> - I think gitlab ships with more features than we will use (CD, docker
> registry, issues / bugs, wiki, analytics, snippets, milestones, repo
> hosting, repo browsing, ... Again the list goes on.) I don't play to
> migrate away from bugs.gentoo.org nor wiki.gentoo.org, nor gitolite. I
> think if we did; then gitlab would be a more compelling option because it
> is a one-stop-shop solution for those use cases.

I don't think there is any requirement to use all of them. Furthermore,
I think some of them may actually be helpful -- say, some Gentoo-
specific projects could use GitLab issue tracker over creating more
Bugzilla components.

> My understanding of gitea is that it works great for not-::gentoo, but
> ::gentoo and gitea don't work well and it would require work upstream to
> fix; other large repos seemed to work OK in gitea (based on our test
> deployment and conversations with gitea upstream.)

Works great for whom? How many deployments are we talking about? To be
honest, I don't think I've stumbled upon a single instance.
On the other hand, GitLab deployments are pretty common -- GNOME, Xfce,
Debian come instantly to my mind. Then, there's Heptapod -- the GitLab
fork for Mercurial.

> Gerrit is widely used for large projects and I'm not worried for ::gentoo
> and we have deployed gerrit and it seems to work fine. Gerrit doesn't have
> CI (we would need to deploy something) and it uses gitweb for repository
> browsing (which we use today.)
>

Not to mention it's ugly and I found it cumbersome to use.

--
Best regards,
Micha? Górny

On Wed, May 27, 2020 at 1:09 AM Brian Dolbec <dolsen@gentoo.org> wrote:

> On Tue, 26 May 2020 20:24:56 -0700
> Alec Warner <antarus@gentoo.org> wrote:
>
> > The TL;DR is that a crack team of infra-folks[0] have been putting
> > together demos of CI services and things like gitlab / gitea / gerrit
> > and so on.
> >
> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> > review / pull reqs, CI services, and deploy services.) Some of these
> > are piecemeal (e.g. gerrit has code review, zuul has CI) and gitea
> > offers repo-hosting but CI is separate (e.g. drone.)
> >
> > On the infra-side, I think we are pretty happy with repo-hosting
> > (gitolite) and repo-serving (gitweb). We are missing a CI piece and a
> > pull-request piece. Most of the users using PRs use either a gitlab
> > or github mirror.
> >
> > I think the value of CI is pretty obvious to me (and I see tons of use
> > cases in Infra.) We could easily build CI into our current repository
> > solution (e.g. gitolite.) However gitolite doesn't really support PRs
> > in a uniform way and so CI is mostly for submitted code; similar to
> > the existing ::gentoo repo CI offered by mgorny.
> >
> > If we build a code review solution (like gitea / gerrit) would people
> > use it? Would you use it if you couldn't merge (because the code
> > review solution can't gpg sign your commits or merges) so a tool like
> > the existing pram tool would be needed to merge?
> >
> > -A
> >
> > [0] Mostly arzano, if I'm honest. I am just the point-y haired
> > manager in this effort.
>
> There are several CI systems that could be used. As for CI, my
> experience has been with buildbot. It would be fairly straightforward
> to integrate with the current gitolite/gitweb hosting.
> It is also extremely flexible in its capabilities, although can be
> difficult to master. It has a good web interface, but it can even be
> run via it's API's using curses, python, bash,.... There is a
> buildbot-travis plugin which is capable of running existing .travis file
> configurations. It also extends its capabilities to include custom
> buildbot step definitions. I have not packaged it yet for gentoo, but
> it is on my todo list. One of buildbot's capabilities is the ability to
> run tests/builds on multiple workers (different arches or whatever)
> both in parallel or series. It could be made to integrate with our
> bugzilla using the python client (pybugs was it?).
>

My understanding is that CI systems are all quite similar. We have
configured gitlab-ci, drone, and zuul as tests and I am familiar with
travis-ci and buildbot from other projects. I'm less concerned about this
aspect tbh. I'm also not super concerned about packaging. E.g. for
gitlab-runner (the worker portion of gitlab-ci) we just deploy upstream
containers and don't package them in ::gentoo at all. Our onprem gitlab is
a container solution; gitea is a container solution; our SSO IDP is a
container solution; gerrit is currently a container solution. These don't
bother me (too much, ugh zuul uses zookeeper? ugh.)

The major differentiators for CI appear to be:
- SCM support (we currently only really care about git compatible systems,
but some contenders only support some providers.)
- builders / workers (how do they deploy). For example gitlab has a
container based work executor while zuul uses ansible; I suspect
- Authentication (ideally should support SAML or openID so we can
integrate with our alpha sso solution for Gentoo.)
- The webUI; e.g is the solution horrible to use / interact with. Hard to
say without a trial, IMHO.

-A


>
> But that still leaves a PR/code review option.
>
>

> Works great for whom? How many deployments are we talking about? To be
> honest, I don't think I've stumbled upon a single instance.
> On the other hand, GitLab deployments are pretty common -- GNOME, Xfce,
> Debian come instantly to my mind. Then, there's Heptapod -- the GitLab
> fork for Mercurial.

KDE also just migrated to GitLab.

>
> > Gerrit is widely used for large projects and I'm not worried for ::gentoo
> > and we have deployed gerrit and it seems to work fine. Gerrit doesn't have
> > CI (we would need to deploy something) and it uses gitweb for repository
> > browsing (which we use today.)
>
> Not to mention it's ugly and I found it cumbersome to use.

Everytime I tried to use Gerrit I got so thouroughly confused that I gave up
after a while.

--
Andreas K. H?ttel
dilfridge@gentoo.org
Gentoo Linux developer
(council, qa, toolchain, base-system, perl, libreoffice)

Hi,

is this really CI _vs_ Code Review? I.e. we can only have one?

CI is nice and helpful. For example I usually don't start review until CI
sets green light but CI alone wouldn't be enough. Thought that Gitlab will
support same kind of hooks similar to how current CI is integrated into
Github, not?

Also, when I could wish something: The problem when doing review on Github
for me is, that we usually create new revisions. Therefore we don't see
what's changed in new revision versus previous revision. So the change to
review looks like an entire new file was added (which is what basically
happened). It would be cool if our solution would be aware of this and
could handle this somehow. But I guess we would have to create our own
solution for this...


--
Regards,
Thomas Deutschmann / Gentoo Linux Developer
fpr: C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5

On 5/27/20 2:16 PM, Thomas Deutschmann wrote:
> The problem when doing review on Github
> for me is, that we usually create new revisions. Therefore we don't see
> what's changed in new revision versus previous revision.
That's my main concern with the current behaviour: a "git diff" often doesn't show a diff against the previous (ebuild) file, it shows a diff against /dev/null :-/

--
Toralf
PGP 23217DA7 9B888F45

Hi,

On Wed, May 27, 2020 at 02:45:29PM +0200, Toralf Förster wrote:
> On 5/27/20 2:16 PM, Thomas Deutschmann wrote:
> > The problem when doing review on Github
> > for me is, that we usually create new revisions. Therefore we don't see
> > what's changed in new revision versus previous revision.
> That's my main concern with the current behaviour: a "git diff" often doesn't show a diff against the previous (ebuild) file, it shows a diff against /dev/null :-/

Indeed, on GitHub it is hard to review, but locally you can add

[diff]
algorithm = patience

to your .gitconfig, and that should help with the diffs even when
the revision changes by moving the file. When copying, it probably
won't help.

We could also try as a policy to split the revision bump from the
changes, i.e. bump the revision in the first commit, then apply the
changes in a second one. That way, one can click on the right commit
to see the differences only, even on GitHub. Then we can squash when
merging locally, since we don't click merge on GitHub anyway.

Cheers,
-Guilherme

On 27/05/2020 15:22, Guilherme Amadio wrote:
> Hi,
>
> On Wed, May 27, 2020 at 02:45:29PM +0200, Toralf Förster wrote:
>> On 5/27/20 2:16 PM, Thomas Deutschmann wrote:
>>> The problem when doing review on Github
>>> for me is, that we usually create new revisions. Therefore we don't see
>>> what's changed in new revision versus previous revision.
>> That's my main concern with the current behaviour: a "git diff" often doesn't show a diff against the previous (ebuild) file, it shows a diff against /dev/null :-/
>
> Indeed, on GitHub it is hard to review, but locally you can add
>
> [diff]
> algorithm = patience
>
> to your .gitconfig, and that should help with the diffs even when
> the revision changes by moving the file. When copying, it probably
> won't help.
>
> We could also try as a policy to split the revision bump from the
> changes, i.e. bump the revision in the first commit, then apply the
> changes in a second one. That way, one can click on the right commit
> to see the differences only, even on GitHub. Then we can squash when
> merging locally, since we don't click merge on GitHub anyway.
>
> Cheers,
> -Guilherme
>
Hi,

That looks like it could lead to errors when merging if the committer
forgets to squash the two commits.

Whether it is on GitHub or another platform, it is probably possible to
have a bot compute the diff with the previous version every time a
commit is pushed to a PR and add that as a comment.

On Wed, May 27, 2020 at 5:16 AM Thomas Deutschmann <whissi@gentoo.org>
wrote:

> Hi,
>
> is this really CI _vs_ Code Review? I.e. we can only have one?
>

I'll try to come back to this. We can make computers do anything, but it's
a question of limited time for me and for Gentoo as a whole.


>
> CI is nice and helpful. For example I usually don't start review until CI
> sets green light but CI alone wouldn't be enough. Thought that Gitlab will
> support same kind of hooks similar to how current CI is integrated into
> Github, not?
>

So I'm basically back to worrying about social contract here. If its a
common Gentoo workflow to:
- Have a user fork a repo (github)
- Submit a PR (github)
- Gentoo CI sees the PR against ::gentoo and runs CI (github)
- CI turns green and comments on the PR (github)
- Human Gentoo dev reviews PR (github) [some iterations of review and
changes with PR author]
- Human Gentoo dev downloads PR patch and applies to Gentoo (Gentoo
Hosted.)

Does this workflow meet the social contract? I would assert it does not.
Now of course Gentoo doesn't "solely rely" on this workflow and other
workflows are available; but if this is a de-facto workflow people are
using...does not that concern the community? It concerns me! Hey we
originally were like 'nah github is OK' and now I'm coming around and
saying 'hey maybe this isn't OK in its current form.' So one idea might be
"what can we do about this particular problem."

What others have done is typically left mirrors of their projects on
github, written bots to auto-close PRs to those repos, and funneled users
somewhere else. We could do that if we wanted.


>
> Also, when I could wish something: The problem when doing review on Github
> for me is, that we usually create new revisions. Therefore we don't see
> what's changed in new revision versus previous revision. So the change to
> review looks like an entire new file was added (which is what basically
> happened). It would be cool if our solution would be aware of this and
> could handle this somehow. But I guess we would have to create our own
> solution for this...
>

So to address your first point above, there are a bunch of (sometimes
conflicting) requirements.

[Github]
A past argument for Github was "we need to be on github because this is
where the users are." I can tell you the users have not left Github and so
without us funneling them somewhere else...I mean if we are going to
continue to accept PRs on github we should probably service them (vs
auto-closing.)

[Code Review]
Basically how do we build a system where a user can go find our code, fork
it, generate a patch, send us the patch, and then be able to run CI on it
and do a review online (as opposed to on bugzilla / in-email.) Currently
this happens on github and I think there is some support for moving those
users to some other solution.

We could:
- Build something on gitlab CE.
- Build something on gitea + a CI solution (drone, zuul, etc.)
- I don't think gerrit credibly meets these requirements because users
can't fork in gerrit and it requires annoying client side tooling.
- The main problem I think with gitlab-CE is no one wants to operate it;
its too complex of a software package and there is a major fear that once
we mgirate everyone on it, something will go wrong and then we will be in
trouble. I don't have this fear with gitea because there are fewer moving
parts.
- The main problem with gitea is that it's relatively unproven and we
already know it needs patches for ::gentoo.

[CI-for-ebuilds]
My understanding is that there are two existing CI workflows (I'm ignoring
the new thing nattka; it's what I'd consider a different workflow.)
(1) Github PRs: A bot watches for PRs against ::gentoo on Github and runs
CI on them.
(2) Post-Submit CI: Every so often we run Ci against ::gentoo and if
something looks super bad we bisect and email that the build is broken.

So right off the bat there are some gaps.
- There is no pre-submit CI on git.gentoo.org at all, so devs can't vet
their changes there. Would anyone like this?
- There is no flexibility; the CI you get is whatever is configured (e.g.
what mgorny set up). This isn't meant as a dig; maybe no one wants to set
up CI at all and just wants to use this stuff; I have no idea.

[Other Projects]
Portage has had CI for ages; but it runs on github and uses travis-CI
soko.git uses hosted gitlab-ci to build, run tests, and upload containers.
https://gitweb.gentoo.org/proj/catalyst.git/ has no CI, but might like some?
https://gitweb.gentoo.org/proj/gentoo-mirrorstats.git/
<https://gitweb.gentoo.org/proj/gentoo-mirrorstats.git/tree/> has no CI but
might like some?
https://gitweb.gentoo.org/proj/sandbox.git/ has no CI but might want some?
I could go on.

So to kind of summarize. If I wanted to just do [CI for other projects] we
could likely just build that now and not need to talk to most of the
community because the CI for other projects is fairly segmented. It's the
part that delivers a lot of value (to me!) but, I'd wager, little value to
the rest of the community. However the Code Review is intertwined with CI;
because I think CI brings a bunch of value into the Code Review process and
there are significant Code Review paths (bugzilla, git patches) that don't
have presubmit coverage that I think could be fixed. This is why I
separated the concerns.

-A


>
> --
> Regards,
> Thomas Deutschmann / Gentoo Linux Developer
> fpr: C4DD 695F A713 8F24 2AA1 5638 5849 7EE5 1D5D 74A5
>

On Wed, May 27, 2020 at 1:14 AM Alec Warner <antarus@gentoo.org> wrote:
> On Tue, May 26, 2020, 23:08 Micha? Górny <mgorny@gentoo.org> wrote:
>>
>> On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
>> > The TL;DR is that a crack team of infra-folks[0] have been putting together
>> > demos of CI services and things like gitlab / gitea / gerrit and so on.
>> >
>> > Some of these come in combined (e.g. gitlab offers repo hosting, code
>> > review / pull reqs, CI services, and deploy services.) Some of these are
>> > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
>> > repo-hosting but CI is separate (e.g. drone.)
>> >
>> > On the infra-side, I think we are pretty happy with repo-hosting (gitolite)
>> > and repo-serving (gitweb). We are missing a CI piece and a pull-request
>> > piece. Most of the users using PRs use either a gitlab or github mirror.
>> >
>> > I think the value of CI is pretty obvious to me (and I see tons of use
>> > cases in Infra.) We could easily build CI into our current repository
>> > solution (e.g. gitolite.) However gitolite doesn't really support PRs in a
>> > uniform way and so CI is mostly for submitted code; similar to the existing
>> > ::gentoo repo CI offered by mgorny.
>> >
>> > If we build a code review solution (like gitea / gerrit) would people use
>> > it? Would you use it if you couldn't merge (because the code review
>> > solution can't gpg sign your commits or merges) so a tool like the existing
>> > pram tool would be needed to merge?
>> >
>>
>> Does GitLab count? Gerrit is just PITA. I think we had some concerns
>> about Gitea, so I'd like to test it before deciding. GitLab OTOH works
>> just fine for a lot of projects, and seems the next best thing after
>> GitHub
>
>
> Gitlab does count (we deployed and tested an onprem version.) I think there are some major issues with it though.
> - Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI approved and many of the features we need are EE only and are not available in CE.

It's very surprising to me that CE wouldn't work for our purposes.
Debian, GNOME, KDE, XFCE, and FreeDesktop have all switched to GitLab
and are using CE. It's hard to believe that Gentoo's usage or
requirements would be so different as to make GitLab a non-viable
option.

What features of EE do you think we need?

On Wed, May 27, 2020 at 2:31 PM Matt Turner <mattst88@gentoo.org> wrote:

> On Wed, May 27, 2020 at 1:14 AM Alec Warner <antarus@gentoo.org> wrote:
> > On Tue, May 26, 2020, 23:08 Micha? Górny <mgorny@gentoo.org> wrote:
> >>
> >> On Tue, 2020-05-26 at 20:24 -0700, Alec Warner wrote:
> >> > The TL;DR is that a crack team of infra-folks[0] have been putting
> together
> >> > demos of CI services and things like gitlab / gitea / gerrit and so
> on.
> >> >
> >> > Some of these come in combined (e.g. gitlab offers repo hosting, code
> >> > review / pull reqs, CI services, and deploy services.) Some of these
> are
> >> > piecemeal (e.g. gerrit has code review, zuul has CI) and gitea offers
> >> > repo-hosting but CI is separate (e.g. drone.)
> >> >
> >> > On the infra-side, I think we are pretty happy with repo-hosting
> (gitolite)
> >> > and repo-serving (gitweb). We are missing a CI piece and a
> pull-request
> >> > piece. Most of the users using PRs use either a gitlab or github
> mirror.
> >> >
> >> > I think the value of CI is pretty obvious to me (and I see tons of use
> >> > cases in Infra.) We could easily build CI into our current repository
> >> > solution (e.g. gitolite.) However gitolite doesn't really support PRs
> in a
> >> > uniform way and so CI is mostly for submitted code; similar to the
> existing
> >> > ::gentoo repo CI offered by mgorny.
> >> >
> >> > If we build a code review solution (like gitea / gerrit) would people
> use
> >> > it? Would you use it if you couldn't merge (because the code review
> >> > solution can't gpg sign your commits or merges) so a tool like the
> existing
> >> > pram tool would be needed to merge?
> >> >
> >>
> >> Does GitLab count? Gerrit is just PITA. I think we had some concerns
> >> about Gitea, so I'd like to test it before deciding. GitLab OTOH works
> >> just fine for a lot of projects, and seems the next best thing after
> >> GitHub
> >
> >
> > Gitlab does count (we deployed and tested an onprem version.) I think
> there are some major issues with it though.
> > - Licensing. Gitlab-CE is available, gitlab-EE is not OSS nor OSI
> approved and many of the features we need are EE only and are not available
> in CE.
>
> It's very surprising to me that CE wouldn't work for our purposes.
> Debian, GNOME, KDE, XFCE, and FreeDesktop have all switched to GitLab
> and are using CE. It's hard to believe that Gentoo's usage or
> requirements would be so different as to make GitLab a non-viable
> option.
>
> What features of EE do you think we need?
>

I know debian spent considerable effort customizing their gitlab. I've not
heavily investigated the other deployments. We set up a demo on
gitlab.gentoo.org already as a test and there are some issues. Many of them
are related to operations and not to the app itself.

[mirroring]
- We can't do pull-based git mirroring (
https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html#pulling-from-a-remote-repository-starter
)
[Auth] Note that we have keycloak now, so we can perhaps workaround these
LDAP issues.
- GItlab doesn't support multiple redundant LDAP servers
- Gitlab doesn't support LDAP group syncing
- Gitlab doesn't support syncing admin users from LDAP
[mgmnt]
- The gitlab terraform provider is pretty bad; and I struggled to get it
to control our admin users; did not leave me excited about managing other
resources (projects, users, hooks, etc.)

The pull-based mirroring is a bit sad, as it would be nice to auto-update
some forks, but it's not a killer feature. I think our new SSO solution
could potentially be a fix for the auth subsystems, but more work there
will be needed.

Another major issue is operating the software. I haven't found anyone to
*run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
mirrors, email and lists are separate, etc. In a world where bugs, wiki,
code, ci, containers, PRs, are all on gitlab and it breaks and we can't fix
it; it will be bad news for all of those things. If the bugzilla machine
breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
wiki, file bugs, commit, run CI, etc.

-A

On Fri, 2020-05-29 at 16:34 -0700, Alec Warner wrote:
> The pull-based mirroring is a bit sad, as it would be nice to auto-update
> some forks, but it's not a killer feature.

Exactly. Especially that our push-based mirroring is better,
and I think that's how we want to populate it.

> I think our new SSO solution
> could potentially be a fix for the auth subsystems, but more work there
> will be needed.

I think SSO should be the primary login to our GitLab, especially for
our users. GitHub login is a must.

> Another major issue is operating the software. I haven't found anyone to
> *run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
> bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
> mirrors, email and lists are separate, etc. In a world where bugs, wiki,
> code, ci, containers, PRs, are all on gitlab and it breaks and we can't fix
> it; it will be bad news for all of those things. If the bugzilla machine
> breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
> wiki, file bugs, commit, run CI, etc.
>

But who says we want to migrate them all into GitLab? I thought our
primary goal was to replace today's GitHub use, i.e. provide
an alternative pipeline for pull/merge requests.

--
Best regards,
Micha? Górny

On Wed, 27 May 2020 01:28:02 -0700
Alec Warner <antarus@gentoo.org> wrote:

> On Wed, May 27, 2020 at 1:09 AM Brian Dolbec <dolsen@gentoo.org>
> wrote:
>
> > On Tue, 26 May 2020 20:24:56 -0700
> > Alec Warner <antarus@gentoo.org> wrote:
> >
> > > The TL;DR is that a crack team of infra-folks[0] have been putting
> > > together demos of CI services and things like gitlab / gitea /
> > > gerrit and so on.
> > >
> > > Some of these come in combined (e.g. gitlab offers repo hosting,
> > > code review / pull reqs, CI services, and deploy services.) Some
> > > of these are piecemeal (e.g. gerrit has code review, zuul has CI)
> > > and gitea offers repo-hosting but CI is separate (e.g. drone.)
> > >
> > > On the infra-side, I think we are pretty happy with repo-hosting
> > > (gitolite) and repo-serving (gitweb). We are missing a CI piece
> > > and a pull-request piece. Most of the users using PRs use either
> > > a gitlab or github mirror.
> > >
> > > I think the value of CI is pretty obvious to me (and I see tons
> > > of use cases in Infra.) We could easily build CI into our current
> > > repository solution (e.g. gitolite.) However gitolite doesn't
> > > really support PRs in a uniform way and so CI is mostly for
> > > submitted code; similar to the existing ::gentoo repo CI offered
> > > by mgorny.
> > >
> > > If we build a code review solution (like gitea / gerrit) would
> > > people use it? Would you use it if you couldn't merge (because
> > > the code review solution can't gpg sign your commits or merges)
> > > so a tool like the existing pram tool would be needed to merge?
> > >
> > > -A
> > >
> > > [0] Mostly arzano, if I'm honest. I am just the point-y haired
> > > manager in this effort.
> >
> > There are several CI systems that could be used. As for CI, my
> > experience has been with buildbot. It would be fairly
> > straightforward to integrate with the current gitolite/gitweb
> > hosting. It is also extremely flexible in its capabilities,
> > although can be difficult to master. It has a good web interface,
> > but it can even be run via it's API's using curses, python,
> > bash,.... There is a buildbot-travis plugin which is capable of
> > running existing .travis file configurations. It also extends its
> > capabilities to include custom buildbot step definitions. I have
> > not packaged it yet for gentoo, but it is on my todo list. One of
> > buildbot's capabilities is the ability to run tests/builds on
> > multiple workers (different arches or whatever) both in parallel or
> > series. It could be made to integrate with our bugzilla using the
> > python client (pybugs was it?).
>
> My understanding is that CI systems are all quite similar. We have
> configured gitlab-ci, drone, and zuul as tests and I am familiar with
> travis-ci and buildbot from other projects. I'm less concerned about
> this aspect tbh. I'm also not super concerned about packaging. E.g.
> for gitlab-runner (the worker portion of gitlab-ci) we just deploy
> upstream containers and don't package them in ::gentoo at all. Our
> onprem gitlab is a container solution; gitea is a container solution;
> our SSO IDP is a container solution; gerrit is currently a container
> solution. These don't bother me (too much, ugh zuul uses zookeeper?
> ugh.)
>
> The major differentiators for CI appear to be:
> - SCM support (we currently only really care about git compatible
> systems, but some contenders only support some providers.)
> - builders / workers (how do they deploy). For example gitlab has a
> container based work executor while zuul uses ansible; I suspect
> - Authentication (ideally should support SAML or openID so we can
> integrate with our alpha sso solution for Gentoo.)
> - The webUI; e.g is the solution horrible to use / interact with.
> Hard to say without a trial, IMHO.
>
> -A
>
>
> >
> > But that still leaves a PR/code review option.
> >
> >

I have another buildbot version bump to 2.8.0 to do. While reading
over the changes. There are changes to the gerrit integration
(GerritEventLogPoller) which I did not know it had or used.

There is also a change in the 2.7.0 buildbot-worker:

* Command buildbot-worker create-worker now supports ipv6 address for
buildmaster connection.

I know that buildbot have been creating buildbot containers since the
1.x series, but I've not worked with them. I am planning to learn how
to use buildbot with containers and kubernetes. I will certainly check
out how that worker command works. whether it is a docker or
kubernetes command or either...


So, buildbot may be a good option since it can integrate with gerrit
and (I think) dynamically deploy workers in containers. So be able to
integrate with our existing git/gitweb/bugzilla as well as github/gitlab

On Fri, May 29, 2020 at 04:34:24PM -0700, Alec Warner wrote:
> Another major issue is operating the software. I haven't found anyone to
> *run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
> bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
> mirrors, email and lists are separate, etc. In a world where bugs, wiki,
> code, ci, containers, PRs, are all on gitlab and it breaks and we can't fix
> it; it will be bad news for all of those things. If the bugzilla machine
> breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
> wiki, file bugs, commit, run CI, etc.

Ping me sometime this week if I don't catch up with you first. I have
some experience running gitlab, so I could help run it.

Thanks,

William

On Fri, May 29, 2020 at 11:17 PM Micha? Górny <mgorny@gentoo.org> wrote:

> On Fri, 2020-05-29 at 16:34 -0700, Alec Warner wrote:
> > The pull-based mirroring is a bit sad, as it would be nice to auto-update
> > some forks, but it's not a killer feature.
>
> Exactly. Especially that our push-based mirroring is better,
> and I think that's how we want to populate it.
>

So you want to keep gitolite then?


>
> > I think our new SSO solution
> > could potentially be a fix for the auth subsystems, but more work there
> > will be needed.
>
> I think SSO should be the primary login to our GitLab, especially for
> our users. GitHub login is a must.
>

I'm fairly sure both gitlab and gitea support this requirement.


>
> > Another major issue is operating the software. I haven't found anyone to
> > *run* gitlab; I'm not eager to do it. Today Gentoo is mostly distributed,
> > bugs are in bugzilla, wiki is on mediawiki, code is on gitolite with N
> > mirrors, email and lists are separate, etc. In a world where bugs, wiki,
> > code, ci, containers, PRs, are all on gitlab and it breaks and we can't
> fix
> > it; it will be bad news for all of those things. If the bugzilla machine
> > breaks we lose bugzilla; if gitlab breaks we lose the ability to edit the
> > wiki, file bugs, commit, run CI, etc.
> >
>
> But who says we want to migrate them all into GitLab? I thought our
> primary goal was to replace today's GitHub use, i.e. provide
> an alternative pipeline for pull/merge requests.
>

Oh I don't, but we then need to disable all of them and restrict their
usage.

-A


>
> --
> Best regards,
> Micha? Górny
>
>

On Sun, 2020-05-31 at 18:42 -0700, Alec Warner wrote:
> On Fri, May 29, 2020 at 11:17 PM Micha? Górny <mgorny@gentoo.org> wrote:
>
> > On Fri, 2020-05-29 at 16:34 -0700, Alec Warner wrote:
> > > The pull-based mirroring is a bit sad, as it would be nice to auto-update
> > > some forks, but it's not a killer feature.
> >
> > Exactly. Especially that our push-based mirroring is better,
> > and I think that's how we want to populate it.
> >
>
> So you want to keep gitolite then?

Yes. I don't think GitLab replaces all the features we use from it
(does it?). Plus, it's been quite reliable so far and I don't think
converting the configs is worth the effort.

--
Best regards,
Micha? Górny

Hi,

Sorry I'm a bit late to the party here but was behind on my emails.

Op wo 27 mei 2020 om 05:25 schreef Alec Warner <antarus@gentoo.org>:

> The TL;DR is that a crack team of infra-folks[0] have been putting
> together demos of CI services and things like gitlab / gitea / gerrit and
> so on.
>
>
I didn't see in the email chain any mention of the Atlassian tools. Can
these be considered? They offer free licenses for open source projects and
I think that most of the requirements are covered by BitBucket
(repo-hosting, repo-serving, code review and pull requests) and Bamboo (CI).

If desired I can setup a demo and/or help maintain the tools, this is my
day job and for those who care about these kind of things I am Atlassian
certified.

Here is more info on their licensing for Open Source projects:

https://www.atlassian.com/software/views/open-source-license-request

br,
Mathy

On Tue, 2020-06-02 at 10:44 +0200, Mathy Vanvoorden wrote:
> Hi,
>
> Sorry I'm a bit late to the party here but was behind on my emails.
>
> Op wo 27 mei 2020 om 05:25 schreef Alec Warner <antarus@gentoo.org>:
>
> > The TL;DR is that a crack team of infra-folks[0] have been putting
> > together demos of CI services and things like gitlab / gitea / gerrit and
> > so on.
> >
> >
> I didn't see in the email chain any mention of the Atlassian tools. Can
> these be considered? They offer free licenses for open source projects and
> I think that most of the requirements are covered by BitBucket
> (repo-hosting, repo-serving, code review and pull requests) and Bamboo (CI).
>
> If desired I can setup a demo and/or help maintain the tools, this is my
> day job and for those who care about these kind of things I am Atlassian
> certified.
>
> Here is more info on their licensing for Open Source projects:
>
> https://www.atlassian.com/software/views/open-source-license-request
>

1. This is not free software.

2. Atlassian has done a few very bad moves lately, and they're unlikely
to survive on the market for very long.

3. The whole point of having something self-hosted is not to rely
on third party hosting it for us.


--
Best regards,
Micha? Górny

Op di 2 jun. 2020 om 10:47 schreef Micha? Górny <mgorny@gentoo.org>:

> 1. This is not free software.
>

It was not clear for me if this was a requirement or not. The license is
free for open source projects. I don't see the difference between companies
offering CE versions of their commercial products, they also "take
advantage" of the communities' contributions.

2. Atlassian has done a few very bad moves lately, and they're unlikely
> to survive on the market for very long.
>

I'm not sure what you are talking about, they have their faults but if they
were headed for real problems I'd probably have heard about it and would
not consider proposing it.

3. The whole point of having something self-hosted is not to rely
> on third party hosting it for us.
>

I'm talking about self-hosted too, not their cloud offering. Bamboo isn't
even available in the cloud.

--
Best Regards,
Mathy

On Tue, Jun 2, 2020 at 1:44 AM Mathy Vanvoorden <mathy@vanvoorden.be> wrote:

> Hi,
>
> Sorry I'm a bit late to the party here but was behind on my emails.
>
> Op wo 27 mei 2020 om 05:25 schreef Alec Warner <antarus@gentoo.org>:
>
>> The TL;DR is that a crack team of infra-folks[0] have been putting
>> together demos of CI services and things like gitlab / gitea / gerrit and
>> so on.
>>
>>
> I didn't see in the email chain any mention of the Atlassian tools. Can
> these be considered? They offer free licenses for open source projects and
> I think that most of the requirements are covered by BitBucket
> (repo-hosting, repo-serving, code review and pull requests) and Bamboo (CI).
>

I inquired about this generally on the nfp list in May:

https://archives.gentoo.org/gentoo-nfp/message/0142bac838ed7cb356e58447c6ee20b0

The conclusion was that the software needed to be released under an open
license (preferably FSF / OSI approved.) So for example, Gitlab CE is OK
(its an OSI approved licensed) but Gitlab EE is not. The source for Gitlab
EE is open (I can go read it) but it's not freely available.


>
> If desired I can setup a demo and/or help maintain the tools, this is my
> day job and for those who care about these kind of things I am Atlassian
> certified.
>
> Here is more info on their licensing for Open Source projects:
>
> https://www.atlassian.com/software/views/open-source-license-request
>

My understanding is that this is a binary-own download that, once
installed, we can request a license to use. We won't have the source code
at all and the license is not free. I don't think it's possible for us to
use this software in Gentoo for this reason.

-A


>
> br,
> Mathy
>

Op di 2 jun. 2020 om 19:51 schreef Alec Warner <antarus@gentoo.org>:

>
> The conclusion was that the software needed to be released under an open
> license (preferably FSF / OSI approved.) So for example, Gitlab CE is OK
> (its an OSI approved licensed) but Gitlab EE is not. The source for Gitlab
> EE is open (I can go read it) but it's not freely available.
>

Ok, I'm not on that list, hence I didn't know the requirement (and would
have responded in May already)


> My understanding is that this is a binary-own download that, once
> installed, we can request a license to use. We won't have the source code
> at all and the license is not free. I don't think it's possible for us to
> use this software in Gentoo for this reason.
>
>
Actually when you have a license you can download the source code from
their service portal. This is a fact for commercial licenses but I'm not
sure if it also applies to open source licenses.


In any case, I just wanted to offer the suggestion and my help. If it isn't
a match, that's just as fine for me ;-)

br,
Mathy