Mailing List Archive

Ebuild limits?
Well, I've been plugging along happily on constructing an ebuild for zimbra.

But because of a) their built-in hardwiring and b) ant-based build/install
system, my ebuild is starting to get pretty big and complex.

In order to keep my own thought patterns straight, I've isolated a lot of the
code in src_compile and src_install to a number of different functions.

But as the size of the ebuild continues to grow, I'm now starting to ask
myself if, as it is currently going, the ebuild might not be accepted.

So I guess I need to know:

a) are there limits to the size and/or complexity of the ebuilds that are
accepted into portage?

b) If there are and I end up going beyond them, is there a recommended
methodology for relocating some of the functionality? I mean, can I put code
into scripts in the files directory and freely execute from there?

Just wondering...

Dave
--
gentoo-dev@gentoo.org mailing list
Re: Ebuild limits? [ In reply to ]
On Sun, 02 Oct 2005 19:58:19 -0400 Dave Nebinger <dnebinger@joat.com>
wrote:
| a) are there limits to the size and/or complexity of the ebuilds that
| are accepted into portage?

Well...

* Whoever ends up maintaining said ebuild must understand it.
* Too much complexity is often a sign that you're doing something
wrong.
* Too much complexity often suggests that an eclass is needed.
* Too much complexity often suggests that upstream's build system is
annoyingly broken.

| b) If there are and I end up going beyond them, is there a
| recommended methodology for relocating some of the functionality? I
| mean, can I put code into scripts in the files directory and freely
| execute from there?

eclass.

--
Ciaran McCreesh : Gentoo Developer (Vim, Shell tools, Fluxbox, Cron)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm
Re: Ebuild limits? [ In reply to ]
On Sunday 02 October 2005 08:08 pm, Ciaran McCreesh wrote:
> On Sun, 02 Oct 2005 19:58:19 -0400 Dave Nebinger <dnebinger@joat.com>
>
> wrote:
> | a) are there limits to the size and/or complexity of the ebuilds that
> | are accepted into portage?
>
> Well...
>
> * Too much complexity is often a sign that you're doing something
> wrong.

Well that's always the possiblity ;-)

> * Too much complexity often suggests that upstream's build system is
> annoyingly broken.

This is it. Zimbra, if you're not aware of it yet, is another one of those
LAMP-like distributions with a twist. Instead of Apache, it's based upon
Tomcat, and their distribution includes a ton of other stuff (i.e. postfix,
mysql, jdk, etc.) that gentoo's already got.

So how I'm approaching it is a little twisted, but the basic steps are:

1. unpack zimbra's distribution.
2. apply patches to get their initial build to live within the sandbox rather
than /opt/zimbra (like they expect).
3. apply patches to their iniitialization scripts to have the configuration
scripts they would normally build in /opt/zimbra in the sandbox, then run
said scripts.
4. Combine info from those configs, known values at build time, and values
from existing (installed) configs, swing back through the code base applying
patches and sed scripts to make the code conform to the local system.
5. build the distribution war files in the sandbox.
6. for installation, re-distribute the files from their scatter-brained
distribution to use a more FHS and/or Gentoo friendly approach.

Step 4 is necessary because they store a lot of default values (inappropriate
values for an existing package installation) within their ldap (populated
from an ldif) as well as hard-coding in the source files (for values they
choose not to go to ldap to retrieve). This is further complicated because
there are shell scripts, java source files, and configuration files to make
passes through to fix things up.

So yeah, things are pretty complicated due to the upstream build system
expectations.

The good news is that the ebuild is full of embedded comments ;-)

All in all, though, this has been a great ebuilding educational experience.
Such a large distrib to boil down into it's component parts, coordinating
unpacking vs compiling vs installing, etc. I'm pretty much exposing myself
to all the knooks and crannies of portage ;-)

> | b) If there are and I end up going beyond them, is there a
> | recommended methodology for relocating some of the functionality? I
> | mean, can I put code into scripts in the files directory and freely
> | execute from there?
>
> eclass.

Cool, eclass it is, but a few more questions arise out of this direction:

1. Can I use /usr/local/portage/eclass for development/testing?
2. Do I create a single eclass, i.e. zimbra.eclass, to represent an eclass for
a specific package or do I generate multiple eclasses based upon
functionality? For example, I've got a gres function, a /var/db/pkg query
system, etc.
3. Do eclasses get submitted on the same bug report as the ebuild submission,
or do they get a bug report of their own?
--
gentoo-dev@gentoo.org mailing list
Re: Ebuild limits? [ In reply to ]
On Sun, 02 Oct 2005 20:48:27 -0400 Dave Nebinger <dnebinger@joat.com>
wrote:
| So yeah, things are pretty complicated due to the upstream build
| system expectations.

I suggest hitting upstream with a cluebat until they make a proper
build system.

| 1. Can I use /usr/local/portage/eclass for development/testing?

Portage is kinda broken with eclasses in overlay... It'll work, so long
as you remember to touch all affected ebuilds every time you change the
eclass.

| 2. Do I create a single eclass, i.e. zimbra.eclass, to represent an
| eclass for a specific package or do I generate multiple eclasses
| based upon functionality?

Well... If it's really an eclass for a whole package, it may not be
worth packaging the thing at all.

| a /var/db/pkg query system,

Yick! Bad bad bad idea.

| 3. Do eclasses get submitted on the same bug report as the ebuild
| submission, or do they get a bug report of their own?

All in one bug's usually easier.

--
Ciaran McCreesh : Gentoo Developer (Vim, Shell tools, Fluxbox, Cron)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm
Re: Ebuild limits? [ In reply to ]
On Sunday 02 October 2005 09:28 pm, Ciaran McCreesh wrote:
> On Sun, 02 Oct 2005 20:48:27 -0400 Dave Nebinger <dnebinger@joat.com>
>
> wrote:
> | So yeah, things are pretty complicated due to the upstream build
> | system expectations.
>
> I suggest hitting upstream with a cluebat until they make a proper
> build system.

Their build system suits their purpose - distribute a LAMP-like system for the
foundation of their web application. I'm sure it will keep them from getting
distracted from questions like 'zimbra works for postfix 2.2 but breaks for
2.2.3'; they provide it all and to use it you're normally stuck with their
3rd party binaries at the version/patch level they give you.

So no matter how hard or how long I was going to hit them with the cluebat,
they're not going to bend.

Their distribution is structured as follows:
/opt/zimbra
/opt/zimbra/bin
/opt/zimbra/conf
...
/opt/zimbra/postfix
/opt/zimbra/postfix/bin
/opt/zimbra/postfix/etc
...
/opt/zimbra/tomcat
/opt/zimbra/tomcat/bin
/opt/zimbra/tomcat/conf
...

Yada yada yada. The alterations to get it to fit under a regular distribution
make the resulting ebuild pretty large.

All for the goal of extracting two web applications and some configuration
info. Go figure.

> | 2. Do I create a single eclass, i.e. zimbra.eclass, to represent an
> | eclass for a specific package or do I generate multiple eclasses
> | based upon functionality?
>
> Well... If it's really an eclass for a whole package, it may not be
> worth packaging the thing at all.

That's the rub. Create eclass(es) to make a single ebuild less complex, or
skip the eclasses and have a larger ebuild.

> | a /var/db/pkg query system,
>
> Yick! Bad bad bad idea.

Yeah, I know. But how else do you answer the question "Hey, Portage, where
did you really install that my.cnf file?"

Obviously the system admin is free to move the my.cnf file or even use a
different file/path altogether. But at least it would give me a starting
value to use at compile time...
--
gentoo-dev@gentoo.org mailing list
Re: Ebuild limits? [ In reply to ]
On Sun, 02 Oct 2005 21:50:07 -0400 Dave Nebinger <dnebinger@joat.com>
wrote:
| Their build system suits their purpose - distribute a LAMP-like
| system for the foundation of their web application. I'm sure it will
| keep them from getting distracted from questions like 'zimbra works
| for postfix 2.2 but breaks for 2.2.3'; they provide it all and to use
| it you're normally stuck with their 3rd party binaries at the
| version/patch level they give you.

Hrm. Does this really need an ebuild? Wouldn't it be better to use the
associated portage-provided packages? Also, how do you intend to handle
security updates?

| > | a /var/db/pkg query system,
| >
| > Yick! Bad bad bad idea.
|
| Yeah, I know. But how else do you answer the question "Hey, Portage,
| where did you really install that my.cnf file?"
|
| Obviously the system admin is free to move the my.cnf file or even
| use a different file/path altogether. But at least it would give me
| a starting value to use at compile time...

I'm starting to think you really shouldn't be ebuilding this lot at
all...

--
Ciaran McCreesh : Gentoo Developer (Vim, Shell tools, Fluxbox, Cron)
Mail : ciaranm at gentoo.org
Web : http://dev.gentoo.org/~ciaranm
Re: Ebuild limits? [ In reply to ]
On Monday 03 October 2005 12:47 am, Ciaran McCreesh wrote:
> On Sun, 02 Oct 2005 21:50:07 -0400 Dave Nebinger <dnebinger@joat.com>
>
> wrote:
> | Their build system suits their purpose - distribute a LAMP-like
> | system for the foundation of their web application. I'm sure it will
> | keep them from getting distracted from questions like 'zimbra works
> | for postfix 2.2 but breaks for 2.2.3'; they provide it all and to use
> | it you're normally stuck with their 3rd party binaries at the
> | version/patch level they give you.
>
> Hrm. Does this really need an ebuild?

I think so. As a stand-alone package it has a number of dependencies that
need to be managed, something Portage handles quite well.

Also, I'm finding that at it's core, portage handles the reorganization of a
distribution quite nicely. Being able to automagically handle the java
dependencies using java-pkg is a godsend. Itematically choosing the
individual files from the distribution build and retargeting them
to /usr/share/webapp with a one-line command... The advantages appear to
outweigh the costs.

> Wouldn't it be better to use the
> associated portage-provided packages?

That's the goal - strip out the parts from their distribution that have
exiting components already within the portage tree (and already installed on
my system). Boils down to a couple of wars for the most part, plus scripts,
things like that.

> Also, how do you intend to handle
> security updates?

For the most part I'd leave that to Portage for the 3rd party components. I'm
expecting that once I get the initial ebuild figured out, when they release a
new distribution I'll just have to re-diff & re-patch if the current patches
stop working.

> | > | a /var/db/pkg query system,
> | >
> | > Yick! Bad bad bad idea.
> |
> | Yeah, I know. But how else do you answer the question "Hey, Portage,
> | where did you really install that my.cnf file?"
> |
> | Obviously the system admin is free to move the my.cnf file or even
> | use a different file/path altogether. But at least it would give me
> | a starting value to use at compile time...
>
> I'm starting to think you really shouldn't be ebuilding this lot at
> all...

I understand your doubts, and trust me I have them to. But the way I see it I
have two basic choices:

1. use their distribution. This discards all of the advantages that portage
provides (i.e. updates to the foundation software). Also means that for each
component in their system that is already on mine, I need to shut mine
down/unmerge them so the system will rely upon theirs. Will really mess up
things like openldap and mysql dependent projects as their distribution
doesn't provide the full development stuff portage would need to handle
builds.

2. create an ebuild to merge only the necessary components into my system,
taking advantage of the fact that I already have working components
installed.

It would seem to me that #2 is the appropriate gentoo-way...

I started working on this because I was initially looking for a good web mail
package to use on my gentoo box (I'm starting a new contract where I won't
have pop3/imap access to my email from the job site). Scope soon creeped to
become a good collaboration suite to use on my gentoo box (because I'd have
calendars, address books, etc. scattered across many different sources).

While I was checking out the options that were already in portage I was also
checking online for other possiblilities. The recent posting to slashdot on
the very subject caught my eye, and when I went through the flash demo at
http://www.zimbra.com I was really impressed with everything I saw.

But to get it up and running I soon discovered I'd have to make one of the
choices from above. Eventually some 1,000 year old geezer might say "He
chose poorly," but who knows - it might work and might catch on.
--
gentoo-dev@gentoo.org mailing list
Re: Ebuild limits? [ In reply to ]
Dave Nebinger posted <200510030146.13596.dnebinger@joat.com>, excerpted
below, on Mon, 03 Oct 2005 01:46:13 -0400:

>> Wouldn't it be better to use the
>> associated portage-provided packages?
>
> That's the goal - strip out the parts from their distribution that have
> exiting components already within the portage tree (and already installed on
> my system). Boils down to a couple of wars for the most part, plus scripts,
> things like that.

If you haven't looked at them yet (you may be well past this, it sounds
like you are well farther along than I am in ebuilding experience -- I
don't think I'd be tackling that job)...

* Check out the split kde ebuilds (konqueror, kcontrol, etc, instead of
kdebase, kmail, knode, the various handheld/PIM builds, instead of kdepim,
etc) to see how an existing in-portage solution manages taking only part
of the source and configuring/building based on already merged components
from the rest of the source.

* Check out the xorg-x11 builds, for a good example of a rather complex
ebuild, broken down into logically digestible pieces, in the form of
multiple functions within the ebuild.

* Both of these use eclasses as well, but the kde ones especially should
be instructive for their use of the kde eclasses to handle the various
split functionality, if you decide to go with eclasses for part of it.

Just because it might be of help...

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman in
http://www.linuxdevcenter.com/pub/a/linux/2004/12/22/rms_interview.html


--
gentoo-dev@gentoo.org mailing list