Mailing List Archive

On Mon, 29 Aug 2005, Grobian wrote:

> Kito wrote:
> >
> > On Aug 27, 2005, at 4:32 PM, Grobian wrote:
> >

[snip]

>
> > Do the initial import of the minimum required packages from the main
> > tree, which shouldn't be too hard, basically a stage1 (see a
> > packages.build file in one of the linux profiles for instance) give or
> > take some things.
> >
> > Create a new profile(s), which should probably be a complete tear
> > down, Finn Thain has had some great suggestions in this area. FINN!
> > Care to jump in here?
>
> ^^^ Finn ^^^ :D

Sure. The idea was this profile inheriance tree:


.-> ppc-od
.-> ppc-darwin <
/ `-> ppc-macos
default-darwin <
\ .-> x86-od
`-> x86-darwin <
`-> x86-macos


The purpose being, "progressive" and "collision-protect" are best given
their own profiles in order to avoid having ebuilds test for the
"collision-protect" feature.

So, I was advocating that an "upstream darwin" profile, {x86,ppc}-darwin,
should be used for anything that didn't bow down to mac os. Hence, the
ppc-macos profile would be taken to mean "behave as a secondary package
manager: play nice with mac os, and test the mac os package database for
dependencies, and use the mac os installer to install them if they are
available in a .pkg".

At the moment, ppc-macos would need the collision-protect feature, but
once we have a feature for prefixed installs, that should be used instead
(by default, at least).

> > The user interface would need to be hashed out as well of course. How
> > do you install/bootstrap it?

It would be nice to have ebuilds that could invoke the Darwin Build
Scripts (and merge the result on a ROOT).

http://www.opendarwin.org/projects/darwinbuild/releases/

Given such ebuilds, surely catalyst can bootstrap it.

> > Where is the local configuration data stored? This is an area that I
> > think would be acceptable to take some Mac OS specific indulgences,
> > such as plists for the main config data(prefix info, search paths,
> > etc), pkgs/dmgs to bootstrap/install, and I also think we should abuse
> > the umbrella Framework mechanism when feasible.
>
> Wow, using plists would be a first start on getting portage widely
> accepted because it includes the buzz word XML. LOL. On a serious
> note, I think Apple has shown XML can work somehow. At least it has an
> open character, and it's great when you can 'script' your Keynote
> presentation by just doing string manipulation in a big XML file.
>
> So I would say, let's try to use this horrible XML on a pilot base for
> small configuration files. Maybe we should do it better than Apple at
> some point because their XML does not always make use of the tree
> structure of XML. For XML I would say: only deal with it if it looks
> appropriate for the case and it is relatively easy to extract the data
> (which is often very flat, as in the .plist files).

I reckon XML is important, though perhaps not in the way you describe. As
I see it, where ever portage is deployed as a secondary package manager,
it needs to consult the primary one. That means that there needs to be a
standard protocol for one package manager to query another.

In the mac os case, we can write a shim to talk XML to portage.

(BTW, a similar protocol, but allowing one package manager to _update_
another could be used to make a universal binary installer. But it would
need rpm, apt-get, portage etc to all implement a standard XML protocol
and API. But this is only really interesting in a closed source context.
And I've digressed. Mentioning XML just reminded me.)

> Let's indeed make it a 'native' application for OSX users. Native in
> the sense of how it installs and looks like. I may give file locations a
> thought today, but maybe I should know the Framework stuff a bit better
> first. Can we install the whole Gentoo stuff in a Framework? or is it
> better to just use a shortest path algorithm and end with /gentoo?
> Actually the user should be able to select a disk to install to during
> install...

I reckon get it working (with an "upstream darwin" profile) in a chroot
stage first (which could double as a boot disk).

[snip]

> > Some random broad philosophy/design goals that I think should be
> > stated...:
> >
> > The repo should never ever never ever EVER rely on anything it
> > doesn't know how to supply itself with, whether that be a tool, a
> > library, knowledge of a filesystem, a host, a protocol, whatever. It
> > doesn't matter how it gets it, it just needs to know at least one way
> > to get it. This implies of course proper implementation of ferringbs
> > BDEPENDS idea.
>
> so, you mean if there is something (a filesystem) portage hasn't
> installed, then we should create the proper handles to deal with the OS
> provided one? As in create a compatibility tool for "fdisk.HFS+". I'm
> a bit clueless on how exactly you want to achieve what you want. Maybe
> I don't understand correctly what you want exactly too.

This is how I would handle that case:

If a program (say fsck_hfs) is available upstream, build it with Dawin
Build, merge it with portage (I expect an eclass for darwin build is
required, and of course an ebuild for diskdev_cmds.)

If a package (say macext2fs) is available as a .pkg or .dmg, download it
with portage, and install it with mac os installer(8). Again, an eclass to
handle these would be useful (this is probably how they are handled
already?)

> > Although we want this for ppc-macos at the moment, it should not be
> > specific to either of those things, ppc, or macos. Adhering to this
> > rule assumes a lot...again the BDEPENDS issue, and keeping as close to
> > the main tree as possible, with those things in place, and done
> > properly(i.e. what it REALLY takes to bootstrap an
> > {x86,amd_64,ppc,ppc64,mips,sh,whatever}-{linux,*bsd,darwin,macos,whatever}
> > toolchain) , you have a sane cross-compile ready repo, that is pretty
> > damn portable.

Darwin is not self-hosting (out of the box). You need a carefully
contrived a build environment to make it so, and this is what the Darwin
Build Scripts do: install that build environment, and automate the builds.

Darwin build uses plists to describe all the packages in a mac os release,
along with build dependencies, and so forth. These build dependencies are
confined to the darwin build chroot, as I understand it. Portage wouldn't
have to know about them, I think it just has to do the merge of the build
product and install the runtime dependencies thereof (of course).

> That would be really great.
>
> > Binary packages have to work. Thats a fun one. But all this done
> > properly, should allow for at least a little more consistency than the
> > current situation. I'm still sold on using xar[1] for this despite the
> > rather heavy deps (they are deps I would want in most any environment
> > anyway - libcurl,ssl,libxml,zlib), and it just fits the bill too well
> > imho, support for most standard arch specific data such bsd flags,
> > ACLs, resource forks ,.etc as well an excellent TOC structure that is
> > perfect for storing environment settings and package metadata.
>
> Again XML. If you do it XML, you have to do it all XML, something Apple
> apparently understood. It appears we will have the blessings if we use
> XML, so I think we should. In the end we can always dump all that XML
> into MonetDB/XQuery to have extremely fast querying over all the files,
> tree based. I think it would seriously be the first project to use
> XQuery and XML for it's configuration. However, if you come to think of
> it, it is a tree, an extensible tree, and might be a much more a logical
> choice than it appears to be.

Why not use one of the open source .pkg tools to generate binary packages?
Kito tells me he has already been able to unpack .pkg format and install
it via portage.

> > Avoid package.provided or anything of its likeness like the plague.
> > This repo needs to describe a toolchain from the ground up, regardless
> > of the host. "What CAN it build and how?", not "What WON'T the host OS
> > let me do".
>
> Uhm, yes please!

Hear, hear!

package.provided must die. In the long run, it is not sustainable to
attempt to use profiles to describe a moving target. If you do that, you
end up with not one profile for 10.4, but profiles for 10.4, 10.4.1 and
10.4.2 because, for example, 10.4.2 fixed a bug in apple's xyz package
that forced apples xyz to be masked in the 10.4.1 profile.

And then you still have the problem that the user may actually be running
10.4.6, thanks to a recent Software Update, while the latest profile is at
10.4.4, thanks to an old emerge --sync, all while /etc/make.profile is
still a symlink to the 10.4.1 profile, thanks to the user neglecting to
re-point it since installation!

> > Keep the number of ebuilds to a bare minimum. We can't get too
> > carried away with maintaining a complete tree, or we risk drifting too
> > far downstream and the zealots pushing Darwin/Mac OS support out of
> > the main tree entirely. That would be bad. This can't go in the
> > direction of a fork, just an experimental branch that will be merged
> > back in at some point.

IMHO, this sounds like a "gentoo-darwin" sub-project to gentoo-alt,
along-side os x and bsd. It isn't really a fork except in as much as the
profile arrangement doesn't really accomodate work on darwin proper (but
then the profile arrangemnet is flawed anyway: it only exists this way
because of the package.provided crutch).

-f
--
gentoo-osx@gentoo.org mailing list

On Aug 29, 2005, at 2:07 AM, Brian Harring wrote:

> Kindly cc me on responses- I'm not on the osx mailing list anymore due
> to too much fricking mail per day :)
>
>>> From: kito
>>
>
>
>>> In the interim it can simply be an overlay, using a portage
>>> snapshot, giving us free reign to do what is needed to get the
>>> important things working without having to worry about Apple
>>> provided files. Then with some simple PATH mangling(think finks /sw/
>>> init) we can start making use of it now and actually be a step
>>> ahead of the game.
>
> You're not stating how the path mangling will occur, without
> addressing this the future merging of this proposed tree and the
> mainline tree is questionable.

Fink handles this using a patched bash with a special set of init
scripts [1]. We would probably have to do something similar, but a
lot more flexible and dynamic of course, we have to support a lot
more archs. Even the current profile setup should offer some of the
functionality required to achieve this.

>
>
>>> Doing this outside the main tree has IMHO worked quite well for g/
>>> fbsd project, and allowed them to get their base-system in order
>>> before they had to go mucking up the tree with hacks that may or
>>> may not be permanent, and is also the reason they aren't hated
>>> quite as much as us by the other projects...
>
> Different angles though; their goal was to nail down a base, and
> integrate the changes *back* into the tree.

Same goal. As I said in my other email, getting too far away from
main tree must be avoided.

>
> Why am I implying this would basically result in a fork of the tree?
> Their modifications were patches, tweaks to the existing ebuilds such
> that they played nice with *bsd; yes, bit more complex then that, but
> roughly the jist.
>
> Swiveling the offset isn't just swiveling root; any alt offset is
> going to require making the ebuild aware of the offset. How?

My current thoughts on this is to store all the package metadata in
Xarchives, probably do it as a portage feature. So we use something
like the ICANINSTALLTO var (gotta find a better name for that, ugh)
and each offset filesystem structure is stored in the archive TOC,
with the optional actual binary package as well if enabled in
FEATURES, along with all the other typical data stored in /var/db/
pkg, almost like a tiered bom(5) file + a lot more.

Initially in practice we would probably ONLY swivel /, with a flat
set of dependencies, and then work up from there.

If we have accurate *DEPENDS, we just have to make sure portage has
sufficient knowledge of how to supply them for every offset.

>
> Likely introduction of a new feature/var- how is this going to be
> built up in a seperate repo, then brought back and merged into the
> tree?

EAPI comes to mind, if this is done right, we start establishing an
ebuild API version that supports offsets.

>
> Mind you I'm not stating that for osx stuff it should be installed
> to / ;
> I'm just trying to point out that the issue of having the offset
> dynamic isn't really addressed, all y'all can do if you don't address
> that issue is build up a repo of ebuilds that have a different set of
> hardcodings, effectively a fork that cannot be merged with the tree
> till a proper solution for swivelling the offset is created.

I'm definitely not interested in a bunch of hacked ebuilds with a
hardcoded prefix. The whole goal would be doing it in a manner where
no matter how/when a stable portage supports it, we would be ready,
and hopefully helped in ushering it along.

>
> The nasty details of it can be found in the -dev thread from a few
> months back re: ICANINSTALLTO ; personally, it's something I'd like
> but not something willing to push atm, due to the fact already have
> enough people screaming that I'm trying to shove more work onto devs.

We would be the devs to not mind some more work =) Instead of an 8
month long ML thread, we can use this as a sandbox and a constant
running proof-of-concept. Should also be a good place to get the UI
side of the portage rewrite hashed out, as emerge and repoman would
definitely need some serious tweaking for this.

>
> Either way, you could split out the bases, but the point re: how to
> have those bases used by other packages (whether via modification of
> ebuilds, or portage) needs to be addressed, and imo someone should
> have a pretty good idea of the dynamic offset bit, so that the repo
> can be merged back in down the line. Yes, overblowing it a bit re: the
> packages that would depend on the base, but it's possible and likely
> will be raised by others if you do this.

Agreed, and I don't think you are overblowing the need to do it
right, which is why i CC'd you, I'd like to have the portage people
involved as much as possible.

--Kito

[1] http://cvs.sourceforge.net/viewcvs.py/fink/base-files/init.sh.in?
rev=1.19&view=markup
--
gentoo-osx@gentoo.org mailing list

On Aug 29, 2005, at 5:30 AM, Finn Thain wrote:

>
>
> On Mon, 29 Aug 2005, Grobian wrote:
>
>> Kito wrote:
>>>
>>> On Aug 27, 2005, at 4:32 PM, Grobian wrote:
>
> At the moment, ppc-macos would need the collision-protect feature, but
> once we have a feature for prefixed installs, that should be used
> instead
> (by default, at least).
>
>>> The user interface would need to be hashed out as well of course.
>>> How
>>> do you install/bootstrap it?
>
> It would be nice to have ebuilds that could invoke the Darwin Build
> Scripts (and merge the result on a ROOT).

Yeap. already planned on using this to build a libSystem, etc.

>
> http://www.opendarwin.org/projects/darwinbuild/releases/
>
> Given such ebuilds, surely catalyst can bootstrap it.
>
>>> Where is the local configuration data stored? This is an area that I
>>> think would be acceptable to take some Mac OS specific indulgences,
>>> such as plists for the main config data(prefix info, search paths,
>>> etc), pkgs/dmgs to bootstrap/install, and I also think we should
>>> abuse
>>> the umbrella Framework mechanism when feasible.
>>
>> Wow, using plists would be a first start on getting portage widely
>> accepted because it includes the buzz word XML. LOL. On a serious
>> note, I think Apple has shown XML can work somehow. At least it
>> has an
>> open character, and it's great when you can 'script' your Keynote
>> presentation by just doing string manipulation in a big XML file.
>>
>> So I would say, let's try to use this horrible XML on a pilot base
>> for
>> small configuration files. Maybe we should do it better than
>> Apple at
>> some point because their XML does not always make use of the tree
>> structure of XML. For XML I would say: only deal with it if it looks
>> appropriate for the case and it is relatively easy to extract the
>> data
>> (which is often very flat, as in the .plist files).
>
> I reckon XML is important, though perhaps not in the way you
> describe. As
> I see it, where ever portage is deployed as a secondary package
> manager,
> it needs to consult the primary one. That means that there needs to
> be a
> standard protocol for one package manager to query another.
>

I'm not sure I agree. I think this gets too close to a
package.provided situation, portage will never know enough about
another systems packages to map their functionality to its own. I
think its preferable to let portage handle what it knows first hand
before trying to force it data from a foreign host.

>
>> Let's indeed make it a 'native' application for OSX users. Native in
>> the sense of how it installs and looks like. I may give file
>> locations a
>> thought today, but maybe I should know the Framework stuff a bit
>> better
>> first. Can we install the whole Gentoo stuff in a Framework? or
>> is it
>> better to just use a shortest path algorithm and end with /gentoo?
>> Actually the user should be able to select a disk to install to
>> during
>> install...
>
> I reckon get it working (with an "upstream darwin" profile) in a
> chroot
> stage first (which could double as a boot disk).
>

I want to start a lot smaller than that first. Think stage1. You
can't boot a stage1, its a just the corelibs and a toolchain pretty
much.



>>> The repo should never ever never ever EVER rely on anything it
>>> doesn't know how to supply itself with, whether that be a tool, a
>>> library, knowledge of a filesystem, a host, a protocol, whatever. It
>>> doesn't matter how it gets it, it just needs to know at least one
>>> way
>>> to get it. This implies of course proper implementation of ferringbs
>>> BDEPENDS idea.
>>
>> so, you mean if there is something (a filesystem) portage hasn't
>> installed, then we should create the proper handles to deal with
>> the OS
>> provided one? As in create a compatibility tool for "fdisk.HFS
>> +". I'm
>> a bit clueless on how exactly you want to achieve what you want.
>> Maybe
>> I don't understand correctly what you want exactly too.
>
> This is how I would handle that case:
>
> If a program (say fsck_hfs) is available upstream, build it with Dawin
> Build, merge it with portage (I expect an eclass for darwin build is
> required, and of course an ebuild for diskdev_cmds.)

About what I was thinking...

>>
>>> Binary packages have to work. Thats a fun one. But all this done
>>> properly, should allow for at least a little more consistency
>>> than the
>>> current situation. I'm still sold on using xar[1] for this
>>> despite the
>>> rather heavy deps (they are deps I would want in most any
>>> environment
>>> anyway - libcurl,ssl,libxml,zlib), and it just fits the bill too
>>> well
>>> imho, support for most standard arch specific data such bsd flags,
>>> ACLs, resource forks ,.etc as well an excellent TOC structure
>>> that is
>>> perfect for storing environment settings and package metadata.
>>
>> Again XML. If you do it XML, you have to do it all XML, something
>> Apple
>> apparently understood. It appears we will have the blessings if
>> we use
>> XML, so I think we should. In the end we can always dump all that
>> XML
>> into MonetDB/XQuery to have extremely fast querying over all the
>> files,
>> tree based. I think it would seriously be the first project to use
>> XQuery and XML for it's configuration. However, if you come to
>> think of
>> it, it is a tree, an extensible tree, and might be a much more a
>> logical
>> choice than it appears to be.
>
> Why not use one of the open source .pkg tools to generate binary
> packages?
> Kito tells me he has already been able to unpack .pkg format and
> install
> it via portage.

Thats just to get extract files...I'm talking about binary packages
that portage can use. I don't like the current tbz2 hack.

I have no interest personally in producing packages with a
proprietary format for portage. Be a nice feature for OS X users and
devs sure, but thats more like an add-on bells-and-whistles type
feature... patches accepted ;)

>
>>> Avoid package.provided or anything of its likeness like the
>>> plague.
>>> This repo needs to describe a toolchain from the ground up,
>>> regardless
>>> of the host. "What CAN it build and how?", not "What WON'T the
>>> host OS
>>> let me do".
>>
>> Uhm, yes please!
>
> Hear, hear!
>
>>> Keep the number of ebuilds to a bare minimum. We can't get too
>>> carried away with maintaining a complete tree, or we risk
>>> drifting too
>>> far downstream and the zealots pushing Darwin/Mac OS support out of
>>> the main tree entirely. That would be bad. This can't go in the
>>> direction of a fork, just an experimental branch that will be merged
>>> back in at some point.
>
> IMHO, this sounds like a "gentoo-darwin" sub-project to gentoo-alt,
> along-side os x and bsd. It isn't really a fork except in as much
> as the
> profile arrangement doesn't really accomodate work on darwin proper
> (but
> then the profile arrangemnet is flawed anyway: it only exists this way
> because of the package.provided crutch).

I was looking at it more as a place to develop some new portage
features...Gentoo/Darwin has always been lurking, this is more in the
area of just getting offsets working.

>
> -f
> --
> gentoo-osx@gentoo.org mailing list
>

--
gentoo-osx@gentoo.org mailing list

> On Aug 29, 2005, at 5:30 AM, Finn Thain wrote:
>
> >I reckon XML is important, though perhaps not in the way you describe.
> >As I see it, where ever portage is deployed as a secondary package
> >manager, it needs to consult the primary one. That means that there
> >needs to be a standard protocol for one package manager to query
> >another.

(I should have mentioned, you don't really need XML for this, you can
generate /etc/package.provided by a wrapper around emerge. Then you can
mask untested stuff in it using a single ppc-macos profile and avoid the
bogus package.provided files in all the present macos profiles.)

> I'm not sure I agree. I think this gets too close to a package.provided
> situation, portage will never know enough about another systems packages
> to map their functionality to its own. I think its preferable to let
> portage handle what it knows first hand before trying to force it data
> from a foreign host.

I'm not proposing that one "injects" non-identical packages under the same
names. Actually, I have been against that since the beginning.

I was thinking of something like, at run time, query the vendor package
manager and use the result to populate the tree with packages like
vendor-apple/sys-devel/xcode-1.5, vendor-sun/app-arch/cpio-x.y.z for
example (please substitute sgi, bsd-ports, redhat or debian etc if you are
hostile to any of my examples).

Apple's XCode is closed source, and sun's cpio is now open. The former
requires an ebuild to invoke installer(8), the latter requires an ebuild
to build it from source. No-one is lying to portage here.

And, if sys-apps/bsd-awk-x.y.z builds the same thing that apple ships, it
can provide vendor-apple/sys-apps/bsd-awk.

Also, the ebuilds for both vendor-apple/sys-apps/bsd-awk and
sys-apps/bsd-awk should provide virtual/awk. So, when arbitrary ebuild foo
wants generic awk (doesn't care about gnu extensions), it can depend on
that virtual (unless virtuals are to be deprecated, in which case foo
somehow has to depend on any vendor, including gentoo).

> >IMHO, this sounds like a "gentoo-darwin" sub-project to gentoo-alt,
> >along-side os x and bsd. It isn't really a fork except in as much as
> >the profile arrangement doesn't really accomodate work on darwin proper
> >(but then the profile arrangemnet is flawed anyway: it only exists this
> >way because of the package.provided crutch).
>
> I was looking at it more as a place to develop some new portage
> features...Gentoo/Darwin has always been lurking, this is more in the
> area of just getting offsets working.

OK, I see what you are getting at now. That was something that I failed to
infer from the email you forwarded to the list. Most of what I said in
reply isn't very relevant to that. Excepting that, if you can leverage
existing packages, prefixed installs are much more useful -- having a
complete set of deps installed on a prefix is not much better than a
stage3 chroot with your home directory bind mounted below it.

-f
--
gentoo-osx@gentoo.org mailing list

Sidenote, no need to cc, just joined the ml for the discussion in the
meantime since people occasionally forget to cc (I know I do).

On Tue, Aug 30, 2005 at 01:09:49PM +1000, Finn Thain wrote:
> > I'm not sure I agree. I think this gets too close to a package.provided
> > situation, portage will never know enough about another systems packages
> > to map their functionality to its own. I think its preferable to let
> > portage handle what it knows first hand before trying to force it data
> > from a foreign host.
>
> I'm not proposing that one "injects" non-identical packages under the same
> names. Actually, I have been against that since the beginning.
>
> I was thinking of something like, at run time, query the vendor package
> manager and use the result to populate the tree with packages like
> vendor-apple/sys-devel/xcode-1.5, vendor-sun/app-arch/cpio-x.y.z for
> example (please substitute sgi, bsd-ports, redhat or debian etc if you are
> hostile to any of my examples).
>
> Apple's XCode is closed source, and sun's cpio is now open. The former
> requires an ebuild to invoke installer(8), the latter requires an ebuild
> to build it from source. No-one is lying to portage here.
>
> And, if sys-apps/bsd-awk-x.y.z builds the same thing that apple ships, it
> can provide vendor-apple/sys-apps/bsd-awk.
>
> Also, the ebuilds for both vendor-apple/sys-apps/bsd-awk and
> sys-apps/bsd-awk should provide virtual/awk. So, when arbitrary ebuild foo
> wants generic awk (doesn't care about gnu extensions), it can depend on
> that virtual (unless virtuals are to be deprecated, in which case foo
> somehow has to depend on any vendor, including gentoo).
The rewrite's domain's abstraction (additionally the goal of binding
the resolver to the domain, and being able to do inter-domain
resolving) would allow for this, but I *really* don't think it'll work
well.

Reasoning is, how do you know that pkg xyz is actually the package
you're after? The expanded restrictions subsystem, specifically
ability to depends based on contents restrictions (I want the pkg that
owns file abc essentially) gives basic ability for this, but it
doesn't cover the abi angle.

What you're proposing could sort of be hacked together to pull off
strictly for src compiles, probably with a good collection of
impossible to quash annoying bugs. Doing it for binaries is a helluva
lot harder though. :)

> > >IMHO, this sounds like a "gentoo-darwin" sub-project to gentoo-alt,
> > >along-side os x and bsd. It isn't really a fork except in as much as
> > >the profile arrangement doesn't really accomodate work on darwin proper
> > >(but then the profile arrangemnet is flawed anyway: it only exists this
> > >way because of the package.provided crutch).
> >
> > I was looking at it more as a place to develop some new portage
> > features...Gentoo/Darwin has always been lurking, this is more in the
> > area of just getting offsets working.
>
> OK, I see what you are getting at now. That was something that I failed to
> infer from the email you forwarded to the list. Most of what I said in
> reply isn't very relevant to that. Excepting that, if you can leverage
> existing packages, prefixed installs are much more useful -- having a
> complete set of deps installed on a prefix is not much better than a
> stage3 chroot with your home directory bind mounted below it.

The rewrite's general core is intended to allow for alt
formats/repos/whatever jammed into it; that said, making seperate
formats play nice with each other (unless they can natively) isn't
something I think is incredibly easy to pull off, as mentioned above.

Thoughts?
~harring

On Mon, 29 Aug 2005, Brian Harring wrote:

> The rewrite's domain's abstraction (additionally the goal of binding the
> resolver to the domain, and being able to do inter-domain resolving)
> would allow for this, but I *really* don't think it'll work well.
>
> Reasoning is, how do you know that pkg xyz is actually the package
> you're after?

Not sure I understand the problem... in glep 37, a virtual is replaced
with a meta-package having a list of depends alternatives. If you add the
vendor package to that list, and then make the vendor packages
"package.preferred" where portage is a secondary package manager, surely
there is no question that package xyz is the one you're after?

(I'm assuming all repos share the same namespace WRT package names... if
cpv's do not have to be unique accross repos, other solutions might be
possible...)

> The expanded restrictions subsystem, specifically ability to depends
> based on contents restrictions (I want the pkg that owns file abc
> essentially) gives basic ability for this

I don't understand why portage would go looking for deps in the
filesystem. If configure can't find them, does it help that portage can?

The last I read about this was here...
http://article.gmane.org/gmane.linux.gentoo.devel/28154
I guess there is more too it?

> but it doesn't cover the abi angle.

I hadn't considered the ABI problem. In the short term I think a synthetic
package.provided is okay, since most multilib machines cater to the
ABI-ignorant.

But, generating package.provided with a wrapper is a hack, so long term, a
vendor package manager (or a shim on top of it) would have to state what
ABI was used for each package, when queried. Knowing the ABIs of the
vendor packages, and knowing the native ABI of the target domain (?), the
resolver could probably do the right thing...

> What you're proposing could sort of be hacked together to pull off
> strictly for src compiles, probably with a good collection of impossible
> to quash annoying bugs. Doing it for binaries is a helluva lot harder
> though. :)

I think these are the bugs which have been the gentoo-osx team's burden
already... brave souls that they are :-)

> The rewrite's general core is intended to allow for alt
> formats/repos/whatever jammed into it; that said, making seperate
> formats play nice with each other (unless they can natively) isn't
> something I think is incredibly easy to pull off, as mentioned above.
>
> Thoughts?

I probably need a better handle on the constraints of the relations
between domains, repos, prefixes... anyway, here is my naive partial
config. This config is premature, of course, but it might help me
understand what is and is not possible in the rewrite if you would kindly
shoot some holes in it ;-)

Notes: I've used a hypothetical feature for collecting information about
vendor packages -- presumably the apple domain also needs an ephemeral vdb
to hold it (like package.provided). I don't know if this vendor domain
needs a repo. I guess the main repo would have to mask broken/testing
vendor deps as it sees fit, including those from other domains...

-f


[vdb]
type = repo
class = portage.installed_pkg.repository
path =

[rsync repo]
type = repo
class = portage.ebuild.repository
path = /opt/gentoo/usr/portage

[ppc-macos]
type = config
ACCEPT_KEYWORDS = "ppc-macos"

[default domain]
type = domain
root = "/opt/gentoo"
repositories = 'rsync repo' vdb
config = ppc-macos

[vendor-apple]
type = config
FEATURES = "query-apple-packages"
ACCEPT_KEYWORDS = "ppc-darwin"

[apple domain]
type = domain
root = "/"
config = vendor-apple



--
gentoo-osx@gentoo.org mailing list

On Tue, Aug 30, 2005 at 09:33:36PM +1000, Finn Thain wrote:
> On Mon, 29 Aug 2005, Brian Harring wrote:
>
<inserting the relevant quote>
> > > I was thinking of something like, at run time, query the vendor package
> > > manager and use the result to populate the tree with packages like
> > > vendor-apple/sys-devel/xcode-1.5, vendor-sun/app-arch/cpio-x.y.z for
> > > example (please substitute sgi, bsd-ports, redhat or debian etc if
> > > you are hostile to any of my examples).
> >
> > The rewrite's domain's abstraction (additionally the goal of binding the
> > resolver to the domain, and being able to do inter-domain resolving)
> > would allow for this, but I *really* don't think it'll work well.
> >
> > Reasoning is, how do you know that pkg xyz is actually the package
> > you're after?
Re-inserted the quote to clarify what I'm talking about; mapping
another pkg managers db into our own requires either a lot of human
intervention, or some dodgy rules that somewhat manage it, with bugs.

> Not sure I understand the problem... in glep 37, a virtual is replaced
> with a meta-package having a list of depends alternatives. If you add the
> vendor package to that list, and then make the vendor packages
> "package.preferred" where portage is a secondary package manager, surely
> there is no question that package xyz is the one you're after?
>
> (I'm assuming all repos share the same namespace WRT package names... if
> cpv's do not have to be unique accross repos, other solutions might be
> possible...)
If you're managing a list of provided packages, my points don't
matter; points were strictly in reference to trying to have portage
query another pkg manager (primary or otherwise) :)

> > The expanded restrictions subsystem, specifically ability to depends
> > based on contents restrictions (I want the pkg that owns file abc
> > essentially) gives basic ability for this
>
> I don't understand why portage would go looking for deps in the
> filesystem. If configure can't find them, does it help that portage can?
Quote above is in reference to the (potentially incorrectly perceived)
idea of directly querying another pkg manager.

> The last I read about this was here...
> http://article.gmane.org/gmane.linux.gentoo.devel/28154
> I guess there is more too it?
Haubi's glep/patches were moreso for pulling off shifted presets,
relevant to the shifted preset goal's y'all have, but not pertinent to
portion of the discussion involving attempting to query apple's
installed db (which is the proposal you put forth) :)

> But, generating package.provided with a wrapper is a hack, so long term, a
> vendor package manager (or a shim on top of it)
My question/concerns raised have been regarding the potential
shim/wrapper. How do you wrap their pkg manager's namespace into
portages namespace? To do so you need to either manage a list of
mappings yourself, or come up with rules (which will have exceptions).

That's what I'm getting at; people don't support interop between dpkg
and rpm (fex) due to the fact the namespaces are different (although
it's possible to do so if you account for abi).

> I probably need a better handle on the constraints of the relations
> between domains, repos, prefixes... anyway, here is my naive partial
> config. This config is premature, of course, but it might help me
> understand what is and is not possible in the rewrite if you would kindly
> shoot some holes in it ;-)
>
> Notes: I've used a hypothetical feature for collecting information about
> vendor packages -- presumably the apple domain also needs an ephemeral vdb
> to hold it (like package.provided). I don't know if this vendor domain
> needs a repo. I guess the main repo would have to mask broken/testing
> vendor deps as it sees fit, including those from other domains...
Offhand, what you're proposing (querying the vendor installed pkg db)
is actually a readonly vdb, if it were implemented.
It's an installed package database you can query (use for solving
deps), but cannot modify.

> [vendor-apple]
> type = config
> FEATURES = "query-apple-packages"
> ACCEPT_KEYWORDS = "ppc-darwin"
Definitions not totally right as mentioned (doesn't matter, just being
nitpicky :), but the crux of my concern is embodied inthe
FEATURES="query-apple-packages".

Mainly, how? How to query the vendor db, and map that back into
portage package namespace?

~harring

On Aug 29, 2005, at 10:26 PM, Brian Harring wrote:
>
> On Tue, Aug 30, 2005 at 01:09:49PM +1000, Finn Thain wrote:
>>>
>>> I was looking at it more as a place to develop some new portage
>>> features...Gentoo/Darwin has always been lurking, this is more in
>>> the
>>> area of just getting offsets working.
>> Excepting that, if you can leverage
>> existing packages, prefixed installs are much more useful -- having a
>> complete set of deps installed on a prefix is not much better than a
>> stage3 chroot with your home directory bind mounted below it.

Maybe so, but we can't have one without the other. First get packages
to install in a prefix, then work up from there. The issue of
leveraging existing packages is currently handled by
package.provided, which we all know doesn't really serve our
purposes, but I see no reason work couldn't be done in parallel on
these issues...if you have an ebuild repo talking to and
understanding a vendor repo such as apples Receipts or whatever, that
will work wether the packages get installed in a prefix or not,
likewise if we have packages that can be installed to an alt-prefix,
they should work regardless of how portage is resolving the deps.

> The rewrite's general core is intended to allow for alt
> formats/repos/whatever jammed into it; that said, making seperate
> formats play nice with each other (unless they can natively) isn't
> something I think is incredibly easy to pull off, as mentioned above.

Right, this would be a great feature, but I look at this as multi-
level deps, which should come later IMHO. My goal for having a branch
of some base packages is to hash out the namespace and all the other
issues of portage managing a flat set of deps under an offset root.
Once that is functional, making the offset repo talk to another repo,
regardless of vendor, host, location, etc could be looked at.

I know that its good to get a solid design before running off and
writing a bunch of code, but it seems to me the portage rewrite has
been thought out sufficiently to allow for this type of feature
expansion in the future, without limiting what we can do right now.

--Kito

--
gentoo-osx@gentoo.org mailing list

On Tue, 30 Aug 2005, Brian Harring wrote:

> On Tue, Aug 30, 2005 at 09:33:36PM +1000, Finn Thain wrote:
> > On Mon, 29 Aug 2005, Brian Harring wrote:
> >
> <inserting the relevant quote>
> > > > I was thinking of something like, at run time, query the vendor
> > > > package manager and use the result to populate the tree with
> > > > packages like vendor-apple/sys-devel/xcode-1.5,
> > > > vendor-sun/app-arch/cpio-x.y.z for example (please substitute sgi,
> > > > bsd-ports, redhat or debian etc if you are hostile to any of my
> > > > examples).
> > >
> > > The rewrite's domain's abstraction (additionally the goal of binding
> > > the resolver to the domain, and being able to do inter-domain
> > > resolving) would allow for this, but I *really* don't think it'll
> > > work well.
> > >
> > > Reasoning is, how do you know that pkg xyz is actually the package
> > > you're after?
> Re-inserted the quote to clarify what I'm talking about; mapping another
> pkg managers db into our own requires either a lot of human
> intervention, or some dodgy rules that somewhat manage it, with bugs.

OK, I see what you mean. You're asking, how does portage know that vendor
package xyz is the portage package abc?

Short answer is package.mask, meta-packages and name mapping.

A particular vendor package version is a known-good dep, as tested by
devs, otherwise it is masked. E.g. package.mask says
>vendor-sun/app-arch/cpio-x.y.z if no higher version has been tested. In
mac os, automated updates mean that most of the time, there will be some
vendor packages that the tree hasn't been tested against. These have to be
masked until the user does emerge sync.

Another part of the problem is solved by meta-packages. If the vendor dep
is still masked after emerge sync, it can't be used, and the next most
"package.preferred" dep will be built and installed straight from the tree
(on a prefix). Once the vendor dep is unmasked, depclean will remove it
again.

Virtuals/meta-packages already permit a bit of slack. The packages that
provide a virtual are not the same, of course, so any ebuild using a
virtual/meta-package as a dep is not going to be too fussy (e.g. needs awk
but not gnu extensions). vendor packages are a good fit here.

> > Not sure I understand the problem... in glep 37, a virtual is replaced
> > with a meta-package having a list of depends alternatives. If you add
> > the vendor package to that list, and then make the vendor packages
> > "package.preferred" where portage is a secondary package manager,
> > surely there is no question that package xyz is the one you're after?
> >
> > (I'm assuming all repos share the same namespace WRT package names...
> > if cpv's do not have to be unique accross repos, other solutions might
> > be possible...)
> If you're managing a list of provided packages, my points don't matter;
> points were strictly in reference to trying to have portage query
> another pkg manager (primary or otherwise) :)

BTW, do repos share a namespace? Presented with the same cpv in several
repos, is portage's behaviour defined yet?

> > > The expanded restrictions subsystem, specifically ability to depends
> > > based on contents restrictions (I want the pkg that owns file abc
> > > essentially) gives basic ability for this
> >
> > I don't understand why portage would go looking for deps in the
> > filesystem. If configure can't find them, does it help that portage
> > can?
> Quote above is in reference to the (potentially incorrectly perceived)
> idea of directly querying another pkg manager.
>
> > The last I read about this was here...
> > http://article.gmane.org/gmane.linux.gentoo.devel/28154 I guess there
> > is more too it?
> Haubi's glep/patches were moreso for pulling off shifted presets,
> relevant to the shifted preset goal's y'all have, but not pertinent to
> portion of the discussion involving attempting to query apple's
> installed db (which is the proposal you put forth) :)
>
> > But, generating package.provided with a wrapper is a hack, so long
> > term, a vendor package manager (or a shim on top of it)
> My question/concerns raised have been regarding the potential
> shim/wrapper. How do you wrap their pkg manager's namespace into
> portages namespace? To do so you need to either manage a list of
> mappings yourself, or come up with rules (which will have exceptions).

My feeling is that the burden of managing the mappings is better than the
burden of managing one package.provided for mac os 10.3, alongside another
for 10.4, etc. (If I'm wrong about that, then this exercise is pointless.)

In some cases the mapping is trivial: take the closed source XCode
package. I would have an ebuild called vendor-apple/sys-devel/xcode-1.5 to
install the DeveloperTools.pkg file, from a distfile taken from apple's
cd. So this ebuild would contain the forward mapping, by storing the
vendor's name in it's env. (Same applies to solaris pkgadd and irix
swinst...)

So the same ebuild also embodies the reverse mapping, and portage can map
a vendor's package-version to the ebuild itself, and populate a vdb with
the ebuild's PV. IOW, the ebuilds themselves provide the 1:M mapping of
vendor name to portage names.

Several vendor packages are sometimes required to make up one of portage's
deps (e.g. vendor-apple/x11-base/xfree would comprise both of X11SDK.pkg
and X11User.pkg). That ebuild would install two vendor packages and
contain two vendor names (basically an N:1 mapping).

Now, what if there is no ebuild to match a vendor package? No mapping. Dep
has to come from the tree instead. Not a big deal for a secondary package
manager.

But what if you want dev-tcltk/tcllib to satisfy an explicit dep on
vendor-apple/dev-tcltk/tcllib? You can't. But I can't see this being a
problem. (Though, you could have a metapackage for tcllib and
package.prefer dev-tcltk/tcllib in your profile.)

I would say that explicit deps on vendor packages can be avoided whereever
said package is open source and we have an ebuild to build it.

Of the open source stuff, those packages that are built from vendor
sources (or patches), are not "vendor packages" in the sense I've used the
term, that is, they don't get installed by a foriegn package manager, and
they don't live under /vendor-*. They are just regular ebuilds.

> That's what I'm getting at; people don't support interop between dpkg
> and rpm (fex) due to the fact the namespaces are different (although
> it's possible to do so if you account for abi).
>
> > I probably need a better handle on the constraints of the relations
> > between domains, repos, prefixes... anyway, here is my naive partial
> > config. This config is premature, of course, but it might help me
> > understand what is and is not possible in the rewrite if you would
> > kindly shoot some holes in it ;-)
> >
> > Notes: I've used a hypothetical feature for collecting information
> > about vendor packages -- presumably the apple domain also needs an
> > ephemeral vdb to hold it (like package.provided). I don't know if this
> > vendor domain needs a repo. I guess the main repo would have to mask
> > broken/testing vendor deps as it sees fit, including those from other
> > domains...
> Offhand, what you're proposing (querying the vendor installed pkg db) is
> actually a readonly vdb, if it were implemented. It's an installed
> package database you can query (use for solving deps), but cannot
> modify.
>
> > [vendor-apple] type = config FEATURES = "query-apple-packages"
> > ACCEPT_KEYWORDS = "ppc-darwin"
> Definitions not totally right as mentioned (doesn't matter, just being
> nitpicky :), but the crux of my concern is embodied inthe
> FEATURES="query-apple-packages".
>
> Mainly, how? How to query the vendor db, and map that back into portage
> package namespace?

Did I read something about the rewrite being modular? Could the shim/query
take the form of a portage plugin that implements the query-apple-packages
feature? Obviously, if implemented the way I descibed above, it would need
to be intimate with certain ebuilds' environments.

-f

>
> ~harring
>
--
gentoo-osx@gentoo.org mailing list

On Wed, Aug 31, 2005 at 01:32:04AM +1000, Finn Thain wrote:
> > > > Reasoning is, how do you know that pkg xyz is actually the package
> > > > you're after?
> > Re-inserted the quote to clarify what I'm talking about; mapping another
> > pkg managers db into our own requires either a lot of human
> > intervention, or some dodgy rules that somewhat manage it, with bugs.
>
> OK, I see what you mean. You're asking, how does portage know that vendor
> package xyz is the portage package abc?
>
> Short answer is package.mask, meta-packages and name mapping.
>
> A particular vendor package version is a known-good dep, as tested by
> devs, otherwise it is masked. E.g. package.mask says
> >vendor-sun/app-arch/cpio-x.y.z if no higher version has been tested. In
> mac os, automated updates mean that most of the time, there will be some
> vendor packages that the tree hasn't been tested against. These have to be
> masked until the user does emerge sync.

Alright, so I'm just being a tool 'coz I thought you were talking
about dynamic mapping (vs dev managed mappings). Nevermind me :)

> BTW, do repos share a namespace? Presented with the same cpv in several
> repos, is portage's behaviour defined yet?
repo's have their own *total* namespace now; an overlay + repo is
different though since an overlay is slaved to a repo.

<=2.1 basically lacks any true support for N repos; you can have a
portdir(+overlays), a vdb, and a bintree. Rewrite has no such
restriction built into it.

> My feeling is that the burden of managing the mappings is better than the
> burden of managing one package.provided for mac os 10.3, alongside another
> for 10.4, etc. (If I'm wrong about that, then this exercise is pointless.)
Actually, I agree; it's cleaner then just autoassuming stuff is there.

> Did I read something about the rewrite being modular? Could the shim/query
> take the form of a portage plugin that implements the query-apple-packages
> feature? Obviously, if implemented the way I descibed above, it would need
> to be intimate with certain ebuilds' environments.

Well, considering I'm seriously considering when/if rewrite is
released, it's released as two packages; portage-core, and
portage-ebuild... yes. Very modular.

There pretty much is one point of required entry into the code;
getting the config obj- from there it loads the code it needs,
instantiating objects on the fly. Aside from the entry point/config
obj, everything else is intended to be configurable.

~harring

On Tue, 30 Aug 2005, Brian Harring wrote:

> On Wed, Aug 31, 2005 at 01:32:04AM +1000, Finn Thain wrote:
>
> > My feeling is that the burden of managing the mappings is better than
> > the burden of managing one package.provided for mac os 10.3, alongside
> > another for 10.4, etc. (If I'm wrong about that, then this exercise is
> > pointless.)
> Actually, I agree; it's cleaner then just autoassuming stuff is there.

Thing is, if you undertake to track another package manager in this way,
you must expect the worst from the vendor. So the question is, just how
robust are such mappings are in the face of upstream mayhem?

Here are some scenarios, where I think the present package.provided method
seems to work better,

1. Vendor changes package manager API.
- user will have no vendor deps until devs update the shim

2. Vendor renames a package.
- user will lose a vendor dep until devs add a new mapping

3. Vendor combines two packages.
- see 2.

4. Vendor bumps a package version.
- see 2.

And here are some scenarios where the proposed method of mapping & masking
is better:

5. Vendor splits a package.
- existing method loses if any part is no longer installed by default
- new method works, given vendor bumps package version

6. Vendor drops a package entirely or renders it broken as a dep.
- existing method loses. e.g. if this happened in mac os 10.3.9, all
10.3 users would have suffered
- new method works, given vendor bumps package version

7. Vendor fixes a previously broken package
- existing method loses. e.g. if this happened in mac os 10.3.9, no
10.3 users benefit, because many still do not have the fix
- new method works, given vendor bumps package version

8. Vendor adds a package already in the tree.
- old method loses if the two packages are not equivalent
- new method wins if a virtual/metapackage can be used to mean "close
enough"

9. Vendor adds a package not already in the tree.
- old method loses because deprecated profiles (e.g. mac os 10.2) will
never get the benefit of that package
- new method wins because a new mapping is all that is required

As I described it, a mapping is a full mapping from vendor PV to portage
PV. So a new mapping is needed when the vendor revs the package (point 4
above).

Now, if we could also have an ebuild without a version number, like
/opt/gentoo/usr/portage/vendor-apple/sys-devel/xcode/xcode.ebuild, it
could just map a vendor name to a portage name. The synthetic vendor
package that shows up in the vdb would get the vendor's full version. In
this case, the devs' work is lessened because a new mapping wouldn't have
to be added every time the vendor revs a package. Instead, package.mask
would provide QA.

-f
--
gentoo-osx@gentoo.org mailing list

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


On Aug 30, 2005, at 11:01 AM, Kito wrote:

> Maybe so, but we can't have one without the other. First get
> packages to install in a prefix, then work up from there.

Amen. If we start really big here, we're just going to end up talking
about this for months and never get anything done. Perhaps we could
agree to work on getting packages to install in a prefix firstly and
develop a procedure for making that happen. I see the rest of the
issues in this thread as separate, even though they are related.

> The issue of leveraging existing packages is currently handled by
> package.provided, which we all know doesn't really serve our
> purposes, but I see no reason work couldn't be done in parallel on
> these issues...

Exactly. Since the solution to this problem seems to be the crux of
the discussion here, perhaps we could split this thread and continue
the discussion about alternatives to package.provided in a new thread
while working out a procedure for installing to a prefix simultaneously.

- --Lina Pezzella
Ebuild & Porting Co-Lead
Gentoo for OS X

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (Darwin)

iD8DBQFDGejjNJ9STR9DbYERAsFlAJwJHdtzCo6BRFIZGfWkkjv8gT2t1wCbBWcX
/k91HnML1Hw66Ijj6itlmsU=
=p6es
-----END PGP SIGNATURE-----
--
gentoo-osx@gentoo.org mailing list