Mailing List Archive

Unification of the Mac builds?
Sorry if I'm out of the loop here, but with Apple's new chip coming out, we
need new a build configuration (which I think has already been started, if
not done).

Perhaps we could take this opportunity to better modularize / unify the
build setup?

As it was last I checked, you really had only two options:

1) Ignore anything mac specific, and get a "unix" build.

2) Get a full "Framework" build, optionally with Universal support.

It would be nice to keep the Framework structure independent of the
Mac-specific features, if possible.

In particular, I'd love to get be able to get the "pythonw" executable
wrapper in an otherwise standard unix build [*].

It would also be nice if it were possible to get universal binaries in a
"unix style" build.

(option 3 would be to simply abandon the Framework Build altogether -- it's
still not clear to me what this really buys mac users)

Any chance of this happening? I'm afraid I know nothing of autoconf, so
can't be much help, but I'd be willing to help out with testing, or
documenting, or anything else that I have the skills to do.

Thanks,

-Chris

[*] The pythonw issue has been a thorn in the side of conda for years.
conda uses a standard unix build on the Mac, for consistency with other
unix systems. But there is no configuration set up to build the pythonw
wrapper outside of a "Framework" build. So instead, conda has creates its
own "pythonw" wrapper -- but that is a bash script that re-directs to a
different executable. This works fine on the command line (or #! line), but
it does not work with setuptools' entry_points. And the setuptools
community hasn't shown any interest in hacking around it. Anyway, few
enough people develop Desktop apps (particularly with conda) so this has
lingered, but it would be nice to fix.









--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov
Re: Unification of the Mac builds? [ In reply to ]
On Jan 8, 2021, at 14:38, Chris Barker via Python-Dev <python-dev@python.org> wrote:
> Sorry if I'm out of the loop here, but with Apple's new chip coming out, we need new a build configuration (which I think has already been started, if not done).
>
> Perhaps we could take this opportunity to better modularize / unify the build setup?
[...]

Chris:

Since the topics you are touching on are fairly arcane macOS-only related issues and some of the issues you bring up will require clarification, I suggest you bring this up on the Python Mac sig mailing list (Pythonmac-SIG@python.org) and let's discuss it there rather than here in python-dev. We can always summarize any recommendations back here if necessary.

Thanks!
--Ned

--
Ned Deily
nad@python.org -- []
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/H3RUV7SOCQ2QSWG3DTAMXDEQEBJQZRME/
Code of Conduct: http://python.org/psf/codeofconduct/
Re: Unification of the Mac builds? [ In reply to ]
> On 8 Jan 2021, at 20:38, Chris Barker via Python-Dev <python-dev@python.org> wrote:
>
> Sorry if I'm out of the loop here, but with Apple's new chip coming out, we need new a build configuration (which I think has already been started, if not done).
>
> Perhaps we could take this opportunity to better modularize / unify the build setup?
>
> As it was last I checked, you really had only two options:
>
> 1) Ignore anything mac specific, and get a "unix" build.
>
> 2) Get a full "Framework" build, optionally with Universal support.
>
> It would be nice to keep the Framework structure independent of the Mac-specific features, if possible.
>
> In particular, I'd love to get be able to get the "pythonw" executable wrapper in an otherwise standard unix build [*].

That’s a feature of the framework build. The unix build is exactly the same as a unix build on other platform. Adding the same feature to the unix build should be possible, but would complicate the build. I have no interest to work on this, but would be willing to review PRs, as long as those aim for feature parity with the framework build. That is, both pythonw(1) and python(1) should redirect through an embedded app bundle.

>
> It would also be nice if it were possible to get universal binaries in a "unix style" build.

Let me sneak away in Guido’s time machine for a while.



Done, just configure “—enable-universalsdk —with-universal-archs=universal2” without specifying a framework build.

>
> (option 3 would be to simply abandon the Framework Build altogether -- it's still not clear to me what this really buys mac users)

In some ways this is historic, but frameworks are still the way to build a self-contained library on macOS. A major thing this buys us is having side-by-side installs of Python.

>
> Any chance of this happening? I'm afraid I know nothing of autoconf, so can't be much help, but I'd be willing to help out with testing, or documenting, or anything else that I have the skills to do.

That’s not really something I intend to work on.

Something I would like to work on, but don’t have enough free time for, is an alternative installation with an application bundle on the top level instead of a framework. Installation would then entail dropping “Python X.Y.app” into your application folder, and uninstallation would be to drop the same bundle into the bin. This might also make it possible to distribute Python through the Mac App Store, although I haven’t checked recently if sandboxing requirements have been relaxed enough to make that worthwhile.

>
> Thanks,
>
> -Chris
>
> [*] The pythonw issue has been a thorn in the side of conda for years. conda uses a standard unix build on the Mac, for consistency with other unix systems. But there is no configuration set up to build the pythonw wrapper outside of a "Framework" build. So instead, conda has creates its own "pythonw" wrapper -- but that is a bash script that re-directs to a different executable. This works fine on the command line (or #! line), but it does not work with setuptools' entry_points. And the setuptools community hasn't shown any interest in hacking around it. Anyway, few enough people develop Desktop apps (particularly with conda) so this has lingered, but it would be nice to fix.

Just don’t use conda ;-). To be blunt, doing this properly is trivial.

Ronald



Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/
Re: Unification of the Mac builds? [ In reply to ]
Ned,

Thanks -- I'll take further discussion to the python-mac list.

Ronald:

That’s a feature of the framework build. The unix build is exactly the same
> as a unix build on other platform. Adding the same feature to the unix
> build should be possible, but would complicate the build. I have no
> interest to work on this, but would be willing to review PRs,
>

fair enough.


> as long as those aim for feature parity with the framework build. That is,
> both pythonw(1) and python(1) should redirect through an embedded app
> bundle.
>

yes, that's what I'm thinking.

Done, just configure “—enable-universalsdk
> —with-universal-archs=universal2” without specifying a framework build.
>

Excellent, thanks!

Something I would like to work on, but don’t have enough free time for, is
> an alternative installation with an application bundle on the top level
> instead of a framework. Installation would then entail dropping “Python
> X.Y.app” into your application folder, and uninstallation would be to drop
> the same bundle into the bin.
>

That would be nice -- and would, I think, actually buy us something useful
from this bundling :-)

Just don’t use conda ;-). To be blunt, doing this properly is trivial.
>

which "this" are you referring to -- and if it's trivial, then why hasn't
anyone figured out how in multiple conversations over years?

But if it is -- tell us how and maybe we will do it (if it's really
trivial, maybe even I, with my limited build skills could do it)

Thanks,

-CHB

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov
Re: Unification of the Mac builds? [ In reply to ]
> On 14 Jan 2021, at 23:03, Chris Barker via Python-Dev <python-dev@python.org> wrote:
>
> Ned,
>
> Thanks -- I'll take further discussion to the python-mac list.
>
> Ronald:
>
> That’s a feature of the framework build. The unix build is exactly the same as a unix build on other platform. Adding the same feature to the unix build should be possible, but would complicate the build. I have no interest to work on this, but would be willing to review PRs,
>
> fair enough.
>
> as long as those aim for feature parity with the framework build. That is, both pythonw(1) and python(1) should redirect through an embedded app bundle.
>
> yes, that's what I'm thinking.
>
> Done, just configure “—enable-universalsdk —with-universal-archs=universal2” without specifying a framework build.
>
> Excellent, thanks!
>
> Something I would like to work on, but don’t have enough free time for, is an alternative installation with an application bundle on the top level instead of a framework. Installation would then entail dropping “Python X.Y.app” into your application folder, and uninstallation would be to drop the same bundle into the bin.
>
> That would be nice -- and would, I think, actually buy us something useful from this bundling :-)

To be honest I’d expect that this will also lead to complaining about how this build is not a unix build ;-)

>
> Just don’t use conda ;-). To be blunt, doing this properly is trivial.
>
> which "this" are you referring to -- and if it's trivial, then why hasn't anyone figured out how in multiple conversations over years?
“This” is having a pythonw that’s the same as in framework builds.

I don’t know. I do know that nobody asked questions about this before.

>
> But if it is -- tell us how and maybe we will do it (if it's really trivial, maybe even I, with my limited build skills could do it)


For anyone reading along that isn’t familiar with macOS: On macOS a number of system libraries, and in particular GUI libraries, require that they are used from an application bundle (‘.app”). For Python we want to be able to just use the command-line for simple scripts (and testing of more complicated applications), instead having to build an app bundle every time.

In framework builds this is solved by two changes relative to a unix build:
1. The regular python binary is in “{sys.prefix}/Resources/Python.app/Contents/MacOS/Python”
2. “{sys.executable}” (or “{sys.prefix}/bin/python3”) is a stub executable that does nothing but execv[1] the regular python binary. The source of this binary is in Mac/Tools/pythonw.c


Note: All of this requires using a shared library build (—enable-shared) because of the way the stub finds sys.prefix.

To add this to the unix build as well requires:
1. Make up a location for the Python.app
2. Change the Makefile(.pre.in) to build the stub executable install files in their new location, but only when installing on macOS with an —enable-shared build
3. Tweak the code in pythonw.c to calculate the correct path to Python.app
4. Possibly tweak the calculation of sys.prefix for unix builds, IIRC the special macOS sauce is enabled for framework builds only
5. Do some debugging because this list is likely incomplete, I typed it from memory


Ronald

[1] for the pedantically correct: it actually uses posix_spawn in exec mode to ensure that the correct CPU architecture is used. That’s why “arch -x86_64 python3.9” works when you use the universal2 installer for python 3.9, if we’d use execv the stub would launch the actual interpreter in native mode.

Ronald

>
> Thanks,
>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris.Barker@noaa.gov <mailto:Chris.Barker@noaa.gov>_______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7YZ4JB5PTL2LKGRCNZF7X4TP7ZYTBW3H/
> Code of Conduct: http://python.org/psf/codeofconduct/



Twitter / micro.blog: @ronaldoussoren
Blog: https://blog.ronaldoussoren.net/
Re: Unification of the Mac builds? [ In reply to ]
Thanks Ronald,

This is really helpful.

-CHB

On Fri, Jan 15, 2021 at 5:43 AM Ronald Oussoren <ronaldoussoren@mac.com>
wrote:

>
>
> On 14 Jan 2021, at 23:03, Chris Barker via Python-Dev <
> python-dev@python.org> wrote:
>
> Ned,
>
> Thanks -- I'll take further discussion to the python-mac list.
>
> Ronald:
>
> That’s a feature of the framework build. The unix build is exactly the
>> same as a unix build on other platform. Adding the same feature to the
>> unix build should be possible, but would complicate the build. I have no
>> interest to work on this, but would be willing to review PRs,
>>
>
> fair enough.
>
>
>> as long as those aim for feature parity with the framework build. That
>> is, both pythonw(1) and python(1) should redirect through an embedded app
>> bundle.
>>
>
> yes, that's what I'm thinking.
>
> Done, just configure “—enable-universalsdk
>> —with-universal-archs=universal2” without specifying a framework build.
>>
>
> Excellent, thanks!
>
> Something I would like to work on, but don’t have enough free time for, is
>> an alternative installation with an application bundle on the top level
>> instead of a framework. Installation would then entail dropping “Python
>> X.Y.app” into your application folder, and uninstallation would be to drop
>> the same bundle into the bin.
>>
>
> That would be nice -- and would, I think, actually buy us something useful
> from this bundling :-)
>
>
> To be honest I’d expect that this will also lead to complaining about how
> this build is not a unix build ;-)
>
>
> Just don’t use conda ;-). To be blunt, doing this properly is trivial.
>>
>
> which "this" are you referring to -- and if it's trivial, then why hasn't
> anyone figured out how in multiple conversations over years?
>
> “This” is having a pythonw that’s the same as in framework builds.
>
> I don’t know. I do know that nobody asked questions about this before.
>
>
> But if it is -- tell us how and maybe we will do it (if it's really
> trivial, maybe even I, with my limited build skills could do it)
>
>
>
> For anyone reading along that isn’t familiar with macOS: On macOS a number
> of system libraries, and in particular GUI libraries, require that they are
> used from an application bundle (‘.app”). For Python we want to be able
> to just use the command-line for simple scripts (and testing of more
> complicated applications), instead having to build an app bundle every time.
>
> In framework builds this is solved by two changes relative to a unix build:
> 1. The regular python binary is in
> “{sys.prefix}/Resources/Python.app/Contents/MacOS/Python”
> 2. “{sys.executable}” (or “{sys.prefix}/bin/python3”) is a stub executable
> that does nothing but execv[1] the regular python binary. The source of
> this binary is in Mac/Tools/pythonw.c
>
>
> Note: All of this requires using a shared library build (—enable-shared)
> because of the way the stub finds sys.prefix.
>
> To add this to the unix build as well requires:
> 1. Make up a location for the Python.app
> 2. Change the Makefile(.pre.in) to build the stub executable install
> files in their new location, but only when installing on macOS with an
> —enable-shared build
> 3. Tweak the code in pythonw.c to calculate the correct path to Python.app
> 4. Possibly tweak the calculation of sys.prefix for unix builds, IIRC the
> special macOS sauce is enabled for framework builds only
> 5. Do some debugging because this list is likely incomplete, I typed it
> from memory
>
>
> Ronald
>
> [1] for the pedantically correct: it actually uses posix_spawn in exec
> mode to ensure that the correct CPU architecture is used. That’s why “arch
> -x86_64 python3.9” works when you use the universal2 installer for python
> 3.9, if we’d use execv the stub would launch the actual interpreter in
> native mode.
>
> Ronald
>
>
> Thanks,
>
> -CHB
>
> --
>
> Christopher Barker, Ph.D.
> Oceanographer
>
> Emergency Response Division
> NOAA/NOS/OR&R (206) 526-6959 voice
> 7600 Sand Point Way NE (206) 526-6329 fax
> Seattle, WA 98115 (206) 526-6317 main reception
>
> Chris.Barker@noaa.gov
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/7YZ4JB5PTL2LKGRCNZF7X4TP7ZYTBW3H/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
>
> —
>
> Twitter / micro.blog: @ronaldoussoren
> Blog: https://blog.ronaldoussoren.net/
>
>

--

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R (206) 526-6959 voice
7600 Sand Point Way NE (206) 526-6329 fax
Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov