Mailing List Archive

Have you considered starting with micropython? It’s made for embedded
systems and fully supports Python 3 syntax. Adding sqlite3 support to it
will be less work than stripping all the I/O from CPython.

—Guido

On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com> wrote:

>
> Hi, all
>
> There is a request to run python in a Linux-based embedded resource
> constrained system with sqlite3 support.
>
> So many features are not required, like posixmodule, signalmodule,
> hashtable ...
> But seems there are some dependencies among the
> Modules/Parser/Python/Objects/Programs...
>
> Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less
> syscalls the better) ?
> Is it possible to do that?
>
> Thank you.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
--
--Guido (mobile)

Hi, Guido

Yes. Micropyhton is also in consideration.
But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?

Or is there a feasible way to strip CPython ?

Thanks.

From: Guido van Rossum <guido@python.org>
Sent: Monday, July 20, 2020 10:45 PM
To: Huang, Yang <yang.huang@intel.com>
Cc: python-dev@python.org
Subject: Re: [Python-Dev] How to customize CPython to a minimal set

Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython.

—Guido

On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>> wrote:

Hi, all

There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.

So many features are not required, like posixmodule, signalmodule, hashtable ...
But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...

Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ?
Is it possible to do that?

Thank you.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-leave@python.org<mailto:python-dev-leave@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/
Code of Conduct: http://python.org/psf/codeofconduct/
--
--Guido (mobile)

I expect it will be unfeasible to strip CPython. If you disagree, try it.
;-)

On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com> wrote:

> Hi, Guido
>
>
>
> Yes. Micropyhton is also in consideration.
>
> But sqlite3 is the first usage. There should be some additional features
> like numpy, scipy... Not sure if micropython supports well?
>
>
>
> Or is there a feasible way to strip CPython ?
>
>
>
> Thanks.
>
>
>
> *From:* Guido van Rossum <guido@python.org>
> *Sent:* Monday, July 20, 2020 10:45 PM
> *To:* Huang, Yang <yang.huang@intel.com>
> *Cc:* python-dev@python.org
> *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
>
>
>
> Have you considered starting with micropython? It’s made for embedded
> systems and fully supports Python 3 syntax. Adding sqlite3 support to it
> will be less work than stripping all the I/O from CPython.
>
>
>
> —Guido
>
>
>
> On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com> wrote:
>
>
> Hi, all
>
> There is a request to run python in a Linux-based embedded resource
> constrained system with sqlite3 support.
>
> So many features are not required, like posixmodule, signalmodule,
> hashtable ...
> But seems there are some dependencies among the
> Modules/Parser/Python/Objects/Programs...
>
> Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less
> syscalls the better) ?
> Is it possible to do that?
>
> Thank you.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> --Guido (mobile)
>
--
--Guido (mobile)

On 2020-07-21 06:33, Huang, Yang wrote:
> Hi, Guido
>
> Yes. Micropyhton is also in consideration.
>
> But sqlite3 is the first usage. There should be some additional features
> like numpy, scipy... Not sure if micropython supports well?
>
> Or is there a feasible way to strip CPython ?
>
> Thanks.

You want a minimal Python, yet with numpy, scipy, ...? Those sound like
contradictory goals to me!

>
> *From:* Guido van Rossum <guido@python.org>
> *Sent:* Monday, July 20, 2020 10:45 PM
> *To:* Huang, Yang <yang.huang@intel.com>
> *Cc:* python-dev@python.org
> *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
>
> Have you considered starting with micropython? It’s made for embedded
> systems and fully supports Python 3 syntax. Adding sqlite3 support to it
> will be less work than stripping all the I/O from CPython.
>
> —Guido
>
> On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com
> <mailto:yang.huang@intel.com>> wrote:
>
>
> Hi, all
>
> There is a request to run python in a Linux-based embedded resource
> constrained system with sqlite3 support.
>
> So many features are not required, like posixmodule, signalmodule,
> hashtable ...
> But seems there are some dependencies among the
> Modules/Parser/Python/Objects/Programs...
>
> Is there a way to tailor CPython 3 to a minimal set with sqlite3
> (the less syscalls the better) ?
> Is it possible to do that?
>
> Thank you.
>
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/GITUFJA4FQNVBTD2MMGV46JDGPDPLKOJ/
Code of Conduct: http://python.org/psf/codeofconduct/

On 21Jul2020 0633, Huang, Yang wrote:
> Yes. Micropyhton is also in consideration.
>
> But sqlite3 is the first usage. There should be some additional features
> like numpy, scipy... Not sure if micropython supports well?
>
> Or is there a feasible way to strip CPython ?

Only by manually removing modules from your own build. However, if you
do that but want to continue using third-party packages, you'll find
that there is very little you can remove (and nothing of significance).

The most interesting modules to omit are ssl and ctypes, because of the
exposure to likely security issues (via OpenSSL and libffi), but
practically every module out there will have some sort of dependency on
these. So you'll end up only removing small/niche modules that might
save you a couple of kilobytes of Python code.

Of the modules you suggested initially, I think you'll find that you
need all of them for SQLite, let alone more complex libraries such as numpy.

In theory, we could have a smaller set of modules, but in practice there
are so many cross-dependencies already out there that there isn't much
to gain right now. It would take a bit of an ecosystem "reset" (i.e.
develop completely new popular libraries with more narrowly scoped
dependencies) to narrow things down at this point, and there's not much
chance of that happening.

Cheers,
Steve
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/TNFG5E6MRLZV6VAYNKZKWOMSBY2WE6UH/
Code of Conduct: http://python.org/psf/codeofconduct/

Thank you for all your comments.
I cannot agree any more. I did try but there were so many dependencies. One change all change.

Micropython is a choice. But not sure if numpy and sqlite3 can be supported well. And what’s the compatibility of the libs in Pypi.


From: Guido van Rossum <guido@python.org>
Sent: Tuesday, July 21, 2020 10:57 PM
To: Huang, Yang <yang.huang@intel.com>
Cc: python-dev@python.org
Subject: Re: [Python-Dev] How to customize CPython to a minimal set

I expect it will be unfeasible to strip CPython. If you disagree, try it. ;-)

On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>> wrote:
Hi, Guido

Yes. Micropyhton is also in consideration.
But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?

Or is there a feasible way to strip CPython ?

Thanks.

From: Guido van Rossum <guido@python.org<mailto:guido@python.org>>
Sent: Monday, July 20, 2020 10:45 PM
To: Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>>
Cc: python-dev@python.org<mailto:python-dev@python.org>
Subject: Re: [Python-Dev] How to customize CPython to a minimal set

Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython.

—Guido

On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com<mailto:yang.huang@intel.com>> wrote:

Hi, all

There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.

So many features are not required, like posixmodule, signalmodule, hashtable ...
But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...

Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ?
Is it possible to do that?

Thank you.
_______________________________________________
Python-Dev mailing list -- python-dev@python.org<mailto:python-dev@python.org>
To unsubscribe send an email to python-dev-leave@python.org<mailto:python-dev-leave@python.org>
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/
Code of Conduct: http://python.org/psf/codeofconduct/
--
--Guido (mobile)
--
--Guido (mobile)

Sadly micropython is not intended to support numerical libraries and other
such complex modules: the support for the Python standard library is pretty
much non-existent.

Kind regards,
Steve


On Wed, Jul 22, 2020 at 3:10 PM Huang, Yang <yang.huang@intel.com> wrote:

> Thank you for all your comments.
>
> I cannot agree any more. I did try but there were so many dependencies.
> One change all change.
>
>
>
> Micropython is a choice. But not sure if numpy and sqlite3 can be
> supported well. And what’s the compatibility of the libs in Pypi.
>
>
>
>
>
> *From:* Guido van Rossum <guido@python.org>
> *Sent:* Tuesday, July 21, 2020 10:57 PM
> *To:* Huang, Yang <yang.huang@intel.com>
> *Cc:* python-dev@python.org
> *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
>
>
>
> I expect it will be unfeasible to strip CPython. If you disagree, try it.
> ;-)
>
>
>
> On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com> wrote:
>
> Hi, Guido
>
>
>
> Yes. Micropyhton is also in consideration.
>
> But sqlite3 is the first usage. There should be some additional features
> like numpy, scipy... Not sure if micropython supports well?
>
>
>
> Or is there a feasible way to strip CPython ?
>
>
>
> Thanks.
>
>
>
> *From:* Guido van Rossum <guido@python.org>
> *Sent:* Monday, July 20, 2020 10:45 PM
> *To:* Huang, Yang <yang.huang@intel.com>
> *Cc:* python-dev@python.org
> *Subject:* Re: [Python-Dev] How to customize CPython to a minimal set
>
>
>
> Have you considered starting with micropython? It’s made for embedded
> systems and fully supports Python 3 syntax. Adding sqlite3 support to it
> will be less work than stripping all the I/O from CPython.
>
>
>
> —Guido
>
>
>
> On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com> wrote:
>
>
> Hi, all
>
> There is a request to run python in a Linux-based embedded resource
> constrained system with sqlite3 support.
>
> So many features are not required, like posixmodule, signalmodule,
> hashtable ...
> But seems there are some dependencies among the
> Modules/Parser/Python/Objects/Programs...
>
> Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less
> syscalls the better) ?
> Is it possible to do that?
>
> Thank you.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/
> Code of Conduct: http://python.org/psf/codeofconduct/
>
> --
>
> --Guido (mobile)
>
> --
>
> --Guido (mobile)
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSNBFJOT3W2ES4OIOQ4LOWMNHHG/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

I don't know if they suit your purposes, but both sqlite3 and a small array
implementation are already available:

https://github.com/micropython/micropython-lib/tree/master/sqlite3
https://github.com/v923z/micropython-ulab

On 7/20/20 10:30 AM, Huang, Yang wrote:
>
> Hi, all
>
> There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
>
> So many features are not required, like posixmodule, signalmodule, hashtable ...
> But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
>
> Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ?
> Is it possible to do that?

CPython comes with the promise of all batteries included, however sometimes it
feels like a complete power plant. For packaging purposes, most Linux distros
make the decision to ship CPython as a set of runtime packages, and a set of
packages used for development (and building C extensions). Breaking that down
for the Debian/Ubuntu packages comes down to these sizes (unpacked,
uncompressed, on x86_64):

minimal:
5516K debian/libpython3.9-minimal
5856K debian/python3.9-minimal

stdlib:
8528K debian/libpython3.9-stdlib
624K debian/python3.9

development:
19468K debian/libpython3.9-dev
25804K debian/libpython3.9-testsuite
1232K debian/python3-distutils
668K debian/python3-lib2to3
548K debian/python3.9-dev

extra modules:
1648K debian/idle-python3.9
5208K debian/python3.9-examples

132K debian/python3-gdbm
844K debian/python3-tk

What you don't see from the sizes, are the extra dependencies which add to the
size, e.g. the X stack for tk, or readline/curses/crypto for stdlib.

The "minimal" set may sound nice, however it's not used in practice, because the
set is not well defined, and it's difficult to keep the minimal set as a
self-contained set of stdlib modules, and the minimal usually keeps growing,
never shrinks.

It looks like this "minimal" set is even too much for your purposed, so you
would have to scale down this set even further.

Matthias
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OZY63O24XP535PQYBAYNFXRF5S3PDV3E/
Code of Conduct: http://python.org/psf/codeofconduct/

There are tools in the python ecosystem that know how to build stand alone apps for
python. These tools can figure out the minimum set of python and .so/.dylib/.dll that are needed.

I use these tools when building apps for Windows and macOS where I do not require
that python is pre-installed. You can use the same technics for unix systems.

1. Start with standard python
2. pip install deps
3. run tool to create minimum app

If you build multiple apps then you can share common files to cut down size as well.

Barry


> On 22 Jul 2020, at 14:31, Huang, Yang <yang.huang@intel.com> wrote:
>
> Thank you for all your comments.
> I cannot agree any more. I did try but there were so many dependencies. One change all change.
>
> Micropython is a choice. But not sure if numpy and sqlite3 can be supported well. And what’s the compatibility of the libs in Pypi.
>
>
> From: Guido van Rossum <guido@python.org <mailto:guido@python.org>>
> Sent: Tuesday, July 21, 2020 10:57 PM
> To: Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>>
> Cc: python-dev@python.org <mailto:python-dev@python.org>
> Subject: Re: [Python-Dev] How to customize CPython to a minimal set
>
> I expect it will be unfeasible to strip CPython. If you disagree, try it. ;-)
>
> On Mon, Jul 20, 2020 at 22:35 Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>> wrote:
> Hi, Guido
>
> Yes. Micropyhton is also in consideration.
> But sqlite3 is the first usage. There should be some additional features like numpy, scipy... Not sure if micropython supports well?
>
> Or is there a feasible way to strip CPython ?
>
> Thanks.
>
> From: Guido van Rossum <guido@python.org <mailto:guido@python.org>>
> Sent: Monday, July 20, 2020 10:45 PM
> To: Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>>
> Cc: python-dev@python.org <mailto:python-dev@python.org>
> Subject: Re: [Python-Dev] How to customize CPython to a minimal set
>
> Have you considered starting with micropython? It’s made for embedded systems and fully supports Python 3 syntax. Adding sqlite3 support to it will be less work than stripping all the I/O from CPython.
>
> —Guido
>
> On Mon, Jul 20, 2020 at 06:48 Huang, Yang <yang.huang@intel.com <mailto:yang.huang@intel.com>> wrote:
>
> Hi, all
>
> There is a request to run python in a Linux-based embedded resource constrained system with sqlite3 support.
>
> So many features are not required, like posixmodule, signalmodule, hashtable ...
> But seems there are some dependencies among the Modules/Parser/Python/Objects/Programs...
>
> Is there a way to tailor CPython 3 to a minimal set with sqlite3 (the less syscalls the better) ?
> Is it possible to do that?
>
> Thank you.
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/ <https://mail.python.org/archives/list/python-dev@python.org/message/ECPLKXQ42VNLHD5DP3RG57L3QTJ77FUT/>
> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>
> --
> --Guido (mobile)
> --
> --Guido (mobile)
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org <mailto:python-dev@python.org>
> To unsubscribe send an email to python-dev-leave@python.org <mailto:python-dev-leave@python.org>
> https://mail.python.org/mailman3/lists/python-dev.python.org/ <https://mail.python.org/mailman3/lists/python-dev.python.org/>
> Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSNBFJOT3W2ES4OIOQ4LOWMNHHG/ <https://mail.python.org/archives/list/python-dev@python.org/message/OMUGASSNBFJOT3W2ES4OIOQ4LOWMNHHG/>
> Code of Conduct: http://python.org/psf/codeofconduct/ <http://python.org/psf/codeofconduct/>