371 replies to this topic

[Beeker]

And which branch can we use for python 3, hardknott, or new branch  ?

 

About python 2, drop it? Or both py2 and py3 ?

[WanWizard]

Oh I just meant that OpenPLi is interested in PY3.

 
Yes, we are !
 

Regarding py, pyc and pyo files.
With PY3 you need the py files anyway.
That will increase the image size.
When you don't ship the pyc files to the image, the image size should be the same as with python 2.
Not tried by myself.

 

I don't see why you "need" the py files, if you have the pyc's. And if you ship the py's,

 

Python will make the pyc's as soon as the files are loaded, which means:

• sort of double the space requirement, a real no for boxes with limited flash
• the pyc's will not be under version control, so when you remove the plugin, the pyc's remain (unless you start hacking with post-rm's, which is imho a bad idea)

So we want the same setup as we currently have: a "plugin.ipk" containing pyc's, and a "plugin-src.ipk" containing the py files.

[WanWizard]

And which branch can we use for python 3, hardknott, or new branch  ?

 

OE you mean? A new branch created from develop.
 

I can setup an automated build from that branch for a limited amount of boxes (for speed reasons) if needed.

[WanWizard]

We've been here before. Everyone else is using *.py files so having something different in plugins is going to cause problems.

 

So we're stuck because "everyone else" (read, OE-A), is doing it wrong?

[Beeker]

You can't import from pyc files as usual.

It seems possible with importlib ( imp.load_source python 3.3- )

 

 

https://docs.python..../importlib.html

 

To import module without .py file.

No experience with it,

Edited by Beeker, 13 December 2021 - 19:58.

[Beeker]

 

And which branch can we use for python 3, hardknott, or new branch  ?

 

OE you mean? A new branch created from develop.
 

I can setup an automated build from that branch for a limited amount of boxes (for speed reasons) if needed.

 

Yes. A new branch from develop for some boxes (for me Ustym 4K pro would be useful)

[Huevos]

 

We've been here before. Everyone else is using *.py files so having something different in plugins is going to cause problems.

 

So we're stuck because "everyone else" (read, OE-A), is doing it wrong?

 

Yep, everyone else is doing it the "wrong" way. Only OpenPLi wants to do it the "right" way.

 

Which, by the way, is a deprecated configuration.

Edited by Huevos, 13 December 2021 - 21:07.

[WanWizard]

Wat is deprecated? The use of pyc's over py's for embedded devices (and other platforms where no source is needed) is well documented.

But all we want is a solution that deals with the two bullet points in post #62. Alternatively we deploy py's and use PYTHONDONTWRITEBYTECODE=1 to avoid pyc's being created, that's fine by me.

 

We could even decide to do that only for hardware with limited flash (which is not an immediate thing to implement).

[Huevos]

Not using *.py and PyCache is deprecated. No idea why you would want to avoid the creation  of the pyc cache. Maybe it is time to stop building out of date hardware.

 

Also, wouldn't it make sense to move to Honister?

[WanWizard]

Just did some "hello world: tests.

 

If the pyc is in the correct directory ( = the directory where the py is ), the import works fine in Python3. The problem is the Python compiler stores them in __pycache__, where the imp loader can't fimd them

 

One solution would be to use pyhton2 to compile to pyc's, but that requires backwards compatble code, so probably not the way forward.
 

Another solution is to use  PYTHONPYCACHEPREFIX in the bb recipe to make sure the pyc's are compiled into the source directory (Python2-style) instead of using __pycache__, and then import works and source can be split from pyc's,

 

There are probably other solutions...

[WanWizard]

Not using *.py and PyCache is deprecated. No idea why you would want to avoid the creation  of the pyc cache. Maybe it is time to stop building out of date hardware.

 

I doubt that, why would Python 3.8 introduce  PYTHONPYCACHEPREFIX to set the location of the pyc file(s), if pycache is deprecated? I find no reference to this deprecation, do you have a docs link?

[WanWizard]

Also, wouldn't it make sense to move to Honister?

 

Have a chat with @rantanplan, he;s doing that bit.

 

I vagely remember the plan was to do hartknott first, then check if there is a need to release an OpenPLi 9 (for whatever reason), and if not, migratie hartknott to honister.

[Huevos]

 

Not using *.py and PyCache is deprecated. No idea why you would want to avoid the creation  of the pyc cache. Maybe it is time to stop building out of date hardware.

 

I doubt that, why would Python 3.8 introduce  PYTHONPYCACHEPREFIX to set the location of the pyc file(s), if pycache is deprecated? I find no reference to this deprecation, do you have a docs link?

 

You misread my post. Using *.py and PyCache is current. Not using *.py and PyCache is deprecated. i.e. using *.pyc that are not in the cache folder.

[WanWizard]

And I asked you for references, as I disagree.

 

If __pycache__ was the required way to go (when deprecated goes to unsupported), why introduce  PYTHONPYCACHEPREFIX in 3.8, which is exactly to avoid having to use __pycache__ ?

 

Clearly, if you want to use pyc's NOT as a runtime cache, but as deployment files, they HAVE to be where import expects them.

[Huevos]

BTW, I am surprised you are building so many boxes. Are people still using Vu+ Solo original? You must have unlimited resources.

[Huevos]

So going back to the original point...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.py"):

Is not going to work on your vision of PLi... and...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.pyc"):

Is not going to work in any other distro...

 

So whoever is converting shared plugins needs to take that into consideration.

[WanWizard]

From what I read, the main argument for using __pycache__ is the ability to use different python interpreter versions. Which is not relevant for us, we know exactly which version is installed.

[Huevos]

And I asked you for references, as I disagree.

 

If __pycache__ was the required way to go (when deprecated goes to unsupported), why introduce  PYTHONPYCACHEPREFIX in 3.8, which is exactly to avoid having to use __pycache__ ?

 

Clearly, if you want to use pyc's NOT as a runtime cache, but as deployment files, they HAVE to be where import expects them.

PYTHONPYCACHEPREFIX environment variable is used so  the PyCache can be a separate parallel filesystem tree, rather than the default __pycache__ subdirectories within each source directory. It is nothing to do with having *.pyc files directly in the source folders.

Edited by Huevos, 13 December 2021 - 22:34.

[WanWizard]

BTW, I am surprised you are building so many boxes. Are people still using Vu+ Solo original? You must have unlimited resources.

 

 

You would be suprised   As long as we can build, we will, but there will come a time when we won't be able to anymore. But if it is up to me, not because we start deploying a python cache.

 

And no, not unlimited, but for now we've got plenty capacity, Even with develop in a continuous build we're not hitting 40% of the capacity (assuming nobody commits a glibc change that triggers a full rebuild ).

[WanWizard]

PYTHONPYCACHEPREFIX environment variable is used so  the PyCache can be a separate parallel filesystem tree, rather than the default __pycache__ subdirectories within each source directory. It is nothing to do with having *.pyc files directly in the source folders.

 

It can be (and is suggested to) used for that, as we compile every py file individually in the bb recipe, so we can set it for every py individually, equal to the path of the py.