Jump to content


Photo

develop: python3 transition


  • Please log in to reply
371 replies to this topic

Re: develop: python3 transition #61 Beeker

  • PLi® Contributor
  • 1,604 posts

+203
Excellent

Posted 13 December 2021 - 18:51

And which branch can we use for python 3, hardknott, or new branch  ?

 

About python 2, drop it? Or both py2 and py3 ?


Dreambox dm920, Uclan Ustym4Kpro, Gigablue UHD TRIO 4K and Dreambox dm8000. Wavefrontier T55 13.0|19.2|23.5|28.2 + Ziggo.


Re: develop: python3 transition #62 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 19:03

Oh I just meant that OpenPLi is interested in PY3.

 
Yes, we are !
 

Regarding py, pyc and pyo files.
With PY3 you need the py files anyway.
That will increase the image size.
When you don't ship the pyc files to the image, the image size should be the same as with python 2.
Not tried by myself.

 

I don't see why you "need" the py files, if you have the pyc's. And if you ship the py's,

 

Python will make the pyc's as soon as the files are loaded, which means:

  • sort of double the space requirement, a real no for boxes with limited flash
  • the pyc's will not be under version control, so when you remove the plugin, the pyc's remain (unless you start hacking with post-rm's, which is imho a bad idea)

So we want the same setup as we currently have: a "plugin.ipk" containing pyc's, and a "plugin-src.ipk" containing the py files.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #63 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 19:04

And which branch can we use for python 3, hardknott, or new branch  ?

 

OE you mean? A new branch created from develop.
 

I can setup an automated build from that branch for a limited amount of boxes (for speed reasons) if needed.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #64 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 19:05

We've been here before. Everyone else is using *.py files so having something different in plugins is going to cause problems.

 

So we're stuck because "everyone else" (read, OE-A), is doing it wrong?


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #65 Beeker

  • PLi® Contributor
  • 1,604 posts

+203
Excellent

Posted 13 December 2021 - 19:57

You can't import from pyc files as usual.

It seems possible with importlib ( imp.load_source python 3.3- )

 

 

https://docs.python..../importlib.html

 

To import module without .py file.

No experience with it,


Edited by Beeker, 13 December 2021 - 19:58.

Dreambox dm920, Uclan Ustym4Kpro, Gigablue UHD TRIO 4K and Dreambox dm8000. Wavefrontier T55 13.0|19.2|23.5|28.2 + Ziggo.


Re: develop: python3 transition #66 Beeker

  • PLi® Contributor
  • 1,604 posts

+203
Excellent

Posted 13 December 2021 - 20:04

 

And which branch can we use for python 3, hardknott, or new branch  ?

 

OE you mean? A new branch created from develop.
 

I can setup an automated build from that branch for a limited amount of boxes (for speed reasons) if needed.

 

Yes. A new branch from develop for some boxes (for me Ustym 4K pro would be useful)


Dreambox dm920, Uclan Ustym4Kpro, Gigablue UHD TRIO 4K and Dreambox dm8000. Wavefrontier T55 13.0|19.2|23.5|28.2 + Ziggo.


Re: develop: python3 transition #67 Huevos

  • PLi® Contributor
  • 4,663 posts

+163
Excellent

Posted 13 December 2021 - 21:02

 

We've been here before. Everyone else is using *.py files so having something different in plugins is going to cause problems.

 

So we're stuck because "everyone else" (read, OE-A), is doing it wrong?

 

Yep, everyone else is doing it the "wrong" way. Only OpenPLi wants to do it the "right" way.

 

Which, by the way, is a deprecated configuration.


Edited by Huevos, 13 December 2021 - 21:07.


Re: develop: python3 transition #68 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 21:54

Wat is deprecated? The use of pyc's over py's for embedded devices (and other platforms where no source is needed) is well documented.

But all we want is a solution that deals with the two bullet points in post #62. Alternatively we deploy py's and use PYTHONDONTWRITEBYTECODE=1 to avoid pyc's being created, that's fine by me.

 

We could even decide to do that only for hardware with limited flash (which is not an immediate thing to implement).


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #69 Huevos

  • PLi® Contributor
  • 4,663 posts

+163
Excellent

Posted 13 December 2021 - 22:06

Not using *.py and PyCache is deprecated. No idea why you would want to avoid the creation  of the pyc cache. Maybe it is time to stop building out of date hardware.

 

Also, wouldn't it make sense to move to Honister?



Re: develop: python3 transition #70 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 22:06

Just did some "hello world: tests.

 

If the pyc is in the correct directory ( = the directory where the py is ), the import works fine in Python3. The problem is the Python compiler stores them in __pycache__, where the imp loader can't fimd them

 

One solution would be to use pyhton2 to compile to pyc's, but that requires backwards compatble code, so probably not the way forward.
 

Another solution is to use  PYTHONPYCACHEPREFIX in the bb recipe to make sure the pyc's are compiled into the source directory (Python2-style) instead of using __pycache__, and then import works and source can be split from pyc's,

 

There are probably other solutions...


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #71 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 22:08

Not using *.py and PyCache is deprecated. No idea why you would want to avoid the creation  of the pyc cache. Maybe it is time to stop building out of date hardware.

 

I doubt that, why would Python 3.8 introduce  PYTHONPYCACHEPREFIX to set the location of the pyc file(s), if pycache is deprecated? I find no reference to this deprecation, do you have a docs link?


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #72 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 22:11

Also, wouldn't it make sense to move to Honister?

 

Have a chat with @rantanplan, he;s doing that bit.

 

I vagely remember the plan was to do hartknott first, then check if there is a need to release an OpenPLi 9 (for whatever reason), and if not, migratie hartknott to honister.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #73 Huevos

  • PLi® Contributor
  • 4,663 posts

+163
Excellent

Posted 13 December 2021 - 22:16

 

Not using *.py and PyCache is deprecated. No idea why you would want to avoid the creation  of the pyc cache. Maybe it is time to stop building out of date hardware.

 

I doubt that, why would Python 3.8 introduce  PYTHONPYCACHEPREFIX to set the location of the pyc file(s), if pycache is deprecated? I find no reference to this deprecation, do you have a docs link?

 

You misread my post. Using *.py and PyCache is current. Not using *.py and PyCache is deprecated. i.e. using *.pyc that are not in the cache folder.



Re: develop: python3 transition #74 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 22:20

And I asked you for references, as I disagree.

 

If __pycache__ was the required way to go (when deprecated goes to unsupported), why introduce  PYTHONPYCACHEPREFIX in 3.8, which is exactly to avoid having to use __pycache__ ?

 

Clearly, if you want to use pyc's NOT as a runtime cache, but as deployment files, they HAVE to be where import expects them.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #75 Huevos

  • PLi® Contributor
  • 4,663 posts

+163
Excellent

Posted 13 December 2021 - 22:22

BTW, I am surprised you are building so many boxes. Are people still using Vu+ Solo original? You must have unlimited resources.



Re: develop: python3 transition #76 Huevos

  • PLi® Contributor
  • 4,663 posts

+163
Excellent

Posted 13 December 2021 - 22:27

So going back to the original point...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.py"):

Is not going to work on your vision of PLi... and...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.pyc"):

Is not going to work in any other distro...

 

So whoever is converting shared plugins needs to take that into consideration.



Re: develop: python3 transition #77 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 22:31

From what I read, the main argument for using __pycache__ is the ability to use different python interpreter versions. Which is not relevant for us, we know exactly which version is installed.


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #78 Huevos

  • PLi® Contributor
  • 4,663 posts

+163
Excellent

Posted 13 December 2021 - 22:33

And I asked you for references, as I disagree.

 

If __pycache__ was the required way to go (when deprecated goes to unsupported), why introduce  PYTHONPYCACHEPREFIX in 3.8, which is exactly to avoid having to use __pycache__ ?

 

Clearly, if you want to use pyc's NOT as a runtime cache, but as deployment files, they HAVE to be where import expects them.

PYTHONPYCACHEPREFIX environment variable is used so  the PyCache can be a separate parallel filesystem tree, rather than the default __pycache__ subdirectories within each source directory. It is nothing to do with having *.pyc files directly in the source folders.


Edited by Huevos, 13 December 2021 - 22:34.


Re: develop: python3 transition #79 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 22:38

BTW, I am surprised you are building so many boxes. Are people still using Vu+ Solo original? You must have unlimited resources.

 

:)

 

You would be suprised :)  As long as we can build, we will, but there will come a time when we won't be able to anymore. But if it is up to me, not because we start deploying a python cache.

 

And no, not unlimited, but for now we've got plenty capacity, Even with develop in a continuous build we're not hitting 40% of the capacity (assuming nobody commits a glibc change that triggers a full rebuild ;)).


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: develop: python3 transition #80 WanWizard

  • PLi® Core member
  • 70,546 posts

+1,813
Excellent

Posted 13 December 2021 - 22:39

PYTHONPYCACHEPREFIX environment variable is used so  the PyCache can be a separate parallel filesystem tree, rather than the default __pycache__ subdirectories within each source directory. It is nothing to do with having *.pyc files directly in the source folders.

 

It can be (and is suggested to) used for that, as we compile every py file individually in the bb recipe, so we can set it for every py individually, equal to the path of the py.
 


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Ultimate (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.



2 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users