develop: python3 transition
40H3X 11 jan 2022
Thanks at @foxbox and @blzr, I could but an update of my local repositories would remove the patch?
blzr 11 jan 2022
1) I suppose this plugin you'll build just once (unless you'd like to build image from scratch for some reason), so you can even patch sources in tmp/work directory
2) if you add a patch in your local vu bsp repo it won't be affected by openpli updates until vu bsp is updated upstream
rantanplan 11 jan 2022
You need to patch your local source to fix py3 syntax errors...
Full 2to3 diff for this plugin attached
(not sure how we'll proceed with vendors specific plugins like this? additional branch in bsp?)
for py3 all bsp changed
athoik 21 jan 2022
Yes, it makes sense to be unhappy, we used to compile all those years .py files to .pyo files.
With python3, now files are still compiled and moved to __pycache__ folder.
There is still a way to bring back py to pyc on same folder: python -m compileall -b .
OE-A already used "compileall -b"
https://github.com/o...91066e69cb9fR25
Edit1. I guess we need a small patch to make legacy by default = True, that will force generating .pyc without big changes.
https://github.com/p...b/compileall.py
Veranderd door athoik, 21 januari 2022 - 20:52
WanWizard 21 jan 2022
Funny to see that when I brought it up, I was beaten to within an inch of my life for such a stupid idea...
Huevos 21 jan 2022
Funny to see that when I brought it up, I was beaten to within an inch of my life for such a stupid idea...
It has been done on OE-A so it is still possible to shoehorn the image into small flash boxes.
No idea how others have resources to build all those boxes. They must have a bank of build servers.
blzr 22 jan 2022
Funny to see that when I brought it up, I was beaten to within an inch of my life for such a stupid idea...
yup, posts #67-#73 from this very thread come to mind somehow...
//anyway, all's well that ends well
athoik 22 jan 2022
Here is the patch for python3 to bring back the .pyc on the same folder.
diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index fe31f43..ba972c8 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -389,7 +389,7 @@ def cache_from_source(path, debug_override=None, *, optimization=None): tag = sys.implementation.cache_tag if tag is None: raise NotImplementedError('sys.implementation.cache_tag is None') - almost_filename = ''.join([(base if base else rest), sep, tag]) + almost_filename = ''.join([(base if base else rest)]) if optimization is None: if sys.flags.optimize == 0: optimization = '' @@ -426,7 +426,7 @@ def cache_from_source(path, debug_override=None, *, optimization=None): head.lstrip(path_separators), filename, ) - return _path_join(head, _PYCACHE, filename) + return _path_join(head, filename) def source_from_cache(path):
root@osmio4k:~/test# ls libpython3.9-1.0_3.9.9-r0_armv7ahf-neon.ipk python3-core_3.9.9-r0_armv7ahf-neon.ipk test.py root@osmio4k:~/test# python -c "import test" hello root@osmio4k:~/test# ls libpython3.9-1.0_3.9.9-r0_armv7ahf-neon.ipk python3-core_3.9.9-r0_armv7ahf-neon.ipk test.py test.pyc
I guess we can even change it to use .pyo instead of .pyc! But nope, let's keep .pyc to recognize 3or2
athoik 22 jan 2022
Also need to bring back the default optimization level to 1 as we used to, because it seems that we are getting multiple files now.
Eg.
WARNING: enigma2-plugin-systemplugins-servicemp3-gitAUTOINC+aa60f9a1d0-r0 do_package: QA Issue: enigma2-plugin-systemplugins-servicemp3: Files/directories were installed but not shipped in any package: /usr/lib/enigma2/python/Plugins/SystemPlugins/ServiceMP3/__init__.pyc /usr/lib/enigma2/python/Plugins/SystemPlugins/ServiceMP3/__init__.opt-1.pyc
diff --git a/Lib/importlib/_bootstrap_external.py b/Lib/importlib/_bootstrap_external.py index fe31f43..6ce18db 100644 --- a/Lib/importlib/_bootstrap_external.py +++ b/Lib/importlib/_bootstrap_external.py @@ -389,13 +389,13 @@ def cache_from_source(path, debug_override=None, *, optimization=None): tag = sys.implementation.cache_tag if tag is None: raise NotImplementedError('sys.implementation.cache_tag is None') - almost_filename = ''.join([(base if base else rest), sep, tag]) + almost_filename = ''.join([(base if base else rest)]) if optimization is None: if sys.flags.optimize == 0: optimization = '' else: optimization = sys.flags.optimize - optimization = str(optimization) + optimization = '' if optimization != '': if not optimization.isalnum(): raise ValueError('{!r} is not alphanumeric'.format(optimization)) @@ -426,7 +426,7 @@ def cache_from_source(path, debug_override=None, *, optimization=None): head.lstrip(path_separators), filename, ) - return _path_join(head, _PYCACHE, filename) + return _path_join(head, filename) def source_from_cache(path): diff --git a/Python/initconfig.c b/Python/initconfig.c index 116ee33..0e85d62 100644 --- a/Python/initconfig.c +++ b/Python/initconfig.c @@ -627,7 +627,7 @@ _PyConfig_InitCompatConfig(PyConfig *config) config->bytes_warning = -1; config->inspect = -1; config->interactive = -1; - config->optimization_level = -1; + config->optimization_level = 1; config->parser_debug= -1; config->write_bytecode = -1; config->verbose = -1; @@ -658,7 +658,7 @@ config_init_defaults(PyConfig *config) config->bytes_warning = 0; config->inspect = 0; config->interactive = 0; - config->optimization_level = 0; + config->optimization_level = 1; config->parser_debug= 0; config->write_bytecode = 1; config->verbose = 0; @@ -1996,7 +1996,7 @@ config_parse_cmdline(PyConfig *config, PyWideStringList *warnoptions, /* case 'J': reserved for Jython */ case 'O': - config->optimization_level++; + config->optimization_level = 1; break; case 'B':
WanWizard 23 jan 2022
Instead of .py you mean?
Yes, absolutely, identical to how we do it now with the .pyo's in the image, and the .py's in a -src package.
athoik 23 jan 2022
That means extra job for hardknott-py3 branch. From .pyo to .py and now back to .pyc
@Hains are you ok with that? Send the patch for .pyc ?
WanWizard 23 jan 2022
Why are those patches needed?
In the current bb's (in develop) we already run a "compileall" to make the pyo's for python2, and split the py's off into a separate package. That would make pyc's for python3, so that part is covered.
Python3 will use the pyc by default, if no py exists in the same directory (and never is the py does exist, which is different from python2, but something I have no issue with, only developers need source on their box).
Beeker 23 jan 2022
That means extra job for hardknott-py3 branch. From .pyo to .py and now back to .pyc
@Hains are you ok with that? Send the patch for .pyc ?
To save boxes with small flash memory I agree.
athoik 23 jan 2022
Why are those patches needed?
In the current bb's (in develop) we already run a "compileall" to make the pyo's for python2, and split the py's off into a separate package. That would make pyc's for python3, so that part is covered.
Python3 will use the pyc by default, if no py exists in the same directory (and never is the py does exist, which is different from python2, but something I have no issue with, only developers need source on their box).
We already changed logic on several packages to include .py. Now we have to move .py back to -src and deploy .pyc files.
Also we need to change some plugins to check for .pyc file existance. Used to check .pyo, changed to .py and now to .pyc
Also we used to delete them! So now we need them back.
find ${D}${libdir}/enigma2/python/ -name '*.pyc' -exec rm {} \;
Veranderd door athoik, 23 januari 2022 - 16:16
athoik 23 jan 2022
@WanWizard, could you please create a python3 branch from current master and provide push,merge rights for Hains, Nautilus7, Athoik (me) for the following:
WanWizard 23 jan 2022
Hains and you should have access now, I've send an invitation to Nautilus7.
nautilus7 23 jan 2022
Hains and you should have access now, I've send an invitation to Nautilus7.
Just accepted. Thanks!