371 replies to this topic

[WanWizard]

I did some digging for the reasoning behind the __pycache__ directory.

 

As the main ones, these are given:

• Bytecode is not compatible across Python versions. If the same module was being imported by different versions of Python, each would overwrite the .pyc file with its own flavor of bytecode on import. This would invalidate the cache for all other versions.
• The .pyc cache could be used even without a corresponding .py file, which allowed some space saving (by distributing only the compiled file). However, if one deleted a .py file but forgot to also remove the .pyc, Python would act as if the module was still present. This was quite confusing, especially for beginners.

Both are in my opinion not valid for our specific use case.

 

We will never have Python variations as Python itself and all it's modules are produced in the same bitbake run, and we're not having issues with beginners, we deal with end-users who don't care about any of this.

[Huevos]

 

BTW, I am surprised you are building so many boxes. Are people still using Vu+ Solo original? You must have unlimited resources.

 

 

You would be suprised   As long as we can build, we will, but there will come a time when we won't be able to anymore. But if it is up to me, not because we start deploying a python cache.

 

And no, not unlimited, but for now we've got plenty capacity, Even with develop in a continuous build we're not hitting 40% of the capacity (assuming nobody commits a glibc change that triggers a full rebuild ).

 

Are you doing a full rebuild (image and feeds)?

 

If you have the capacity fair enough, but how does the image perform on one of those old relics like the original Solo?

 

We don't have unlimited resources so minimum cut-off has been set at 750 MHz dual core processor.

[WanWizard]

Are you doing a full rebuild (image and feeds)?

 

No, no point rebuilding things that haven't changed. But bitbake takes care of that. We just run automated ( custom scripted ) make image / make feed's.

 

If you have the capacity fair enough, but how does the image perform on one of those old relics like the original Solo?

 

User experience probably varies, depending on expectation and what you're used to.

 

The oldest box I have here is the XP1000, which doesn't even have an FHD framebuffer. If your scope is "I want to watch TV", only use a softcam, and don't need millisecond response, I guess it is usable. I've used a VU Zero for a very long time as my "travel box" for use in hotels, etc, and that worked fine.

 

But don't do a direct comparison with a recent ARM box...

[Huevos]

Yes, bitbake handles it but by full rebuild I meant image and feeds. For me that takes about 30 minutes per machine on subsequent builds.

[WanWizard]

Our buildservers do that in about 9 to 12 minutes, if memory serves me well, if nothing has changed it builds 24 packages (there are some we always build, and some I still need to fix).

 

Our build has 4 stages: image, feed, regenerate package indexes, rsync to the distribution server.

 

We can run parallel builds if needed, we have multiple buildservers. And we build on NFS so all can access the same tmp.

[Huevos]

Task count here is about 17000. What is it on PLi?

 

And package count is normally around 65.

Edited by Huevos, 14 December 2021 - 01:27.

[Taapat]

So going back to the original point...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.py"):

Is not going to work on your vision of PLi... and...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.pyc"):

Is not going to work in any other distro...
 
So whoever is converting shared plugins needs to take that into consideration.

 

 
Maybe I missed something but I personally think that such a check is not correct.
The fact that a python file exists does not mean that it is working properly.
I think it is better to use try in the following cases:

try:
        from Plugins.Extensions.PicturePlayer.plugin import Pic_Thumb, picshow
        PicPlayerAviable = True
except ImportError:
        pass

Edited by Taapat, 14 December 2021 - 07:48.

[Huevos]

 

So going back to the original point...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.py"):

Is not going to work on your vision of PLi... and...

if fileExists("/usr/lib/enigma2/python/Plugins/Extensions/PicturePlayer/plugin.pyc"):

Is not going to work in any other distro...
 
So whoever is converting shared plugins needs to take that into consideration.

 

 
Maybe I missed something but I personally think that such a check is not correct.
The fact that a python file exists does not mean that it is working properly.
I think it is better to use try in the following cases:

try:
        from Plugins.Extensions.PicturePlayer.plugin import Pic_Thumb, picshow
        PicPlayerAviable = True
except ImportError:
        pass

We are kind of going off topic here... I just wanted Beeker to be aware of the PLi view about *.py files... but anyway...

 

You can do that if you actually want to do the import, but sometimes you don't. In OpenViX (and OpenATV) we have a isPluginInstalled() method, so we can do things like this:

SystemInfo["HDMICEC"] = (fileExists("/dev/hdmi_cec") or fileExists("/dev/misc/hdmi_cec0")) and isPluginInstalled("HdmiCEC")

 

[littlesat]

Actually checking if a plugin is available via a file is a 'bad' thing... better check if the application is just there by checking the available plugin list... Having a function isPluginInstalled is indeed a nice and correct thing to have... but it should analyze it in a proper way and not this way...

 

def isPluginInstalled(pluginName, pluginFile="plugin", pluginType=None):
                for type in [x for x in listdir(scopePlugins) if x != "__pychache__" and isdir(pathjoin(scopePlugins, x))]:
                                for extension in ["", "o", "c"]:
                                                if isfile(pathjoin(scopePlugins, type, pluginName, "%s.py%s" % (pluginFile, extension))):
                                                                if pluginType and type != pluginType:
                                                                                continue
                                                                return True
                return False

Edited by littlesat, 14 December 2021 - 11:37.

[littlesat]

Better consider to analyze it somehow like this... (I know translation might be chalenging.... but what if we add a not translated field).

def isPluginInstalled(pluginName, pluginType=None):
    return pluginName in [p.name for p in plugins.getPlugins(where=pluginType)]

Edited by littlesat, 14 December 2021 - 11:44.

[Huevos]

 

Better consider to analyze it somehow like this... (I know translation might be chalenging.... but what if we add a not translated field).

def isPluginInstalled(pluginName, pluginType=None):
    return pluginName in [p.name for p in plugins.getPlugins(where=pluginType)]

Which means it is only available after plugins have loaded, right?

[littlesat]

Yes that is indeed another 'point' that might be challenging as within SystemInfo.py's first execution the plugins are not loaded yet.... So maybe we need to look for a point to load them earlier -or- at least investigate it...

 

Actually it is still wrong to do the check by checking for a file existing... sounds like it was a quick solution to check if a plugin was available avoiding challenges instead of doing it right (and maybe, I know, in the past I also did use this method... maybe the picture player check was years ago my thing).

 

At this moment due to my circumstances I do not have the overview anymore if the plugins are already 'loaded' here... I just look at the code and think the method by checking files, while you also obtain lists, is not really OK...

 

At the end this is a bit of off topic... and it interferes with a py3 migration. So it would be nice to put this on a kind of 'something to investigate' list and just fix it for py3....  and maybe consider via glob to do something with a wild card instead of looping through "", "c", "o"...  For the py3 transfer we can just check for the plugin file like e.g. this.... and then we can include wildcards...

import glob
 
if glob.glob('C:/temp/Test_Folder/file*.txt'):
    print ("File exist with starting string file")

Edited by littlesat, 14 December 2021 - 13:51.

[WanWizard]

Task count here is about 17000. What is it on PLi?

 

And package count is normally around 65.

 

Just ran a local build for the vuduo4k, 5648 tasks for build-image, 8427 tasks for build-feed. And one extra for the regeneration of the package indexes.

 

I noticed some time ago OE-A has quite a lot more, but that is probably due to duplication in the way the OE repo is organized. A lot of the recipes are not relevant for any given machine, but are parsed and counted.

 

6 boxes, 58 minutes. On my machine (which is identical to our buildservers, but uses local SATA instead of SSD over NFS):

all.upload-generic-feeds:2021-12-14 14:24:22: OK
armv7ahf-neon.upload-generic-feeds:2021-12-14 14:24:30: OK
armv7vehf-neon-vfpv4.upload-generic-feeds:2021-12-14 14:24:30: OK
cortexa15hf-neon-vfpv4.upload-generic-feeds:2021-12-14 14:24:33: OK
dual.build-feed:2021-12-14 14:24:06: OK
dual.build-image:2021-12-14 14:22:01: OK
dual.upload-feed:2021-12-14 14:24:22: OK
dual.upload-image:2021-12-14 14:24:21: OK
h9.build-feed:2021-12-14 14:15:34: OK
h9.build-image:2021-12-14 14:13:20: OK
h9.upload-feed:2021-12-14 14:24:19: OK
h9.upload-image:2021-12-14 14:24:18: OK
mips32el-nf.upload-generic-feeds:2021-12-14 14:24:27: OK
mips32el.upload-generic-feeds:2021-12-14 14:24:25: OK
sf8008.build-feed:2021-12-14 13:57:47: OK
sf8008.build-image:2021-12-14 13:54:37: OK
sf8008.upload-feed:2021-12-14 14:24:15: OK
sf8008.upload-image:2021-12-14 14:24:15: OK
vipert2c.build-feed:2021-12-14 13:46:47: OK
vipert2c.build-image:2021-12-14 13:43:18: OK
vipert2c.upload-feed:2021-12-14 14:24:11: OK
vipert2c.upload-image:2021-12-14 14:24:10: OK
vuduo4k.build-feed:2021-12-14 13:37:05: OK
vuduo4k.build-image:2021-12-14 13:33:49: OK
vuduo4k.upload-feed:2021-12-14 14:24:09: OK
vuduo4k.upload-image:2021-12-14 14:24:08: OK
xp1000.build-feed:2021-12-14 14:07:45: OK
xp1000.build-image:2021-12-14 14:04:23: OK
xp1000.upload-feed:2021-12-14 14:24:17: OK
xp1000.upload-image:2021-12-14 14:24:17: OK
> FINISHED, failed: 0, build took 0 day(s) 00:58:17

Edited by WanWizard, 14 December 2021 - 15:36.

[zeros]

Are you planning public images for python3 as well? If yes, I'm grateful, if you can add Lunix3-4K as well. I wouldn't want to start a building myself, but I'd love to test.

I can see under openpli-oe-core now python3 branch exists... 

[WanWizard]

I haven't planned anything yet, but in terms of testing, if would be logical to build python3 images for lets say 10 boxes, the more is tested, the quicker we can make the move.

 

@Beeker,

 

I've created the python3 branch for OE, and I've send you a github invite, so I can give you push access to it.

[Beeker]

Thanks.

[WanWizard]

You should have write access now.

 

We do have to think (and be careful) about rebase strategy now, to keep develop, python3 and hardknott in sync.

[el bandido]

Task counts for the osmio4kplus building from the OpenPLli python3 branch:

Image = 6282 tasks
Image + feed = 10283 tasks

[rantanplan]

You have to take a closer look.
So far this is only an branch of develop.
It is now adjusted towards python3.
There is still some work to be done.
So far you can't build a python3 image from it.

[WanWizard]

I'm just wondering: is it really needed to have both a python3 branch and a hardknott / honister branch, or can the two be combined ?

 

As far as I can see, there is no overlap in the work needed for both, and it saves a lot of rebasing and test image building...