Jump to content


Photo

Memory leak in MovieList.py


  • Please log in to reply
23 replies to this topic

#1 Persian Prince

  • Senior Member
  • 1,982 posts

+247
Excellent

Posted 30 November 2020 - 16:26

Hello,

 

I was talking to phunkyfish earlier about Kodi addon for enigma2 and seems there's a problem in all Open images and that's in MovieList.py

 

Please check https://github.com/E...ebif/issues/751


Open Vision sources: https://github.com/OpenVisionE2


Re: Memory leak in MovieList.py #2 Abu Baniaz

  • PLi® Contributor
  • 2,414 posts

+61
Good

Posted 30 November 2020 - 16:30

This was one of the threads in 2012 that first showed issue
https://forums.openp...ilepush-buffer/

Maybe the thread in Dev section may help once it is added
https://forums.openp...es-in-filepush/

Edited by Abu Baniaz, 30 November 2020 - 16:33.


Re: Memory leak in MovieList.py #3 Persian Prince

  • Senior Member
  • 1,982 posts

+247
Excellent

Posted 30 November 2020 - 16:45

Well I don't have access to that section as I'm not a developer/contributor probably but lets focus on this:

 


I have made a copy of MovieList.py from enigma2 and then i have removed all the GUI parts.
I think the leak is in the GUI part of MovieList.py.

 

by jbleyel


Open Vision sources: https://github.com/OpenVisionE2


Re: Memory leak in MovieList.py #4 WanWizard

  • PLi® Core member
  • 68,303 posts

+1,718
Excellent

Posted 30 November 2020 - 16:47

@Abu Baniaz,

 

Is this related to the issue Erik and Simon are looking at?


Currently in use: VU+ Duo 4K (2xFBC S2), VU+ Solo 4K (1xFBC S2), uClan Usytm 4K Pro (S2+T2), Octagon SF8008 (S2+T2), Zgemma H9.2H (S2+T2)

Due to my bad health, I will not be very active at times and may be slow to respond. I will not read the forum or PM on a regular basis.

Many answers to your question can be found in our new and improved wiki.


Re: Memory leak in MovieList.py #5 Abu Baniaz

  • PLi® Contributor
  • 2,414 posts

+61
Good

Posted 30 November 2020 - 17:04

Not sure, maybe I should have kept quiet  :(.  filepush is a denominator.

 

I don't think any of the regular E2 testers use their E2 boxes as "Kodi box" sources (setup that Phunkyfish uses regularly) so do not experience the issue regularly.  The issue seems to affect streaming to other boxes too, which is more likely to affect more E2 testers.

 

Thread on Vix here where Phunkyfish started a thread:

https://www.world-of-satellite.com/showthread.php?63317-Possible-memory-leak-need-help-on-how-to-figure-it-out

He has confirmed leak is in movielist.py

 

The commits Simon has made to ViX have not been pushed to Public image, so Phunkyfish cant test those.



Re: Memory leak in MovieList.py #6 Erik Slagter

  • PLi® Core member
  • 46,951 posts

+541
Excellent

Posted 30 November 2020 - 17:37

The issue we're working on is a deadlock, not a memory leak.


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: Memory leak in MovieList.py #7 MiLo

  • PLi® Core member
  • 14,042 posts

+298
Excellent

Posted 6 December 2020 - 09:33

Since MovieList is all Python code, memory "leak" can only be caused by some client keeping a reference to some object alive.

 

I'm unfamiliar with the webinterface, I lost track of how it works long ago.

 

It would be really heplful to know which call is causing it.


Real musicians never die - they just decompose

Re: Memory leak in MovieList.py #8 Erik Slagter

  • PLi® Core member
  • 46,951 posts

+541
Excellent

Posted 6 December 2020 - 10:24

I'd very interested in how was established that there is memory and it's exactly there, in MoveList.py. I am a bit sceptical for the moment.


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: Memory leak in MovieList.py #9 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 6 December 2020 - 11:20

I did look at it, but I have not enough time at the moment.

According to the GitHub comments it’s this call:
.../web/movielist?dirname=

I can confirm that memory usage increases. Maybe getting the info object causes the problem:
https://github.com/O...vieList.py#L635
At least when I comment it out and set “info = None” memory usage increase is less. But that has to be tested /analyzed more.

By the way: When I adapt WebIf so that MovieList is not used, memory usage also increases, but much less than with MovieList.

Edited by betacentauri, 6 December 2020 - 11:21.

Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: Memory leak in MovieList.py #10 Erik Slagter

  • PLi® Core member
  • 46,951 posts

+541
Excellent

Posted 6 December 2020 - 11:55

How is determined that memory usage increases? As most, well-known sources of memory usage, are actually not accurate at all (I think of "free" and "ps" statistics).


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: Memory leak in MovieList.py #11 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 6 December 2020 - 13:23

I know it might not be 100% accurate, but I looked at top output.
User on GitHub says that box is inaccessible after some hours. So I think there is really a problem with a memory leak.
Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: Memory leak in MovieList.py #12 Erik Slagter

  • PLi® Core member
  • 46,951 posts

+541
Excellent

Posted 6 December 2020 - 17:49

That could very well be buggy drivers, they're all over the place.

 

Simple test: restart enigma every day and see if the memory no longer runs out (not by consulting top, but by the lack of (hopefully) hangs). I bet the issue remains, even when enigma is restarted frequently (which would release all of it's memory).


* Wavefrontier T90 with 28E/23E/19E/13E via SCR switches 2 x 2 x 6 user bands
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.


Re: Memory leak in MovieList.py #13 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 12 December 2020 - 12:23

Here a MovieList object is created:

https://github.com/E.../movies.py#L150

A few lines later load() is called. It fills the movielist.list.

The movielist.list contains pointers to info(created in c++ code) objects. The info objects are ePtr.

 

At the end of the procedure "del movielist" is called. From my point of view this should also implicitly call something like this "del movielist.list". Or?

 

But when I replace at the end of the procedure "del movielist" with ""del movielist.list" memory consumption (according to top) doesn't grow much any longer. So it looks like a big difference.

Why does a "del movielist" don't free the objects in the movielist.list?

 

(If really needed, I can add debug statements in the constructor/destructor of the info object to confirm my observation)


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: Memory leak in MovieList.py #14 MiLo

  • PLi® Core member
  • 14,042 posts

+298
Excellent

Posted 12 December 2020 - 13:56

MovieList is a GUI thingy, so that might make it behave strange when there's no actual GUI to display. Probably deleting that "list" object makes it release things a bit earlier.

 

The "del movielist" at the end is a no-op, it doesn't actually do anything. The method ends there, so all variables go out of scope there anyway and the whole local namespace gets deleted there anyway.

 

"del someobject" does not call the objects destructor, it just removes the local reference "someobject" from the namespace. It will call the destructor then if this happens to be the last reference to it.

 

(Example: a=[1,2]; b=a; del a; This will delete "a" but not delete the actual array since b still has a reference to it)

 

 

Usually objects not getting freed when you expect it are a result of cyclic references. In Python, the garbage collector will find and destroy such objects. but it's hard to predict when that will run. Sometimes, something is keeping a reference alive. Tricky to find.


Real musicians never die - they just decompose

Re: Memory leak in MovieList.py #15 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 12 December 2020 - 14:16

Yes, that's what I also thought.

I don't think that waiting until the garbage collector has made his job, helps here. Even when I wait minutes there is no difference. So most likely there is still a reference alive. But I currently don't see where.


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: Memory leak in MovieList.py #16 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 12 December 2020 - 14:49

I made a little change in MovieList.py. I changed this line https://github.com/O...vieList.py#L635 to

info = None

That means justStubInfo object is used everytime.

Then I have added a

print("RefCount", sys.getrefcount(justStubInfo))

here: https://github.com/E.../movies.py#L237

(and an import for sys and justStubInfo is needed)

 

I can see that refcount grows and grows with each call of the website.

When change it to "del movielist.list" ref count stays the same. So there is really a problem.

 


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: Memory leak in MovieList.py #17 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 12 December 2020 - 16:54

I found the problem. MovieList creates c++ object

self.l = eListboxPythonMultiContent()

and gives a reference to self.buildMovieListEntry to it

self.l.setBuildFunc(self.buildMovieListEntry)

MovieList will not be destroyed until the reference is gone, but that will never happen as it references to itself.

 

As confirmation I used this and refcount don't grow anymore:

movielist.l.setBuildFunc(None)

How to solve this without manually call "movielist.l.setBuildFunc(None)" ???
 


Edited by betacentauri, 12 December 2020 - 16:54.

Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: Memory leak in MovieList.py #18 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 12 December 2020 - 17:07

This seems to do the trick

self.l.setBuildFunc(weakref.ref(self.buildMovieListEntry))

@MiLo: What do you think? Is this the correct way to solve the problem?


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: Memory leak in MovieList.py #19 MiLo

  • PLi® Core member
  • 14,042 posts

+298
Excellent

Posted 12 December 2020 - 17:29

Usually the best way to break a self-ref chain is to have a third level, e.g. typically with subscriptions:

 

a.parent = b

b.subscriptions.append(a)

 

Now there's a cyclic dependency. But 'a' must store 'b' to be able to unsubscribe.

One way to solve:

a.parent = b.subscriptions

 

This way, a doesn't increase the refcount to b, and a can still unsubscribe when it goes out of scope.

 

 

In this particular case though, it's not as easy to break. A weak ref would help yes. Another thing might be to call self.l.setBuildFunc(...) only when needed, and set it back to None again when not building a list (in the load() function maybe?)

 

(I'm not a big fan of weakref, but in this case, it's the lesser evil compared to keeping that cylic dependency in. Probably the issue here is the refcounting done in the C++ part, which does not have a garbage-collector or any other way to detect cyclic dependencies)


Edited by MiLo, 12 December 2020 - 17:32.

Real musicians never die - they just decompose

Re: Memory leak in MovieList.py #20 betacentauri

  • PLi® Core member
  • 7,185 posts

+323
Excellent

Posted 12 December 2020 - 18:00

Another thing might be to call self.l.setBuildFunc(...) only when needed, and set it back to None again when not building a list (in the load() function maybe?)

 

Don't know whether this is possible easily. I guess build function is also called when the list changes (when it's called e.g. via the MovieSelection screen). And that can also happen in the c++ part.

So in this specific case it would maybe work, but in general it's not my favorite solution as also other components are affected: EpgList, TimerList, GraphMultiEpg

 

So I will create a patch tomorrow or on Monday which uses weakrefs.


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04


3 user(s) are reading this topic

0 members, 2 guests, 0 anonymous users


    Bing (1)