Jump to content


Photo

EPGImport mod


  • Please log in to reply
72 replies to this topic

Re: EPGImport mod #21 Dimitrij

  • PLi® Core member
  • 6,737 posts

+222
Excellent

Posted 12 September 2013 - 15:32

Is this available in the feed? As an update to xmltv importer or a seperate package?

First, it is necessary to solve the problem http://openpli.org/f...ndpost&p=372921

 

Question to experts...

Why not stop subprocess at the end of the import?


Lunix3-4K/VU+solo4k(thanks guys to gisclub.tv)


Re: EPGImport mod #22 MiLo

  • PLi® Core member
  • 13,927 posts

+294
Excellent

Posted 13 September 2013 - 06:38

"popen" is broken when running Enigma2 (just like os.system). No one knows why. You get an out-of-memory error on many systems if you use it.

Use enigma's own eConsoleApp and it will work just fine. Don't use os.system, don't use os.popen.

Edited by MiLo, 13 September 2013 - 06:39.

Real musicians never die - they just decompose

Re: EPGImport mod #23 gorski

  • Senior Member
  • 1,700 posts

+46
Good

Posted 13 September 2013 - 17:51

Thanx, Milo!

 

I hope it's not a problem to switch to other bits, from those broken ones... Any tips/tricks, please?


<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>

Re: EPGImport mod #24 Dimitrij

  • PLi® Core member
  • 6,737 posts

+222
Excellent

Posted 13 September 2013 - 18:20

"popen" is broken when running Enigma2 (just like os.system). No one knows why. You get an out-of-memory error on many systems if you use it.

Use enigma's own eConsoleApp and it will work just fine. Don't use os.system, don't use os.popen.

The problem in the wrong/incorrect threads.

This problem is solved in the long DMM.

 just warn when eTimers/eSockeNotifiers are stopped from wrong thread

http://git.opendream...7bb58a9ea53cb1  and etc...

 

Example:

EPGImport mod-->plugin.py-->def doneImport...

	if reboot:
		if Screens.Standby.inStandby:
			print>>log, "[EPGImport] Restart enigma2"
			restartEnigma(True)
		else:
			msg = _("EPG Import finished, %d events") % epgimport.eventCount + "\n" + _("You must restart Enigma2 to load the EPG data,\nis this OK?")
			_session.openWithCallback(restartEnigma, MessageBox, msg, MessageBox.TYPE_YESNO, timeout = 15, default = True)
			print>>log, "[EPGImport] Need restart enigma2"
	else:
		if config.plugins.epgimport.parse_autotimer.value and fileExists("/usr/lib/enigma2/python/Plugins/Extensions/AutoTimer/plugin.py"):
			try:
				from Plugins.Extensions.AutoTimer.plugin import autotimer
				if autotimer is None:
					from Plugins.Extensions.AutoTimer.AutoTimer import AutoTimer
					autotimer = AutoTimer()
				autotimer.parseEPGAsync(simulateOnly=False)
				print>>log, "[EPGImport] Run start parse AutoTimer"
				checkDeepstandby(_session, parse=True)
			except:
				print>>log, "[EPGImport] Could not start AutoTimer"
				checkDeepstandby(_session, parse=False)
		else:
			checkDeepstandby(_session, parse=False)

class checkDeepstandby:
	def __init__(self, session, parse=False):
		self.session = session
		self.parse = parse
		self.count = 0
		self.FirstwaitCheck = None
		if self.parse:
			self.FirstwaitCheck = enigma.eTimer()
			self.FirstwaitCheck.callback.append(self.runCheckDeepstandby)
			self.FirstwaitCheck.startLongTimer(40) 
			print>>log, "[EPGImport] Wait for parse AutoTimer 40 sec"
		else:
			self.runCheckDeepstandby()

	def runCheckDeepstandby(self):
		print>>log, "[EPGImport] Run check deep standby after import"

If self.FirstwaitCheck.startLongTimer delay > 50 sec,def runCheckDeepstandby never executed

 

Sorry for my bad English.


Edited by Dima73, 13 September 2013 - 18:22.

Lunix3-4K/VU+solo4k(thanks guys to gisclub.tv)


Re: EPGImport mod #25 MiLo

  • PLi® Core member
  • 13,927 posts

+294
Excellent

Posted 13 September 2013 - 18:59


"popen" is broken when running Enigma2 (just like os.system). No one knows why. You get an out-of-memory error on many systems if you use it.

Use enigma's own eConsoleApp and it will work just fine. Don't use os.system, don't use os.popen.

The problem in the wrong/incorrect threads.
This problem is solved in the long DMM.


That link only gives me:

404 - Unknown commit object
Real musicians never die - they just decompose

Re: EPGImport mod #26 Dimitrij

  • PLi® Core member
  • 6,737 posts

+222
Excellent

Posted 13 September 2013 - 20:06

Sorri

http://git.opendreambox.org/?p=enigma2.git;a=commit;h=d70d6252cbc87bdb137cb5b8c27bb58a9ea53cb1

Lunix3-4K/VU+solo4k(thanks guys to gisclub.tv)


Re: EPGImport mod #27 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 13 September 2013 - 20:41

no, I'm quite sure that's not related.

The popen/os.system issue is fork returning an error with errno 12 (ENOMEM), which is not related to what the commitlog describes.

Re: EPGImport mod #28 gorski

  • Senior Member
  • 1,700 posts

+46
Good

Posted 13 September 2013 - 22:16

Here are 4 crashlogs, if that can help at all....

 

It's Amiko Alien2 STB with STi 7162 processor, so SH4...

 

Enigma2 is OpenPLi basis, on the one hand, Duck Tales GIT + AR-P - however, it's PBNIGMA, essentially.

 

The EPG Import is for Python 2.7 mipsel but I ripped it out of XP1000 and put into this SH4 rec and it worked... for a while... and then the "jitters" started... similar to what others reported...

 

Best of luck to coders and thanx in advance!

Attached Files


<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>

Re: EPGImport mod #29 Dimitrij

  • PLi® Core member
  • 6,737 posts

+222
Excellent

Posted 14 September 2013 - 06:57

no, I'm quite sure that's not related.

The popen/os.system issue is fork returning an error with errno 12 (ENOMEM), which is not related to what the commitlog describes.

This is the tip of the iceberg.
I think
wrong/incorrect threads explains the error if use os.popen/os.system, eTimers/eSockeNotifiers.

Please make a test of this post http://openpli.org/f...ndpost&p=373256

I think you are very surprised by the results.


Lunix3-4K/VU+solo4k(thanks guys to gisclub.tv)


Re: EPGImport mod #30 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 14 September 2013 - 12:11

Please make a test of this post http://openpli.org/f...ndpost&p=373256
I think you are very surprised by the results.

not possible, it's all closed source.

Re: EPGImport mod #31 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 14 September 2013 - 15:12

And as I said earlier, I don't think this (filedescriptors, timers, pthreads) has anything to do with the fork issue.
The fork issue is simple; linux does not like it very much when you try to fork a process which has a virtual memory footprint of 100% (or more) of the physical memory.
And e2 happens to be such a process, on setopboxes with a relatively small amount of ram.

But the same thing will happen on a linux pc, when you try to fork a memory-hogging process such as e2.
(though there aren't many such processes around in the real world, and pc installations usually have swap, which helps avoid linux reporting
ENOMEM)

So this is normal fork behaviour, and for that reason you should not fork e2.
A couple of years ago, I replaced the fork in eConsoleAppContainer by a vfork, just to avoid these issues.
Therefore, please use eConsoleAppContainer, and avoid popen & friends from e2 context.

Re: EPGImport mod #32 Dimitrij

  • PLi® Core member
  • 6,737 posts

+222
Excellent

Posted 14 September 2013 - 15:19

pieterg

Let's look at a specific case.
Why is not performed
eTimer if it is greater than 50 seconds.

 

EPGImport mod-->plugin.py-->def doneImport...

	if reboot:
		if Screens.Standby.inStandby:
			print>>log, "[EPGImport] Restart enigma2"
			restartEnigma(True)
		else:
			msg = _("EPG Import finished, %d events") % epgimport.eventCount + "\n" + _("You must restart Enigma2 to load the EPG data,\nis this OK?")
			_session.openWithCallback(restartEnigma, MessageBox, msg, MessageBox.TYPE_YESNO, timeout = 15, default = True)
			print>>log, "[EPGImport] Need restart enigma2"
	else:
		if config.plugins.epgimport.parse_autotimer.value and fileExists("/usr/lib/enigma2/python/Plugins/Extensions/AutoTimer/plugin.py"):
			try:
				from Plugins.Extensions.AutoTimer.plugin import autotimer
				if autotimer is None:
					from Plugins.Extensions.AutoTimer.AutoTimer import AutoTimer
					autotimer = AutoTimer()
				autotimer.parseEPGAsync(simulateOnly=False)
				print>>log, "[EPGImport] Run start parse AutoTimer"
				checkDeepstandby(_session, parse=True)
			except:
				print>>log, "[EPGImport] Could not start AutoTimer"
				checkDeepstandby(_session, parse=False)
		else:
			checkDeepstandby(_session, parse=False)

class checkDeepstandby:
	def __init__(self, session, parse=False):
		self.session = session
		self.parse = parse
		self.count = 0
		self.FirstwaitCheck = None
		if self.parse:
			self.FirstwaitCheck = enigma.eTimer()
			self.FirstwaitCheck.callback.append(self.runCheckDeepstandby)
			self.FirstwaitCheck.startLongTimer(40) #!!!!!!!!!!!!!!!!
			print>>log, "[EPGImport] Wait for parse AutoTimer 40 sec"
		else:
			self.runCheckDeepstandby()

	def runCheckDeepstandby(self):
		print>>log, "[EPGImport] Run check deep standby after import"

If self.FirstwaitCheck.startLongTimer delay > 50 sec,def runCheckDeepstandby never executed


Edited by Dima73, 14 September 2013 - 15:20.

Lunix3-4K/VU+solo4k(thanks guys to gisclub.tv)


Re: EPGImport mod #33 MiLo

  • PLi® Core member
  • 13,927 posts

+294
Excellent

Posted 14 September 2013 - 16:43

A couple of years ago, I replaced the fork in eConsoleAppContainer by a vfork, just to avoid these issues.

Hmm, couldn't we also do the same modification in Python then?
Real musicians never die - they just decompose

Re: EPGImport mod #34 MiLo

  • PLi® Core member
  • 13,927 posts

+294
Excellent

Posted 14 September 2013 - 16:44

I've also thought about replacing os.system with something that throws an exception kindly asking the user to never call this method and use eConsoleApp instead.
Real musicians never die - they just decompose

Re: EPGImport mod #35 radxnl

  • Senior Member
  • 1,527 posts

+57
Good

Posted 14 September 2013 - 17:19

I've also thought about replacing os.system with something that throws an exception kindly asking the user to never call this method and use eConsoleApp instead.


+1

Re: EPGImport mod #36 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 15 September 2013 - 12:07


A couple of years ago, I replaced the fork in eConsoleAppContainer by a vfork, just to avoid these issues.

Hmm, couldn't we also do the same modification in Python then?

Yes, probably. But things like os.system block the mainloop, so people should still use eConsoleApp instead...

Re: EPGImport mod #37 gorski

  • Senior Member
  • 1,700 posts

+46
Good

Posted 7 October 2013 - 23:10

Sounds like quite a conundrum, sadly...

 

I know a guy who is now working on this - any more pointers, please?

 

This seems to be a really interesting point, if one were to crack it and improve the whole of E2, in the process...

 

Thanx!


<span style='font-family: comic sans ms,cursive'>"Enlightenment is man's emergence from his self-incurred immaturity. Immaturity is the inability to use one's own understanding without the guidance of another. This immaturity is self-incurred if its cause is not lack of understanding, but lack of resolution and courage to use it without the guidance of another. The motto of enlightenment is therefore: Sapere aude! Have courage to use your own understanding!</span><br /> <br /><span style='font-family: comic sans ms,cursive'>Laziness and cowardice are the reasons why such a large proportion of men, even when nature has long emancipated them from alien guidance..." I. Kant, "Political writings" (1784)</span><br /> <br /><span style='font-family: comic sans ms,cursive'><a class='bbc_url' href='<a class='bbc_url' href='http://eserver.org/p...lightenment.txt'>http://eserver.org/p...ent.txt</a>'><a class='bbc_url' href='http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a>'>http://www.english.upenn.edu/~mgamer/Etexts/kant.html</a></a> - the jolly text on Enlightenment, at the basis of Modernity...</span>

Re: EPGImport mod #38 athoik

  • PLi® Core member
  • 7,999 posts

+300
Excellent

Posted 8 October 2013 - 08:28

What if we use posix_spawn ?

http://stackoverflow...cesses-on-linux
http://twistedmatrix...rac/ticket/5710

Or turn on overcommit?

http://bryanmarty.co...14/forking-jvm/
Wavefield T90: 0.8W - 1.9E - 4.8E - 13E - 16E - 19.2E - 23.5E - 26E - 33E - 39E - 42E - 45E on EMP Centauri DiseqC 16/1
Unamed: 13E Quattro - 9E Quattro on IKUSI MS-0916

Re: EPGImport mod #39 MiLo

  • PLi® Core member
  • 13,927 posts

+294
Excellent

Posted 8 October 2013 - 19:58

overcommit is already enabled.

 

Just don't use os.system. It's really really really bad because it blocks the GUI thread. And 99% of the silly things that people use os.system for (e.g. renaming files) can be done perfectly well from within Python without spending all that effort on forking a new process.


Real musicians never die - they just decompose

Re: EPGImport mod #40 Dimitrij

  • PLi® Core member
  • 6,737 posts

+222
Excellent

Posted 30 December 2014 - 14:44

Ver. 1.5
-add option "Load EPG only services in bouquets"(yes/no)


Lunix3-4K/VU+solo4k(thanks guys to gisclub.tv)





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users