72 replies to this topic

[Dimitrij]

Is this available in the feed? As an update to xmltv importer or a seperate package?

First, it is necessary to solve the problem http://openpli.org/f...ndpost&p=372921

 

Question to experts...

Why not stop subprocess at the end of the import?

[MiLo]

"popen" is broken when running Enigma2 (just like os.system). No one knows why. You get an out-of-memory error on many systems if you use it.

Use enigma's own eConsoleApp and it will work just fine. Don't use os.system, don't use os.popen.

Edited by MiLo, 13 September 2013 - 06:39.

[gorski]

Thanx, Milo!

 

I hope it's not a problem to switch to other bits, from those broken ones... Any tips/tricks, please?

[Dimitrij]

"popen" is broken when running Enigma2 (just like os.system). No one knows why. You get an out-of-memory error on many systems if you use it.

Use enigma's own eConsoleApp and it will work just fine. Don't use os.system, don't use os.popen.

The problem in the wrong/incorrect threads.

This problem is solved in the long DMM.

 just warn when eTimers/eSockeNotifiers are stopped from wrong thread

http://git.opendream...7bb58a9ea53cb1  and etc...

 

Example:

EPGImport mod-->plugin.py-->def doneImport...

	if reboot:
		if Screens.Standby.inStandby:
			print>>log, "[EPGImport] Restart enigma2"
			restartEnigma(True)
		else:
			msg = _("EPG Import finished, %d events") % epgimport.eventCount + "\n" + _("You must restart Enigma2 to load the EPG data,\nis this OK?")
			_session.openWithCallback(restartEnigma, MessageBox, msg, MessageBox.TYPE_YESNO, timeout = 15, default = True)
			print>>log, "[EPGImport] Need restart enigma2"
	else:
		if config.plugins.epgimport.parse_autotimer.value and fileExists("/usr/lib/enigma2/python/Plugins/Extensions/AutoTimer/plugin.py"):
			try:
				from Plugins.Extensions.AutoTimer.plugin import autotimer
				if autotimer is None:
					from Plugins.Extensions.AutoTimer.AutoTimer import AutoTimer
					autotimer = AutoTimer()
				autotimer.parseEPGAsync(simulateOnly=False)
				print>>log, "[EPGImport] Run start parse AutoTimer"
				checkDeepstandby(_session, parse=True)
			except:
				print>>log, "[EPGImport] Could not start AutoTimer"
				checkDeepstandby(_session, parse=False)
		else:
			checkDeepstandby(_session, parse=False)

class checkDeepstandby:
	def __init__(self, session, parse=False):
		self.session = session
		self.parse = parse
		self.count = 0
		self.FirstwaitCheck = None
		if self.parse:
			self.FirstwaitCheck = enigma.eTimer()
			self.FirstwaitCheck.callback.append(self.runCheckDeepstandby)
			self.FirstwaitCheck.startLongTimer(40) 
			print>>log, "[EPGImport] Wait for parse AutoTimer 40 sec"
		else:
			self.runCheckDeepstandby()

	def runCheckDeepstandby(self):
		print>>log, "[EPGImport] Run check deep standby after import"

If self.FirstwaitCheck.startLongTimer delay > 50 sec,def runCheckDeepstandby never executed

 

Sorry for my bad English.

Edited by Dima73, 13 September 2013 - 18:22.

[MiLo]

"popen" is broken when running Enigma2 (just like os.system). No one knows why. You get an out-of-memory error on many systems if you use it.

Use enigma's own eConsoleApp and it will work just fine. Don't use os.system, don't use os.popen.

The problem in the wrong/incorrect threads.
This problem is solved in the long DMM.

That link only gives me:

404 - Unknown commit object

[Dimitrij]

Sorri

http://git.opendreambox.org/?p=enigma2.git;a=commit;h=d70d6252cbc87bdb137cb5b8c27bb58a9ea53cb1

[pieterg]

no, I'm quite sure that's not related.

The popen/os.system issue is fork returning an error with errno 12 (ENOMEM), which is not related to what the commitlog describes.

[gorski]

Here are 4 crashlogs, if that can help at all....

 

It's Amiko Alien2 STB with STi 7162 processor, so SH4...

 

Enigma2 is OpenPLi basis, on the one hand, Duck Tales GIT + AR-P - however, it's PBNIGMA, essentially.

 

The EPG Import is for Python 2.7 mipsel but I ripped it out of XP1000 and put into this SH4 rec and it worked... for a while... and then the "jitters" started... similar to what others reported...

 

Best of luck to coders and thanx in advance!

Attached Files

•  enigma2_crash_1378171862.log   17.44KB   1 downloads
•  enigma2_crash_1378200755.log   17.44KB   0 downloads
•  enigma2_crash_1378577261.log   17.45KB   0 downloads
•  enigma2_crash_1378685364.log   17.44KB   2 downloads

[Dimitrij]

no, I'm quite sure that's not related.

The popen/os.system issue is fork returning an error with errno 12 (ENOMEM), which is not related to what the commitlog describes.

This is the tip of the iceberg.
I think wrong/incorrect threads explains the error if use os.popen/os.system, eTimers/eSockeNotifiers.

Please make a test of this post http://openpli.org/f...ndpost&p=373256

I think you are very surprised by the results.

[pieterg]

Please make a test of this post http://openpli.org/f...ndpost&p=373256
I think you are very surprised by the results.

not possible, it's all closed source.

[pieterg]

And as I said earlier, I don't think this (filedescriptors, timers, pthreads) has anything to do with the fork issue.
The fork issue is simple; linux does not like it very much when you try to fork a process which has a virtual memory footprint of 100% (or more) of the physical memory.
And e2 happens to be such a process, on setopboxes with a relatively small amount of ram.

But the same thing will happen on a linux pc, when you try to fork a memory-hogging process such as e2.
(though there aren't many such processes around in the real world, and pc installations usually have swap, which helps avoid linux reporting
ENOMEM)

So this is normal fork behaviour, and for that reason you should not fork e2.
A couple of years ago, I replaced the fork in eConsoleAppContainer by a vfork, just to avoid these issues.
Therefore, please use eConsoleAppContainer, and avoid popen & friends from e2 context.

[Dimitrij]

pieterg

Let's look at a specific case.
Why is not performed eTimer if it is greater than 50 seconds.

 

EPGImport mod-->plugin.py-->def doneImport...

	if reboot:
		if Screens.Standby.inStandby:
			print>>log, "[EPGImport] Restart enigma2"
			restartEnigma(True)
		else:
			msg = _("EPG Import finished, %d events") % epgimport.eventCount + "\n" + _("You must restart Enigma2 to load the EPG data,\nis this OK?")
			_session.openWithCallback(restartEnigma, MessageBox, msg, MessageBox.TYPE_YESNO, timeout = 15, default = True)
			print>>log, "[EPGImport] Need restart enigma2"
	else:
		if config.plugins.epgimport.parse_autotimer.value and fileExists("/usr/lib/enigma2/python/Plugins/Extensions/AutoTimer/plugin.py"):
			try:
				from Plugins.Extensions.AutoTimer.plugin import autotimer
				if autotimer is None:
					from Plugins.Extensions.AutoTimer.AutoTimer import AutoTimer
					autotimer = AutoTimer()
				autotimer.parseEPGAsync(simulateOnly=False)
				print>>log, "[EPGImport] Run start parse AutoTimer"
				checkDeepstandby(_session, parse=True)
			except:
				print>>log, "[EPGImport] Could not start AutoTimer"
				checkDeepstandby(_session, parse=False)
		else:
			checkDeepstandby(_session, parse=False)

class checkDeepstandby:
	def __init__(self, session, parse=False):
		self.session = session
		self.parse = parse
		self.count = 0
		self.FirstwaitCheck = None
		if self.parse:
			self.FirstwaitCheck = enigma.eTimer()
			self.FirstwaitCheck.callback.append(self.runCheckDeepstandby)
			self.FirstwaitCheck.startLongTimer(40) #!!!!!!!!!!!!!!!!
			print>>log, "[EPGImport] Wait for parse AutoTimer 40 sec"
		else:
			self.runCheckDeepstandby()

	def runCheckDeepstandby(self):
		print>>log, "[EPGImport] Run check deep standby after import"

If self.FirstwaitCheck.startLongTimer delay > 50 sec,def runCheckDeepstandby never executed

Edited by Dima73, 14 September 2013 - 15:20.

[MiLo]

A couple of years ago, I replaced the fork in eConsoleAppContainer by a vfork, just to avoid these issues.

Hmm, couldn't we also do the same modification in Python then?

[MiLo]

I've also thought about replacing os.system with something that throws an exception kindly asking the user to never call this method and use eConsoleApp instead.

[radxnl]

I've also thought about replacing os.system with something that throws an exception kindly asking the user to never call this method and use eConsoleApp instead.

+1

[pieterg]

A couple of years ago, I replaced the fork in eConsoleAppContainer by a vfork, just to avoid these issues.

Hmm, couldn't we also do the same modification in Python then?

Yes, probably. But things like os.system block the mainloop, so people should still use eConsoleApp instead...

[gorski]

Sounds like quite a conundrum, sadly...

 

I know a guy who is now working on this - any more pointers, please?

 

This seems to be a really interesting point, if one were to crack it and improve the whole of E2, in the process...

 

Thanx!

[athoik]

What if we use posix_spawn ?

http://stackoverflow...cesses-on-linux
http://twistedmatrix...rac/ticket/5710

Or turn on overcommit?

http://bryanmarty.co...14/forking-jvm/

[MiLo]

overcommit is already enabled.

 

Just don't use os.system. It's really really really bad because it blocks the GUI thread. And 99% of the silly things that people use os.system for (e.g. renaming files) can be done perfectly well from within Python without spending all that effort on forking a new process.

[Dimitrij]

Ver. 1.5
-add option "Load EPG only services in bouquets"(yes/no)