38 replies to this topic

[antony23]

Off course the import takes some time. 

But you can use the ready made epg.dat files in the import.  Deselect the the XMLTV sources and scroll down in the list.

The import of the epg.dat files is very fast.  However the import counter does not work, and during the import the receiver is unresponsive.  But this is only for a minute or so (depnding on the amount of sources you import).

 

Willy

 

Why epg.dat import sources are considered "experimental"? What the problem with it?

[doglover]

When trying these out I experienced some crashes.  Could not pinpoint where the error should have been, or even if it was related to these files.

 

Decided to release them anyway.  And if errors showed up, they should have been reported.  Up to now, not much of an error showed up.

Both the symptoms I mentioned before are inherent to the files and method of import and cannot be changed.

 

The situation has not changed since the introduction of this sort of files, and therefore they are still called "experimental".

Also these epg.dat files are sort of a black box.  They either work or they do not.  Not much support I can give on these.

 

BTW: these files are generated with the rytec XMLTV and the rytec channels files and should contain the same information.  Also if channels are missing you should report them in the same way as for the XMLTV files.  The correction has to take place on that level.

 

Willy

 

PS:  read this:  https://forums.openp...g-epgdat-files/

Edited by doglover, 27 December 2016 - 16:04.

[antony23]

Thanks Willy for the explanation.

 

I tried it, had no crashes.
Anyway i exprimented some times where the file was not saved (not donwloaded?). I'm doing more tests.

 

But i have a question... what happens if option "clear epg before import" is turned off? It merges the old file with the new ones, or it replaces the old file anyway?

Edited by antony23, 29 December 2016 - 01:10.

[antony23]

Ok, fast checking code it seems the file in deleted anyway before import. The only difference between turning on or off "clear epg before import" in this case (dat sources) is to flush or not epg in memory.

 

Where i can see the log of EPG Importer plugin to check the reason it is not written sometimes?

[doglover]

Try the log:

 

* Where is the epgimport.log

 

Open the XMLTV importer.

Press now the info or epg button on your remote.

Now you see the log.  This log can be saved, and you find the saved fle in /tmp/

 

Clear epg before import does not seem to work properly when using epg.dat files.  Well it clears the epg in memory but does that before it imports any source.  So if you select more than one source it clears the epg and import the first source, it clears the  epg again and import the next source.  So it loses all imports from the first source.

Of course if you import only one source you will be OK.

 

Willy

[Pr2]

Hi,

 

There is a problem if you enable clear epg before import and importing .dat files is that in the loop, the content is cleared for every file... and the result is that only the latest epg.dat loaded remains in EPG data.  So for the time being it is better to not enable the clear epg before import when you import severals epg.dat files.

 

I have open up a ticket for this.

 

And when I test import epg.dat files it was also working fine for me (and it was fast).

 

Pr2 

[antony23]

Ok, i found a bug saving epg.dat file. 

Anyway i'm checking oe-alliance EPG Importer version source, i don't know it openpli use have same sources.

 

The bug happens when using twisted thread for DAT file.

I try to make a working patch, or create a new issue (where? openpli or oe-alliance?)

 

Thanks

[antony23]

Opened an issue (and relative patch) on oe-alliance/XMLTV-Import. There the line number to correct is 176.

 

In your sources (OpenPLi/enigma2-plugin-extensions-xmltvimport), the line number is 173.

 

Thanks.

[Dimitrij]

maybe

		if self.source.parser == 'epg.dat':
			if twisted.python.runtime.platform.supportsThreads():
				print>>log, "[EPGImport] Using twisted thread for DAT file"
				threads.deferToThread(self.readEpgDatFile, filename, deleteFile).addCallback(lambda ignore: self.nextImport())
			else:
				self.readEpgDatFile(filename, deleteFile)
				return
		if filename.endswith('.gz'):
			self.fd = gzip.open(filename, 'rb')
		else:
			self.fd = open(filename, 'rb')
-		if deleteFile:
+		if deleteFile and self.source.parser != 'epg.dat':
			try:
				print>>log, "[EPGImport] unlink", filename
				os.unlink(filename)
			except Exception, e:
				print>>log, "[EPGImport] warning: Could not remove '%s' intermediate" % filename, e

[antony23]

I simply moved "return" one tab back.

if source is epg.dat, next callback should be ONLY readEpgDatFile (no other code), with or without new thread (not only without new thread as it already is).

 

I tested my patch and all is working good now.

 

Here the fix on oe-alliance git.

Edited by antony23, 4 January 2017 - 00:06.

[antony23]

Before i become crazy, could someone help me in understanding if and when epg.dat file is deleted at enigma2 starting?

 

Could happen that /etc/enigma2/epg.dat file is deleted at eEPGCache::load() here for any reason?

 

I disabled epg save, so there is no possibility epg.dat is deleted or overwrited at enigma2 stop.

Edited by antony23, 5 January 2017 - 00:27.

[WanWizard]

search for unlink() calls.

 

The answer is no, it is only deleted there if it is for example corrupt. Remember, I told you before it is only a backup file, so there is no need to keep it once it is loaded.

[antony23]

search for unlink() calls.

 

The answer is no, it is only deleted there if it is for example corrupt. Remember, I told you before it is only a backup file, so there is no need to keep it once it is loaded.

 

Thanks, i made a stupid test. Put a corrupted eog.dat inside and started enigma in debug mode. The file was deleted because it was corrupt! In fact unlink call is on cacheCorrupt function.

[antony23]

So, if epg.dat downloaded from rytech is corrupted, it is immediately deleted.

 

Other point, maybe @doglover can answer me.

Why epg.dat saved importing xmltv rytech Italy is about 8Mb, and epg.dat downloaded directly from Italy rytech dat file is 15 Mb?

[doglover]

So, if epg.dat downloaded from rytech is corrupted, it is immediately deleted.

 

Other point, maybe @doglover can answer me.

Why epg.dat saved importing xmltv rytech Italy is about 8Mb, and epg.dat downloaded directly from Italy rytech dat file is 15 Mb?

 

 

When you import the epg on an enigma2 receiver, only the channels are imported which you have in your lamedb.  So only the epg is imported for the channels the receiver has access to.  All the others are simply ignored.  When you save this data to an epg.dat file, of course only the imported data is saved.

 

Making epg.dat files off-line you do not not know which channels are available, so all channels are included.  Also all the cable channels for instance.  In some instances this triples the amount of channels.

And probably this is the reason if you try to read these epg.dat files during start-up of enigma2 are failing.  They are meant to be imported by the importer or the BlackHole EPG importer.

 

Willy

Edited by doglover, 5 January 2017 - 08:40.

[littlesat]

I just merged the solution from dimitrij....

[WanWizard]

That doesn't address the issue Doglover is referring to.

[Dimitrij]

Try the log:

 

* Where is the epgimport.log

 

Open the XMLTV importer.

Press now the info or epg button on your remote.

Now you see the log.  This log can be saved, and you find the saved fle in /tmp/

 

Clear epg before import does not seem to work properly when using epg.dat files.  Well it clears the epg in memory but does that before it imports any source.  So if you select more than one source it clears the epg and import the first source, it clears the  epg again and import the next source.  So it loses all imports from the first source.

Of course if you import only one source you will be OK.

 

Willy

I do not understand.
Cleaning is done once before importing.

[antony23]

@Dimitrij

I explain to you. There is no function that merge more than one epg.dat file. Each time epg.dat is uncompressing and saved, it overwrite the precedent one. Check code please (readEpgDatFile function).

 

@doglover

Thanks for epxlanation. Anyway i don't think it is the cause of deleting of epg.dat at startup, because it doesn't happen always. It is hard to catch its deleting. I can't beleave the problem is corruption, because it is the same file downloaded by rytec and already loaded after its download (so file is good) -> i can say this because i disabled saving epg at reboot/restart/shutdown.

 

@littlesat

Is there a reason do you merged solution by Dimitrij and not mine (moreover already committed on oe-core)? I tested mine solution and it works. Someone tested Dimitrij's solution? 
If yuo are not interesting in my fix and bug catch, simply say it. I'm a software developer, not in python, but as you can imagine i can understand what i do.

 

Maybe you and Dimitrij can explain me why calling "self.readEpgDatFile(filename, deleteFile)" should return, and calling "threads.deferToThread(self.readEpgDatFile, filename, deleteFile).addCallback(lambda ignore: self.nextImport())" shoud not return. What is the difference?!

Moreover, not return at that point, could cause problem in any case (even with Dimitrij solution), simply because that part of code is not thread safe....

Edited by antony23, 6 January 2017 - 22:54.