10 replies to this topic

[DimitarCC]

So i noticed OpenPLi version of EPGImport plugin greatly differs from OE-A one....

But there is a problem in it that prevent downloading epg if the url is like

https://epg.url.com

From that is downloaded file like epg.xml.gz. but the filename is not in the url so EPGImport treat it as invalid.

The filename have to be gotten from the content-disposition header.

I have fixed the OE-A version but since Pli one differs too much can someone (that is more in deep in it) look at that issue? Thanks!

[Abu Baniaz]

Now that OE-A images no longer use the branding module so prevalently, maybe they can be better aligned.

[DimitarCC]

Well if someone willing to merge/align it....

Is not little works though...

[Dimitrij]

	def nextImport(self):
		self.closeReader()
		if not self.sources:
			self.closeImport()
			return
		self.source = self.sources.pop()
		print("[EPGImport] nextImport, source=", self.source.description, file=log)
		self.fetchUrl(self.source.url)

	def fetchUrl(self, filename):
		if filename.startswith('http:') or filename.startswith('https:') or filename.startswith('ftp:'):
			self.do_download(filename, self.afterDownload, self.downloadFail)
		else:
			self.afterDownload(None, filename, deleteFile=False)

	def do_download(self, sourcefile, afterDownload, downloadFail):
		path = bigStorage(9000000, '/tmp', '/media/cf', '/media/mmc', '/media/usb', '/media/hdd')
		filename = os.path.join(path, 'epgimport')
		ext = os.path.splitext(sourcefile)[1]
		# Keep sensible extension, in particular the compression type
		if ext and len(ext) < 6:
			filename += ext
		sourcefile = sourcefile.encode('utf-8')
		sslcf = SNIFactory(sourcefile) if sourcefile.decode().startswith('https:') else None
		print("[EPGImport] Downloading: " + sourcefile.decode() + " to local path: " + filename, file=log)
		if self.source.nocheck == 1:
			print("[EPGImport] Not cheching the server since nocheck is set for it: " + sourcefile.decode(), file=log)
			downloadPage(sourcefile, filename, timeout=90, contextFactory=sslcf).addCallbacks(afterDownload, downloadFail, callbackArgs=(filename, True))
			return filename
		else:
			if self.checkValidServer(sourcefile) == 1:
				downloadPage(sourcefile, filename, timeout=90, contextFactory=sslcf).addCallbacks(afterDownload, downloadFail, callbackArgs=(filename, True))
				return filename
			else:
				self.downloadFail("checkValidServer reject the server")

	def afterDownload(self, result, filename, deleteFile=False):
		print("[EPGImport] afterDownload", filename, file=log)
		if not os.path.exists(filename):
			self.downloadFail("File not exists")
			return
		try:
			if not os.path.getsize(filename):
				raise Exception("File is empty")
		except Exception as e:
			self.downloadFail(e)
			return
		if self.source.parser == 'epg.dat':
			if twisted.python.runtime.platform.supportsThreads():
				print("[EPGImport] Using twisted thread for DAT file", file=log)
				threads.deferToThread(self.readEpgDatFile, filename, deleteFile).addCallback(lambda ignore: self.nextImport())
			else:
				self.readEpgDatFile(filename, deleteFile)
				return
		if filename.endswith('.gz'):
			self.fd = gzip.open(filename, 'rb')
			try:
				# read a bit to make sure it's a gzip file
				self.fd.read(10)
				self.fd.seek(0, 0)
			except Exception as e:
				print("[EPGImport] File downloaded is not a valid gzip file", filename, file=log)
				self.downloadFail(e)
				return
		elif filename.endswith('.xz') or filename.endswith('.lzma'):
			try:
				import lzma
			except ImportError:
				from backports import lzma
			self.fd = lzma.open(filename, 'rb')
			try:
				# read a bit to make sure it's an xz file
				self.fd.read(10)
				self.fd.seek(0, 0)
			except Exception as e:
				print("[EPGImport] File downloaded is not a valid xz file", filename, file=log)
				self.downloadFail(e)
				return
		else:
			self.fd = open(filename, 'rb')
		if deleteFile and self.source.parser != 'epg.dat':
			try:
				print("[EPGImport] unlink", filename, file=log)
				os.unlink(filename)
			except Exception as e:
				print("[EPGImport] warning: Could not remove '%s' intermediate" % filename, e, file=log)
		self.channelFiles = self.source.channels.downloadables()
		if not self.channelFiles:
			self.afterChannelDownload(None, None)
		else:
			filename = random.choice(self.channelFiles)
			self.channelFiles.remove(filename)
			self.do_download(filename, self.afterChannelDownload, self.channelDownloadFail)

	def afterChannelDownload(self, result, filename, deleteFile=True):
		print("[EPGImport] afterChannelDownload", filename, file=log)
		if filename:
			try:
				if not os.path.getsize(filename):
					raise Exception("File is empty")
			except Exception as e:
				self.channelDownloadFail(e)
				return
		if twisted.python.runtime.platform.supportsThreads():
			print("[EPGImport] Using twisted thread", file=log)
			threads.deferToThread(self.doThreadRead, filename).addCallback(lambda ignore: self.nextImport())
			deleteFile = False # Thread will delete it
		else:
			self.iterator = self.createIterator(filename)
			reactor.addReader(self)
		if deleteFile and filename:
			try:
				os.unlink(filename)
			except Exception as e:
				print("[EPGImport] warning: Could not remove '%s' intermediate" % filename, e, file=log)

def afterDownload(self, result, filename, deleteFile=False):

result  What is this?

Edited by Dimitrij, Today, 12:15.

[DimitarCC]

@Dimitrij ? 

The problem is the filename is not known initially. It needs to be gotten from content-disposition....

I dont see that in the code you have posted...

 

probably is the result...I dont know....As i said there is too much difference from OE-A

Edited by DimitarCC, Today, 12:18.

[Pr2]

You should perhaps read this thread:

 

https://forums.openp...es-explanation/

 

In fact Doglover has sometimes issues with some mirrors that he cannot update anymore but the mirrors still provide the EPG files. So people were downloading EPG which were outdated leading to empty EPG and start complaining so the solution was to temporarily exclude the server by editing the rytec.sources.xml.

 

To workaround this a "LastUpdate.txt" is created when the upload is successfull this file is checked before downloading the EPG, by default is check is turned on in OpenPLi version.

 

OE-A decide to not implement this and/or to change the default for the check to off. So you need to adjust your xxxx.sources.xml files with this:

 

example: <source type="gen_xmltv" nocheck="1" channels="http://epg.in.ua/epg...annels.xml.gz">

 

nocheck is not checking for LastUpdate.txt

 

I guess that it's the problem that you encountered.

[DimitarCC]

Thats not the issue....The detection of the filename is done based on the url...But for urls that dont contain filename in it it fails to detect what is the archive type (gz,zip,xml) so it cancel the download.

[Pr2]

How to you want the EPGImporter to know which file to download if you don't provide its name?

I think that the filename to download is a mandatory parameter.

[DimitarCC]

No. The filename is in http response headers... But epgimport doesnt take it currently...

[Dimitrij]

No. The filename is in http response headers... But epgimport doesnt take it currently...

Give me a working example of a URL, I'll see how it works.

[DimitarCC]

I have sent you PM