36 replies to this topic

[mfaraj57]

still no success with twisted 15 in windows10 and python 2.7

but still need important dependency service_identiy for valid certificate to confirm not working

 

Edited by mfaraj57, 1 October 2017 - 22:03.

[mfaraj57]

After further testing and  downloading images by twisted web i discovered that not all https have problem
as example this image downloaded successfully without disabling certificate

https://www.vidics.to/static/90/1/Casper-12641.JPEG

as follow

def downloadimage(self, localfile, webfile, index):
        image_url='https://www.vidics.to/static/90/1/Casper-12641.JPEG'
        webfile=image_url
        print "image_url--twested",image_url
        print "ssl verivication enabled--twisted"
        from twisted.web import client        
        client.downloadPage(webfile, localfile).addCallback(self.downloadback, index, localfile).addErrback(self.downloaderror, index)

11:20:10.3652 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:733 downloadimage image_url--twsied https://www.vidics.to/static/90/1/Casper-12641.JPEG
11:20:10.3656 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:734 downloadimage ssl verivication enabled--twsited
11:20:11.8874 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:740 downloadback success-twisteded 

but still ssl.handshake problem for all images from same sever

def downloadimage(self, localfile, webfile, index):
        image_url='https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg'
        webfile=image_url
        print "image_url--twsied",image_url
        print "ssl verivication enabled--twsited"
        from twisted.internet import _sslverify
        _sslverify.platformTrust = lambda : None 
        
        from twisted.web import client
        
        client.downloadPage(webfile, localfile).addCallback(self.downloadback, index, localfile).addErrback(self.downloaderror, index)
11:39:35.6861 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:734 downloadimage ssl verivication enabled--twsited
11:39:35.6941 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:733 downloadimage image_url--twsied https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg
11:39:35.6946 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:734 downloadimage ssl verivication enabled--twsited
libpng warning: iCCP: known incorrect sRGB profile
11:39:36.0782 { E } Plugins/Extensions/TSmedia/screens/addonshow.py:748 downloaderror error-twsied [Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]
/usr/lib/enigma2/python/e2reactor.py:184:_doReadOrWrite

Now the problem is limited to some secure links and not to all while these can be downloaded by other methods like requests we consider it is bug in twisted web in managing handshake error,at least with version 13 which used with our stbs.

Now upgrading to 14 or to latest 17 could solve this minor problem but things will be not easyif possible as i discovered from trying to install version 17 of twisted web ,that it is different,more complex, a lot of features mostly not required to stb,s as well as a series of dependent files some of them will not be suitable for stb,s.

Edited by mfaraj57, 2 October 2017 - 14:06.

[mfaraj57]

seem the ssl-handshake error is common and find failure from other servers

webfile https://images.beinsports.com/ET-AeOHhPzFkeIJfq8oS4s5lbY8=/410x260/smart/harrykane-cropped_ama3i49599ly1r0yxio9ds5xm.jpg
19:05:33.8940 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:709 newPage localfile /tmp/TSmedia/harrykane-cropped_ama3i49599ly1r0yxio9ds5xm.jpg
19:05:34.3735 { E } Plugins/Extensions/TSmedia/screens/addonshow.py:742 downloaderror  [Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]

[athoik]

The SNI is very common today, and will be adopted by all in few years.

I guess we should update manually twisted in develop (or pyro).

[WanWizard]

SNI is used by virtually all webservers today, nobody has IP's per webserver spare, or is willing to use them up for such an excuse.

[ozzsurf]

can you edit twsied to add

curl -k

image_url='curl -k https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg'

or

curl -k -O -J 

[ozzsurf]

this works for me keeping the same file name

curl -k -O -J  'https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg'

[mfaraj57]

Where work with you and how you apply it?

Edited by mfaraj57, 2 October 2017 - 19:04.

[mfaraj57]

The problem you want to solve needs external library pycurl to apply through python but requests method is already used and very efficent.
I am not the man to talk about twisted web but twisted web has special internal process to fetch urls and introducing other methods need building twisted from zero.

[athoik]

import os
from OpenSSL import SSL
from twisted.web import client
from twisted.internet import ssl, reactor
from twisted.internet._sslverify import ClientTLSOptions

class SNIFactory(ssl.ClientContextFactory):
    def __init__(self, hostname = None):
        self.hostname = hostname

    def getContext(self):
        ctx = self._contextFactory(self.method)
        if self.hostname:
            ClientTLSOptions(self.hostname, ctx)
        return ctx

image_url="https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg"
localfile="/tmp/image.jpg"

def downloadback(data):
        print 'download success'
        reactor.stop()

def downloaderror(data):
        print "error_data",data
        reactor.stop()

sniFactory = SNIFactory('www.almstba.tv')
print "downloading image by twisted web..."
client.downloadPage(image_url, localfile, sniFactory).addCallback(downloadback).addErrback(downloaderror)
reactor.run()
print localfile, "exists", os.path.exists(localfile)

Above works for me...

# python testW.py
downloading image by twisted web...
download success
/tmp/image.jpg exists True
# md5sum /tmp/image.jpg
754448d51d9faf113e7ee0037af5b8b4  /tmp/image.jpg

In order to get there I had to:

python-attrs_17.2.0-r0_mips32el.ipk
python-constantly_15.1.0-r0_mips32el.ipk
python-incremental_17.5.0-r0_mips32el.ipk
python-pyasn1-modules_0.1.4-r0_mips32el.ipk
python-pyasn1_0.3.6-r0_mips32el.ipk
python-service-identity_17.0.0-r0_mips32el.ipk
python-twisted-core_17.9.0-r0_mips32el.ipk
python-twisted-protocols_17.9.0-r0_mips32el.ipk
python-twisted-web_17.9.0-r0_mips32el.ipk

Update recipe for twisted to 17.9.0
Create recipe for incremental
Create recipe for constantly
Create recipe for service_identity
Create recipe for attrs
Update recipe for pyasn1 to 0.3.6 (maybe not required, becase later I notice that missing depend was on pyasn1-modules)
Create recipe for pyasn1-modules

And the most important!!! Because downloadPage is obsolete (or at least that is what I understand), I had to create a custom SNI Factory.

Using httpclient (https://github.com/r...s/httpclient.py) worked without modifications.

[mfaraj57]

Good athoic and exceĺlent result
Then twisted 17 and custom sni factory solved the problem.
Going to test tomorrow with other links.

[ims]

does exist some solution for twisted and https yet ?

 

I must changed downloading my queue with filepaths (cca 12-200 files) as 'single file' downloading with requests... it is slow ...

 

 

 

[littlesat]

Yep it is called rocko or sumo .... or use urllib(2) instead....

[ims]

Speed for multple files downloading with urllib2 is similar as for requests.get(url) ... both is for nothing ...

[littlesat]

oe-a (openatv) made a patch with a newer twistedweb in 2016... but they never 'pushed' it to openembeded... They just forked and then patched their openembeded environment.

Now rocko and sumo have even a more modern twistedweb... So we must have some patience here...

(see also the threads about serienstreamer plugin)

Edited by littlesat, 17 July 2018 - 14:17.

[Erik Slagter]

So you (both) know what to do

[littlesat]

I suggest wait for sumo.... then we get if for free without ugly patches and time invested to create the ugly patch... others seem to go for...