Jump to content


Photo

Twisted web failed to download https image


  • Please log in to reply
36 replies to this topic

Re: Twisted web failed to download https image #21 mfaraj57

  • Senior Member
  • 1,557 posts

+279
Excellent

Posted 1 October 2017 - 22:02

still no success with twisted 15 in windows10 and python 2.7

but still need important dependency service_identiy for valid certificate to confirm not working

 

screenshot_41.png


Edited by mfaraj57, 1 October 2017 - 22:03.


Re: Twisted web failed to download https image #22 mfaraj57

  • Senior Member
  • 1,557 posts

+279
Excellent

Posted 2 October 2017 - 14:05

After further testing and  downloading images by twisted web i discovered that not all https have problem
as example this image downloaded successfully without disabling certificate

https://www.vidics.to/static/90/1/Casper-12641.JPEG

as follow

def downloadimage(self, localfile, webfile, index):
        image_url='https://www.vidics.to/static/90/1/Casper-12641.JPEG'
        webfile=image_url
        print "image_url--twested",image_url
        print "ssl verivication enabled--twisted"
        from twisted.web import client        
        client.downloadPage(webfile, localfile).addCallback(self.downloadback, index, localfile).addErrback(self.downloaderror, index)

11:20:10.3652 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:733 downloadimage image_url--twsied https://www.vidics.to/static/90/1/Casper-12641.JPEG
11:20:10.3656 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:734 downloadimage ssl verivication enabled--twsited
11:20:11.8874 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:740 downloadback success-twisteded 

but still ssl.handshake problem for all images from same sever

def downloadimage(self, localfile, webfile, index):
        image_url='https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg'
        webfile=image_url
        print "image_url--twsied",image_url
        print "ssl verivication enabled--twsited"
        from twisted.internet import _sslverify
        _sslverify.platformTrust = lambda : None 
        
        from twisted.web import client
        
        client.downloadPage(webfile, localfile).addCallback(self.downloadback, index, localfile).addErrback(self.downloaderror, index)
11:39:35.6861 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:734 downloadimage ssl verivication enabled--twsited
11:39:35.6941 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:733 downloadimage image_url--twsied https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg
11:39:35.6946 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:734 downloadimage ssl verivication enabled--twsited
libpng warning: iCCP: known incorrect sRGB profile
11:39:36.0782 { E } Plugins/Extensions/TSmedia/screens/addonshow.py:748 downloaderror error-twsied [Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]
/usr/lib/enigma2/python/e2reactor.py:184:_doReadOrWrite

Now the problem is limited to some secure links and not to all while these can be downloaded by other methods like requests we consider it is bug in twisted web in managing handshake error,at least with version 13 which used with our stbs.

Now upgrading to 14 or to latest 17 could solve this minor problem but things will be not easyif possible as i discovered from trying to install version 17 of twisted web ,that it is different,more complex, a lot of features mostly not required to stb,s as well as a series of dependent files some of them will not be suitable for stb,s.


Edited by mfaraj57, 2 October 2017 - 14:06.


Re: Twisted web failed to download https image #23 mfaraj57

  • Senior Member
  • 1,557 posts

+279
Excellent

Posted 2 October 2017 - 18:10

seem the ssl-handshake error is common and find failure from other servers
webfile https://images.beinsports.com/ET-AeOHhPzFkeIJfq8oS4s5lbY8=/410x260/smart/harrykane-cropped_ama3i49599ly1r0yxio9ds5xm.jpg
19:05:33.8940 {   } Plugins/Extensions/TSmedia/screens/addonshow.py:709 newPage localfile /tmp/TSmedia/harrykane-cropped_ama3i49599ly1r0yxio9ds5xm.jpg
19:05:34.3735 { E } Plugins/Extensions/TSmedia/screens/addonshow.py:742 downloaderror  [Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]



Re: Twisted web failed to download https image #24 athoik

  • PLi® Core member
  • 7,922 posts

+294
Excellent

Posted 2 October 2017 - 18:16

The SNI is very common today, and will be adopted by all in few years.

I guess we should update manually twisted in develop (or pyro).
Wavefield T90: 0.8W - 1.9E - 4.8E - 13E - 16E - 19.2E - 23.5E - 26E - 33E - 39E - 42E - 45E on EMP Centauri DiseqC 16/1
Unamed: 13E Quattro - 9E Quattro on IKUSI MS-0916

Re: Twisted web failed to download https image #25 WanWizard

  • Forum Moderator
    PLi® Core member
  • 48,468 posts

+824
Excellent

Posted 2 October 2017 - 18:27

SNI is used by virtually all webservers today, nobody has IP's per webserver spare, or is willing to use them up for such an excuse.


Currently in use: VU+Duo 4K (2xFBC S2), Amiko Viper T2C (T2), SAB Alpha Triple HD (S2+T2), Zgemma H3.T2C (T/C), Zgemma H6 (fallback), VU+Zero (fallback)

Many answers to your question can be found in our new and improved wiki.

note: I do not provide support via PM !

 


Re: Twisted web failed to download https image #26 ozzsurf

  • Senior Member
  • 119 posts

+3
Neutral

Posted 2 October 2017 - 18:31

can you edit twsied to add

curl -k
image_url='curl -k https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg'

or

curl -k -O -J 


Re: Twisted web failed to download https image #27 ozzsurf

  • Senior Member
  • 119 posts

+3
Neutral

Posted 2 October 2017 - 18:45

this works for me keeping the same file name

curl -k -O -J  'https://www.almstba.tv/video/uploads/thumbs/936acfcba-1.jpg'


Re: Twisted web failed to download https image #28 mfaraj57

  • Senior Member
  • 1,557 posts

+279
Excellent

Posted 2 October 2017 - 19:03

Where work with you and how you apply it?

Edited by mfaraj57, 2 October 2017 - 19:04.


Re: Twisted web failed to download https image #29 mfaraj57

  • Senior Member
  • 1,557 posts

+279
Excellent

Posted 2 October 2017 - 19:49

The problem you want to solve needs external library pycurl to apply through python but requests method is already used and very efficent.
I am not the man to talk about twisted web but twisted web has special internal process to fetch urls and introducing other methods need building twisted from zero.

Re: Twisted web failed to download https image #30 athoik

  • PLi® Core member
  • 7,922 posts

+294
Excellent

Posted 2 October 2017 - 21:19

import os
from OpenSSL import SSL
from twisted.web import client
from twisted.internet import ssl, reactor
from twisted.internet._sslverify import ClientTLSOptions

class SNIFactory(ssl.ClientContextFactory):
    def __init__(self, hostname = None):
        self.hostname = hostname

    def getContext(self):
        ctx = self._contextFactory(self.method)
        if self.hostname:
            ClientTLSOptions(self.hostname, ctx)
        return ctx

image_url="https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg"
localfile="/tmp/image.jpg"

def downloadback(data):
        print 'download success'
        reactor.stop()

def downloaderror(data):
        print "error_data",data
        reactor.stop()

sniFactory = SNIFactory('www.almstba.tv')
print "downloading image by twisted web..."
client.downloadPage(image_url, localfile, sniFactory).addCallback(downloadback).addErrback(downloaderror)
reactor.run()
print localfile, "exists", os.path.exists(localfile)
Above works for me...

# python testW.py
downloading image by twisted web...
download success
/tmp/image.jpg exists True
# md5sum /tmp/image.jpg
754448d51d9faf113e7ee0037af5b8b4  /tmp/image.jpg
In order to get there I had to:


python-attrs_17.2.0-r0_mips32el.ipk
python-constantly_15.1.0-r0_mips32el.ipk
python-incremental_17.5.0-r0_mips32el.ipk
python-pyasn1-modules_0.1.4-r0_mips32el.ipk
python-pyasn1_0.3.6-r0_mips32el.ipk
python-service-identity_17.0.0-r0_mips32el.ipk
python-twisted-core_17.9.0-r0_mips32el.ipk
python-twisted-protocols_17.9.0-r0_mips32el.ipk
python-twisted-web_17.9.0-r0_mips32el.ipk
Update recipe for twisted to 17.9.0
Create recipe for incremental
Create recipe for constantly
Create recipe for service_identity
Create recipe for attrs
Update recipe for pyasn1 to 0.3.6 (maybe not required, becase later I notice that missing depend was on pyasn1-modules)
Create recipe for pyasn1-modules

And the most important!!! Because downloadPage is obsolete (or at least that is what I understand), I had to create a custom SNI Factory.

Using httpclient (https://github.com/r...s/httpclient.py) worked without modifications.
Wavefield T90: 0.8W - 1.9E - 4.8E - 13E - 16E - 19.2E - 23.5E - 26E - 33E - 39E - 42E - 45E on EMP Centauri DiseqC 16/1
Unamed: 13E Quattro - 9E Quattro on IKUSI MS-0916

Re: Twisted web failed to download https image #31 mfaraj57

  • Senior Member
  • 1,557 posts

+279
Excellent

Posted 2 October 2017 - 22:15

Good athoic and exceĺlent result
Then twisted 17 and custom sni factory solved the problem.
Going to test tomorrow with other links.

Re: Twisted web failed to download https image #32 ims

  • PLi® Core member
  • 11,135 posts

+168
Excellent

Posted 17 July 2018 - 13:10

does exist some solution for twisted and https yet ?

 

I must changed downloading my queue with filepaths (cca 12-200 files) as 'single file' downloading with requests... it is slow ...

 
 
 


Kdo nic nedělá, nic nezkazí!

Re: Twisted web failed to download https image #33 littlesat

  • PLi® Core member
  • 49,224 posts

+527
Excellent

Posted 17 July 2018 - 13:28

Yep it is called rocko or sumo ;).... or use urllib(2) instead....


WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Twisted web failed to download https image #34 ims

  • PLi® Core member
  • 11,135 posts

+168
Excellent

Posted 17 July 2018 - 14:03

Speed for multple files downloading with urllib2 is similar as for requests.get(url) ... both is for nothing ...


Kdo nic nedělá, nic nezkazí!

Re: Twisted web failed to download https image #35 littlesat

  • PLi® Core member
  • 49,224 posts

+527
Excellent

Posted 17 July 2018 - 14:15

oe-a (openatv) made a patch with a newer twistedweb in 2016... but they never 'pushed' it to openembeded... They just forked and then patched their openembeded environment.

Now rocko and sumo have even a more modern twistedweb... So we must have some patience here...

(see also the threads about serienstreamer plugin)


Edited by littlesat, 17 July 2018 - 14:17.

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: Twisted web failed to download https image #36 Erik Slagter

  • PLi® Core member
  • 45,353 posts

+498
Excellent

Posted 17 July 2018 - 19:50

So you (both) know what to do ;)


* Wavefrontier T90 with 28E/23E/19E/13E/9E/4.8E/0.8W/5W via SCR switches 2 x 2 x 6 user bands
* Ziggo digital cable TV (FTA)
I don't read PM -> if you have something to ask or to report, do it in the forum so others can benefit. I don't take freelance jobs.
Ik lees geen PM -> als je iets te vragen of te melden hebt, doe het op het forum, zodat anderen er ook wat aan hebben.

Re: Twisted web failed to download https image #37 littlesat

  • PLi® Core member
  • 49,224 posts

+527
Excellent

Posted 18 July 2018 - 07:07

I suggest wait for sumo.... then we get if for free without ugly patches and time invested to create the ugly patch... others seem to go for...

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W





0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users