Jump to content


Photo

Twisted web failed to download https image


  • Please log in to reply
36 replies to this topic

#1 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 30 September 2017 - 06:28

I am using box with python 2.7.13 and openssl 1.0.2k and got these errors when downloading https images using twisted web.

I do not have problem in downloading https by other methods but need twisted web because i want to download images simultaneously in background without stopping code execution. 

 

 

image url example   

https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg
/usr/lib/enigma2/python/e2reactor.py:184:_doReadOrWrite
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:208:doRead
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:214:_dataReceived
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:415:dataReceived
--- <exception caught here> ---
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:335:_checkHandshakeStatus
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1426:do_handshake
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1174:_raise_ssl_error
/usr/lib/python2.7/site-packages/OpenSSL/_util.py:48:exception_from_error_queue


Edited by mfaraj57, 30 September 2017 - 06:29.


Re: Twisted web failed to download https image #2 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 30 September 2017 - 06:33

This is description for error

Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]
/usr/lib/enigma2/python/e2reactor.py:184:_doReadOrWrite



Re: Twisted web failed to download https image #3 WanWizard

  • Forum Moderator
    PLi® Core member
  • 41,663 posts

+665
Excellent

Posted 30 September 2017 - 13:13

Server has a self-signed certificate? ca-certificates not installed? ca-certificates not up to date?

 

If this is OpenPLi 4, it's SSL implementation is broken, and quite a bit of work to fix it, which we won't do anymore. SSL issues are fixed in 6.0.


Many answers to your question can be found in our new and improved wiki.

Currently in active use: VU+Solo 4K (1xFBC, 2xS2), VU+Zero, Edision OS mini+, Amiko Viper 2TC, Zgemma H3.2TC, Zgemma H6

For testing purposes: XP1000, Formuler F1 (2xS2), Miraclebox Premium Micro (S2+C/T), ET7500 (S2), ET8500 (S2), Zgemma H2.H (S2+C), Zgemma H5.2TC, SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, HD2400 (4xS2), ET10000 (4xS2), VU+Duo2 (1xS2), Edision OS nino


Re: Twisted web failed to download https image #4 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 30 September 2017 - 13:55

It is not openpli 4.0,it is openatv 6.1 with python 2.7.13,will test in openpli 6.0 but expect same error,it is openssl porblem

Re: Twisted web failed to download https image #5 WanWizard

  • Forum Moderator
    PLi® Core member
  • 41,663 posts

+665
Excellent

Posted 30 September 2017 - 15:54

OpenPli 6 has 2.7.13 as well. I suspect it is a server side issue, not a client side issue.


Many answers to your question can be found in our new and improved wiki.

Currently in active use: VU+Solo 4K (1xFBC, 2xS2), VU+Zero, Edision OS mini+, Amiko Viper 2TC, Zgemma H3.2TC, Zgemma H6

For testing purposes: XP1000, Formuler F1 (2xS2), Miraclebox Premium Micro (S2+C/T), ET7500 (S2), ET8500 (S2), Zgemma H2.H (S2+C), Zgemma H5.2TC, SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, HD2400 (4xS2), ET10000 (4xS2), VU+Duo2 (1xS2), Edision OS nino


Re: Twisted web failed to download https image #6 athoik

  • PLi® Core member
  • 7,415 posts

+261
Excellent

Posted 30 September 2017 - 16:41

No problem here using python-requests ...

Python 2.7.12 (default, Jul 12 2017, 23:36:04)
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> res = requests.get("https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg")
>>> res
<Response [200]>
>>> len(res.text)
18001

Wavefield T90: 0.8W - 1.9E - 4.8E - 13E - 16E - 19.2E - 23.5E - 26E - 33E - 39E - 42E - 45E on EMP Centauri DiseqC 16/1
Unamed: 13E Quattro - 9E Quattro on IKUSI MS-0916

Re: Twisted web failed to download https image #7 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 30 September 2017 - 18:19

yes,no problem with requests in my box and no problem with urllib2 or other methods

i can not use requests because i want to download at least 15 images and difficult to stop the code execution until download all images completed,while twisted web will download images in the background without stopping the code execution.

 

this is my code for requests and twisted web and the response

import os
from twisted.web import client
import time
image_url="https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg"
localfile="/tmp/image.jpg"
def getimage():
    import requests
    res = requests.get(image_url)
    print "res", res
    print "len",len(res.text)

getimage()
def downloadback(data):
        print "data",data
        print 'download success'

def downloaderror(data):
        print "error_data",data    

print "downloading image by twisted web..."
client.downloadPage(image_url, localfile).addCallback(downloadback).addErrback(downloaderror)
time.sleep(30)
print localfile +" exists",os.path.exists(localfile)

screenshot_39.png


Edited by mfaraj57, 30 September 2017 - 18:20.


Re: Twisted web failed to download https image #8 MiLo

  • PLi® Core member
  • 13,796 posts

+289
Excellent

Posted 30 September 2017 - 19:18

Maybe twisted wants you to supply a callback for certificate verification?

A quick google on the topic reveals this:
https://twistedmatri.../howto/ssl.html
Real musicians never die - they just decompose

Re: Twisted web failed to download https image #9 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 30 September 2017 - 21:59

Thanks for the link

according to the link documentation and others ,they make intentionally difficult to disable twisted certificate but some places claimed this code will disable certificate verification

from twisted.internet import _sslverify
_sslverify.platformTrust = lambda : None 

but i do not get success with it ,may need more testing and applying through real code.



Re: Twisted web failed to download https image #10 MiLo

  • PLi® Core member
  • 13,796 posts

+289
Excellent

Posted 1 October 2017 - 09:53

If it works, the problem when doing that is that it applies globally to all of enigma. It means you basically kill all https security on the box, since without certificate validation, all the other SSL features like encryption become pointless.
Real musicians never die - they just decompose

Re: Twisted web failed to download https image #11 athoik

  • PLi® Core member
  • 7,415 posts

+261
Excellent

Posted 1 October 2017 - 10:32

It's not validation it's the negotiation that fails!

I made some changes to test code posted, but didn't find the right parameters (chiphers) to support the negotiation.

Most probably need to examine the negotiation using wireshark.

Will post modified test later.

Edited by athoik, 1 October 2017 - 10:33.

Wavefield T90: 0.8W - 1.9E - 4.8E - 13E - 16E - 19.2E - 23.5E - 26E - 33E - 39E - 42E - 45E on EMP Centauri DiseqC 16/1
Unamed: 13E Quattro - 9E Quattro on IKUSI MS-0916

Re: Twisted web failed to download https image #12 WanWizard

  • Forum Moderator
    PLi® Core member
  • 41,663 posts

+665
Excellent

Posted 1 October 2017 - 11:04

Hence my remark that the issue is more server side, not client side.


Many answers to your question can be found in our new and improved wiki.

Currently in active use: VU+Solo 4K (1xFBC, 2xS2), VU+Zero, Edision OS mini+, Amiko Viper 2TC, Zgemma H3.2TC, Zgemma H6

For testing purposes: XP1000, Formuler F1 (2xS2), Miraclebox Premium Micro (S2+C/T), ET7500 (S2), ET8500 (S2), Zgemma H2.H (S2+C), Zgemma H5.2TC, SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, HD2400 (4xS2), ET10000 (4xS2), VU+Duo2 (1xS2), Edision OS nino


Re: Twisted web failed to download https image #13 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 1 October 2017 - 11:35

If it is server side,why no problem in getting image by requests method?

Re: Twisted web failed to download https image #14 WanWizard

  • Forum Moderator
    PLi® Core member
  • 41,663 posts

+665
Excellent

Posted 1 October 2017 - 13:45

To be more complete, it is a combination.

 

For security reasons, this server has both SSLv2 and SSLv3 disabled, and only allows TLS (all versions, see https://www.ssllabs....v&s=104.28.2.40). Which explains the " sslv3 alert handshake failure", it doesn't do SSLv3.

 

So you need to convince twisted to use TLS if SSL fails (don't ask me how, I'm a network guy, not a python guy ;)).


Many answers to your question can be found in our new and improved wiki.

Currently in active use: VU+Solo 4K (1xFBC, 2xS2), VU+Zero, Edision OS mini+, Amiko Viper 2TC, Zgemma H3.2TC, Zgemma H6

For testing purposes: XP1000, Formuler F1 (2xS2), Miraclebox Premium Micro (S2+C/T), ET7500 (S2), ET8500 (S2), Zgemma H2.H (S2+C), Zgemma H5.2TC, SAB TripleAlpha (S2+C/T), Galaxy 4K (FBC), VU Zero 4K, HD2400 (4xS2), ET10000 (4xS2), VU+Duo2 (1xS2), Edision OS nino


Re: Twisted web failed to download https image #15 athoik

  • PLi® Core member
  • 7,415 posts

+261
Excellent

Posted 1 October 2017 - 13:56

The problem once again is the famous SNI ;)

from socket import socket
from OpenSSL.SSL import TLSv1_METHOD, Context, Connection

def main():
    client = socket()
    print 'Connecting...',
    client.connect(("www.almstba.tv", 443))
    print 'connected', client.getpeername()

    client_ssl = Connection(Context(TLSv1_METHOD), client)
    client_ssl.set_connect_state()
    #client_ssl.set_tlsext_host_name("www.almstba.tv")
    client_ssl.do_handshake()
    print 'Server subject is', client_ssl.get_peer_certificate().get_subject()
    client_ssl.close()

if __name__ == '__main__':
        main()

If you try above with or without set_tlsext_host_name ... then:

# without SNI
# python client.py
Connecting... connected ('104.28.3.40', 443)
Traceback (most recent call last):
  File "client.py", line 18, in <module>
    main()
  File "client.py", line 13, in main
    client_ssl.do_handshake()
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1424, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1172, in _raise_ssl_error
    _raise_current_error()
  File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]

# With SNI
# python client.py 
Connecting... connected ('104.28.2.40', 443)
Server subject is <X509Name object '/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=sni182069.cloudflaressl.com'>
I am sure that twisted has a flag for SNI...
Wavefield T90: 0.8W - 1.9E - 4.8E - 13E - 16E - 19.2E - 23.5E - 26E - 33E - 39E - 42E - 45E on EMP Centauri DiseqC 16/1
Unamed: 13E Quattro - 9E Quattro on IKUSI MS-0916

Re: Twisted web failed to download https image #16 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 1 October 2017 - 19:29

If it works, the problem when doing that is that it applies globally to all of enigma. It means you basically kill all https security on the box, since without certificate validation, all the other SSL features like encryption become pointless.

In general i agree with you but we are talking about small Linux environment and small box for singe user not for big company with large system,difficult to understand this restriction and increasing complexity to avoid  small risk or not all 



Re: Twisted web failed to download https image #17 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 1 October 2017 - 19:38

To be more complete, it is a combination.

 

For security reasons, this server has both SSLv2 and SSLv3 disabled, and only allows TLS (all versions, see https://www.ssllabs....v&s=104.28.2.40). Which explains the " sslv3 alert handshake failure", it doesn't do SSLv3.

 

So you need to convince twisted to use TLS if SSL fails (don't ask me how, I'm a network guy, not a python guy ;)).

yes,looks reasonable ,not always could explain things by simple logic with absence of complete knowledge.

As example downloading website by urllib2 revealed error while no problem when downloading same link by  requests.get_url .

.Also received from many users error(bad request) when downloading link by urllib2  while no problem with me for same link,same image ,same python version



Re: Twisted web failed to download https image #18 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 1 October 2017 - 19:52

The problem once again is the famous SNI ;)
 

from socket import socket
from OpenSSL.SSL import TLSv1_METHOD, Context, Connection

def main():
    client = socket()
    print 'Connecting...',
    client.connect(("www.almstba.tv", 443))
    print 'connected', client.getpeername()

    client_ssl = Connection(Context(TLSv1_METHOD), client)
    client_ssl.set_connect_state()
    #client_ssl.set_tlsext_host_name("www.almstba.tv")
    client_ssl.do_handshake()
    print 'Server subject is', client_ssl.get_peer_certificate().get_subject()
    client_ssl.close()

if __name__ == '__main__':
        main()
If you try above with or without set_tlsext_host_name ... then:

# without SNI
# python client.py
Connecting... connected ('104.28.3.40', 443)
Traceback (most recent call last):
  File "client.py", line 18, in <module>
    main()
  File "client.py", line 13, in main
    client_ssl.do_handshake()
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1424, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1172, in _raise_ssl_error
    _raise_current_error()
  File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]

# With SNI
# python client.py 
Connecting... connected ('104.28.2.40', 443)
Server subject is <X509Name object '/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=sni182069.cloudflaressl.com'>
I am sure that twisted has a flag for SNI...

 

Thanks athoik,good work

Then we know now it is SNI issue,but we have to find solution for twisted web,i know it is not popular like other methods for downloading links,but no true alternative for it in downloading multiple files at once.

twisted web is complex enough at least for me with a lot of documentation but i am like Wanwizard  "I'm a network guy, not a python guy"  and not having enough skill  and  apart from do not have enough motivation to go very deep in this subject.

time to thanks you and others for your efforts for SNI for opensource images and remember this frequently when every time facing frequent SNI problem or disaster in dreamos system in my dm900with dmm images


Edited by mfaraj57, 1 October 2017 - 19:53.


Re: Twisted web failed to download https image #19 athoik

  • PLi® Core member
  • 7,415 posts

+261
Excellent

Posted 1 October 2017 - 19:58

We need to use twisted >= 14.
Wavefield T90: 0.8W - 1.9E - 4.8E - 13E - 16E - 19.2E - 23.5E - 26E - 33E - 39E - 42E - 45E on EMP Centauri DiseqC 16/1
Unamed: 13E Quattro - 9E Quattro on IKUSI MS-0916

Re: Twisted web failed to download https image #20 mfaraj57

  • Senior Member
  • 1,460 posts

+240
Excellent

Posted 1 October 2017 - 20:27

This is log for downloading the image by requests and twisted web in windows with python 2.7 and twisted web 13.0

also twisted web failed while requests is ok

D:\TSmediaTools\Kodi\plugins\plugin.video.ciraonline\wsyspath.py
wsyspath default
wsyspath default
downloading image by requests
res <Response [200]>
len 19035
downloading image by twisted web...
d:/image.jpg exists False

will try twisted 15 latest available for windows






0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users