36 replies to this topic

[mfaraj57]

I am using box with python 2.7.13 and openssl 1.0.2k and got these errors when downloading https images using twisted web.

I do not have problem in downloading https by other methods but need twisted web because i want to download images simultaneously in background without stopping code execution. 

 

 

image url example   

https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg

/usr/lib/enigma2/python/e2reactor.py:184:_doReadOrWrite
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:208:doRead
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:214:_dataReceived
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:415:dataReceived
--- <exception caught here> ---
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:335:_checkHandshakeStatus
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1426:do_handshake
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1174:_raise_ssl_error
/usr/lib/python2.7/site-packages/OpenSSL/_util.py:48:exception_from_error_queue

Edited by mfaraj57, 30 September 2017 - 06:29.

[mfaraj57]

This is description for error

Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'sslv3 alert handshake failure')]
/usr/lib/enigma2/python/e2reactor.py:184:_doReadOrWrite

[WanWizard]

Server has a self-signed certificate? ca-certificates not installed? ca-certificates not up to date?

 

If this is OpenPLi 4, it's SSL implementation is broken, and quite a bit of work to fix it, which we won't do anymore. SSL issues are fixed in 6.0.

[mfaraj57]

It is not openpli 4.0,it is openatv 6.1 with python 2.7.13,will test in openpli 6.0 but expect same error,it is openssl porblem

[WanWizard]

OpenPli 6 has 2.7.13 as well. I suspect it is a server side issue, not a client side issue.

[athoik]

No problem here using python-requests ...

Python 2.7.12 (default, Jul 12 2017, 23:36:04)
[GCC 6.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> res = requests.get("https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg")
>>> res
<Response [200]>
>>> len(res.text)
18001

[mfaraj57]

yes,no problem with requests in my box and no problem with urllib2 or other methods

i can not use requests because i want to download at least 15 images and difficult to stop the code execution until download all images completed,while twisted web will download images in the background without stopping the code execution.

 

this is my code for requests and twisted web and the response

import os
from twisted.web import client
import time
image_url="https://www.almstba.tv/video/uploads/thumbs/718173fd7-1.jpg"
localfile="/tmp/image.jpg"
def getimage():
    import requests
    res = requests.get(image_url)
    print "res", res
    print "len",len(res.text)

getimage()
def downloadback(data):
        print "data",data
        print 'download success'

def downloaderror(data):
        print "error_data",data    

print "downloading image by twisted web..."
client.downloadPage(image_url, localfile).addCallback(downloadback).addErrback(downloaderror)
time.sleep(30)
print localfile +" exists",os.path.exists(localfile)

Edited by mfaraj57, 30 September 2017 - 18:20.

[MiLo]

Maybe twisted wants you to supply a callback for certificate verification?

A quick google on the topic reveals this:
https://twistedmatri.../howto/ssl.html

[mfaraj57]

Thanks for the link

according to the link documentation and others ,they make intentionally difficult to disable twisted certificate but some places claimed this code will disable certificate verification

from twisted.internet import _sslverify
_sslverify.platformTrust = lambda : None 

but i do not get success with it ,may need more testing and applying through real code.

[MiLo]

If it works, the problem when doing that is that it applies globally to all of enigma. It means you basically kill all https security on the box, since without certificate validation, all the other SSL features like encryption become pointless.

[athoik]

It's not validation it's the negotiation that fails!

I made some changes to test code posted, but didn't find the right parameters (chiphers) to support the negotiation.

Most probably need to examine the negotiation using wireshark.

Will post modified test later.

Edited by athoik, 1 October 2017 - 10:33.

[WanWizard]

Hence my remark that the issue is more server side, not client side.

[mfaraj57]

If it is server side,why no problem in getting image by requests method?

[WanWizard]

To be more complete, it is a combination.

 

For security reasons, this server has both SSLv2 and SSLv3 disabled, and only allows TLS (all versions, see https://www.ssllabs....v&s=104.28.2.40). Which explains the " sslv3 alert handshake failure", it doesn't do SSLv3.

 

So you need to convince twisted to use TLS if SSL fails (don't ask me how, I'm a network guy, not a python guy ).

[athoik]

The problem once again is the famous SNI

from socket import socket
from OpenSSL.SSL import TLSv1_METHOD, Context, Connection

def main():
    client = socket()
    print 'Connecting...',
    client.connect(("www.almstba.tv", 443))
    print 'connected', client.getpeername()

    client_ssl = Connection(Context(TLSv1_METHOD), client)
    client_ssl.set_connect_state()
    #client_ssl.set_tlsext_host_name("www.almstba.tv")
    client_ssl.do_handshake()
    print 'Server subject is', client_ssl.get_peer_certificate().get_subject()
    client_ssl.close()

if __name__ == '__main__':
        main()

If you try above with or without set_tlsext_host_name ... then:

# without SNI
# python client.py
Connecting... connected ('104.28.3.40', 443)
Traceback (most recent call last):
  File "client.py", line 18, in <module>
    main()
  File "client.py", line 13, in main
    client_ssl.do_handshake()
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1424, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1172, in _raise_ssl_error
    _raise_current_error()
  File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]

# With SNI
# python client.py 
Connecting... connected ('104.28.2.40', 443)
Server subject is <X509Name object '/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=sni182069.cloudflaressl.com'>

I am sure that twisted has a flag for SNI...

[mfaraj57]

If it works, the problem when doing that is that it applies globally to all of enigma. It means you basically kill all https security on the box, since without certificate validation, all the other SSL features like encryption become pointless.

In general i agree with you but we are talking about small Linux environment and small box for singe user not for big company with large system,difficult to understand this restriction and increasing complexity to avoid  small risk or not all 

[mfaraj57]

To be more complete, it is a combination.

 

For security reasons, this server has both SSLv2 and SSLv3 disabled, and only allows TLS (all versions, see https://www.ssllabs....v&s=104.28.2.40). Which explains the " sslv3 alert handshake failure", it doesn't do SSLv3.

 

So you need to convince twisted to use TLS if SSL fails (don't ask me how, I'm a network guy, not a python guy ).

yes,looks reasonable ,not always could explain things by simple logic with absence of complete knowledge.

As example downloading website by urllib2 revealed error while no problem when downloading same link by  requests.get_url .

.Also received from many users error(bad request) when downloading link by urllib2  while no problem with me for same link,same image ,same python version

[mfaraj57]

The problem once again is the famous SNI
 

from socket import socket
from OpenSSL.SSL import TLSv1_METHOD, Context, Connection

def main():
    client = socket()
    print 'Connecting...',
    client.connect(("www.almstba.tv", 443))
    print 'connected', client.getpeername()

    client_ssl = Connection(Context(TLSv1_METHOD), client)
    client_ssl.set_connect_state()
    #client_ssl.set_tlsext_host_name("www.almstba.tv")
    client_ssl.do_handshake()
    print 'Server subject is', client_ssl.get_peer_certificate().get_subject()
    client_ssl.close()

if __name__ == '__main__':
        main()

If you try above with or without set_tlsext_host_name ... then:

# without SNI
# python client.py
Connecting... connected ('104.28.3.40', 443)
Traceback (most recent call last):
  File "client.py", line 18, in <module>
    main()
  File "client.py", line 13, in main
    client_ssl.do_handshake()
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1424, in do_handshake
    self._raise_ssl_error(self._ssl, result)
  File "/usr/lib/python2.7/site-packages/OpenSSL/SSL.py", line 1172, in _raise_ssl_error
    _raise_current_error()
  File "/usr/lib/python2.7/site-packages/OpenSSL/_util.py", line 48, in exception_from_error_queue
    raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]

# With SNI
# python client.py 
Connecting... connected ('104.28.2.40', 443)
Server subject is <X509Name object '/OU=Domain Control Validated/OU=PositiveSSL Multi-Domain/CN=sni182069.cloudflaressl.com'>

I am sure that twisted has a flag for SNI...

 

Thanks athoik,good work

Then we know now it is SNI issue,but we have to find solution for twisted web,i know it is not popular like other methods for downloading links,but no true alternative for it in downloading multiple files at once.

twisted web is complex enough at least for me with a lot of documentation but i am like Wanwizard  "I'm a network guy, not a python guy"  and not having enough skill  and  apart from do not have enough motivation to go very deep in this subject.

time to thanks you and others for your efforts for SNI for opensource images and remember this frequently when every time facing frequent SNI problem or disaster in dreamos system in my dm900with dmm images

Edited by mfaraj57, 1 October 2017 - 19:53.

[athoik]

We need to use twisted >= 14.

[mfaraj57]

This is log for downloading the image by requests and twisted web in windows with python 2.7 and twisted web 13.0

also twisted web failed while requests is ok

D:\TSmediaTools\Kodi\plugins\plugin.video.ciraonline\wsyspath.py
wsyspath default
wsyspath default
downloading image by requests
res <Response [200]>
len 19035
downloading image by twisted web...
d:/image.jpg exists False

will try twisted 15 latest available for windows