20 replies to this topic

[microboi37]

I'd like to end prematurely a call to urllib2.urlopen() if DNS resolution fails. Calling urllib2.urlopen(url, timeout) timeouts only if the network is up. If network is down it waits for 30 seconds because it can't resolve url. If you use IP instead of DNS, timeout value is being respected.

 

Using:

signal.signal(signal.SIGALRM, timeout_handler)
 
signal.alarm(2)
urllib2.urlopen(url, 2)
signal.alarm(0)

does't work. If url is a FQDN then it stucks for 30 seconds on urllib2.urlopen.

[microboi37]

Ok found something about twisted.web.client. I'm digging into it.

Edited by pistacio, 30 January 2019 - 19:01.

[mfaraj57]

Python requests is good choice,twisted web sometimes fails with secure https

Edited by mfaraj57, 30 January 2019 - 21:08.

[microboi37]

Yes but twisted client knows how to timeout DNS requests while urllib2.urlopen does not.

[mfaraj57]

Yes,but the typical use of twisted web to run in seperate thread for running processes.
Requests has also timeout argument and it is very simple,efficient and easy to control.

[microboi37]

The timeout argument of python functions only work on connect and read on an already resolved IP. They don't work on DNS resolution. If the box cannot resolve DNS the timeout does not work because the problem starts before connect.I was amazed when I discovered this behaviour.

[microboi37]

I've found out another interesting thing, signals don't work in python 2.7 while they work with all older and newer versions.

Edited by pistacio, 30 January 2019 - 23:39.

[littlesat]

Why not try urlGet from twisted?

[microboi37]

I'm starting to use getPage from twisted.web.client. What module is urlGet from?

[microboi37]

It's now clear to me that all plugins in enigma2 use twisted, from youtube plugin to epgimport. Twisted is a robust solution and it's the way to go. It's a pity I can't use signals in python 2.7 but this is it.

[littlesat]

Sorry I meant getPage 

[betacentauri]

But getPage is quite “problematic” if you want to load a website which only accept TLS1.2. At least in the current used twisted version.

[littlesat]

I thought this issue was solved in openpli7.0 as if was bases on a newer twisted...

[WanWizard]

Yeah, should not be a problem. I do have a ca-certificate update in the planning, the one from Pyro is terribly out of date.

[betacentauri]

You still need a special factory to be able to load the website. I can post an example this evening or tomorrow.

[microboi37]

You still need a special factory to be able to load the website. I can post an example this evening or tomorrow.

 

Yes please

[WanWizard]

I do have a ca-certificate update in the planning, the one from Pyro is terribly out of date.

 

Bumped in it develop to 20180409, the current version in OE master, so that will go live in 7.1-release.

[betacentauri]

Escape character is '^]'.

openpli 7.0-release gbquad4k

gbquad4k login: root
Last login: Fri Feb  1 17:44:03 CET 2019 on pts/0
root@gbquad4k:~# more /tmp/test 
from twisted.web.client import getPage
from twisted.internet import reactor

def print_data(data):
        print data
        if reactor.running:
                reactor.stop()

def errorCallback(err):
        print err
        if reactor.running:
                reactor.stop()

if __name__ == '__main__':
        d = getPage('https://www.magentasport.de/api/v2/mobile/page/52')
        d.addCallback(print_data)
        d.addErrback(errorCallback)
        reactor.run()
root@gbquad4k:~# python /tmp/test 
[Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'tlsv1 alert internal error')]
/usr/lib/python2.7/site-packages/twisted/internet/posixbase.py:614:_doReadOrWrite
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:205:doRead
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:211:_dataReceived
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:315:dataReceived
--- <exception caught here> ---
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:235:_checkHandshakeStatus
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1716:do_handshake
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1456:_raise_ssl_error
/usr/lib/python2.7/site-packages/OpenSSL/_util.py:54:exception_from_error_queue
]

Website only supports TLS1.2 and TLS1.1.

[betacentauri]

This works

from twisted.web.client import getPage                                                                            
from twisted.internet import reactor                                                                              
from twisted.internet import ssl as twisted_ssl                                                                   
from twisted.internet._sslverify import ClientTLSOptions                                                          
                                                                                                                  
class clientContextFactory(twisted_ssl.ClientContextFactory):                                                     
        def getContext(self):                                                                                     
                ctx = self._contextFactory(self.method)                                                           
                ClientTLSOptions('www.magentasport.de', ctx)                                                      
                return ctx                                                                                        
                                                                                                                  
                                                                                                                  
def print_data(data):                                                                                             
        print data                                                                                                
        if reactor.running:                                                                                       
                reactor.stop()                                                                                    
                                                                                                                  
def errorCallback(err):                                                                                           
        print err                                                                                                 
        if reactor.running:                                                                                       
                reactor.stop()                                                                                    
                                                                                                                  
if __name__ == '__main__':                                                                                        
        cf = clientContextFactory()                                                                               
        d = getPage('https://www.magentasport.de/api/v2/mobile/page/52', contextFactory = cf)                     
        d.addCallback(print_data)                                                                                 
        d.addErrback(errorCallback)                                                                               
        reactor.run()    

(see also here: https://forums.openp...-2#entry776809)

This was not possible in 6.2.

 

 

But I don't really like it. Is there another possibility to download data without blocking?

[betacentauri]

Well, the mentioned https://github.com/r...s/httpclient.py seems really to be a good alternative.