Jump to content


Photo

urllib2.urlopen and signal.alert


  • Please log in to reply
20 replies to this topic

#1 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 30 January 2019 - 17:09

I'd like to end prematurely a call to urllib2.urlopen() if DNS resolution fails. Calling urllib2.urlopen(url, timeout) timeouts only if the network is up. If network is down it waits for 30 seconds because it can't resolve url. If you use IP instead of DNS, timeout value is being respected.

 

Using:

signal.signal(signal.SIGALRM, timeout_handler)
 
signal.alarm(2)
urllib2.urlopen(url, 2)
signal.alarm(0)

does't work. If url is a FQDN then it stucks for 30 seconds on urllib2.urlopen.



Re: urllib2.urlopen and signal.alert #2 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 30 January 2019 - 19:01

Ok found something about twisted.web.client. I'm digging into it.


Edited by pistacio, 30 January 2019 - 19:01.


Re: urllib2.urlopen and signal.alert #3 mfaraj57

  • Senior Member
  • 1,491 posts

+254
Excellent

Posted 30 January 2019 - 21:07

Python requests is good choice,twisted web sometimes fails with secure https

Edited by mfaraj57, 30 January 2019 - 21:08.


Re: urllib2.urlopen and signal.alert #4 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 30 January 2019 - 21:31

Yes but twisted client knows how to timeout DNS requests while urllib2.urlopen does not.



Re: urllib2.urlopen and signal.alert #5 mfaraj57

  • Senior Member
  • 1,491 posts

+254
Excellent

Posted 30 January 2019 - 22:20

Yes,but the typical use of twisted web to run in seperate thread for running processes.
Requests has also timeout argument and it is very simple,efficient and easy to control.

Re: urllib2.urlopen and signal.alert #6 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 30 January 2019 - 22:29

The timeout argument of python functions only work on connect and read on an already resolved IP. They don't work on DNS resolution. If the box cannot resolve DNS the timeout does not work because the problem starts before connect.I was amazed when I discovered this behaviour.



Re: urllib2.urlopen and signal.alert #7 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 30 January 2019 - 23:39

I've found out another interesting thing, signals don't work in python 2.7 while they work with all older and newer versions.


Edited by pistacio, 30 January 2019 - 23:39.


Re: urllib2.urlopen and signal.alert #8 littlesat

  • PLi® Core member
  • 47,186 posts

+494
Excellent

Posted 30 January 2019 - 23:56

Why not try urlGet from twisted?

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: urllib2.urlopen and signal.alert #9 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 31 January 2019 - 09:26

I'm starting to use getPage from twisted.web.client. What module is urlGet from?



Re: urllib2.urlopen and signal.alert #10 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 31 January 2019 - 09:49

It's now clear to me that all plugins in enigma2 use twisted, from youtube plugin to epgimport. Twisted is a robust solution and it's the way to go. It's a pity I can't use signals in python 2.7 but this is it.



Re: urllib2.urlopen and signal.alert #11 littlesat

  • PLi® Core member
  • 47,186 posts

+494
Excellent

Posted 31 January 2019 - 11:46

Sorry I meant getPage 


WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: urllib2.urlopen and signal.alert #12 betacentauri

  • PLi® Core member
  • 4,946 posts

+233
Excellent

Posted 31 January 2019 - 20:10

But getPage is quite “problematic” if you want to load a website which only accept TLS1.2. At least in the current used twisted version.
Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: urllib2.urlopen and signal.alert #13 littlesat

  • PLi® Core member
  • 47,186 posts

+494
Excellent

Posted 1 February 2019 - 00:11

I thought this issue was solved in openpli7.0 as if was bases on a newer twisted...

WaveFrontier 28.2E | 23.5E | 19.2E | 16E | 13E | 10/9E | 7E | 5E | 1W | 4/5W | 15W


Re: urllib2.urlopen and signal.alert #14 WanWizard

  • Forum Moderator
    PLi® Core member
  • 45,625 posts

+737
Excellent

Posted 1 February 2019 - 01:19

Yeah, should not be a problem. I do have a ca-certificate update in the planning, the one from Pyro is terribly out of date.


Currently in use: VU+Duo 4K (2xFBC S2), Amiko Viper T2C (T2), SAB Alpha Triple HD (S2+T2), Zgemma H3.2TC ©, Zgemma H6 (fallback), VU+Zero (fallback)

Many answers to your question can be found in our new and improved wiki.

note: I do not provide support via PM !

 


Re: urllib2.urlopen and signal.alert #15 betacentauri

  • PLi® Core member
  • 4,946 posts

+233
Excellent

Posted 1 February 2019 - 07:13

You still need a special factory to be able to load the website. I can post an example this evening or tomorrow.
Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: urllib2.urlopen and signal.alert #16 pistacio

  • Senior Member
  • 66 posts

0
Neutral

Posted 1 February 2019 - 08:57

You still need a special factory to be able to load the website. I can post an example this evening or tomorrow.

 

Yes please



Re: urllib2.urlopen and signal.alert #17 WanWizard

  • Forum Moderator
    PLi® Core member
  • 45,625 posts

+737
Excellent

Posted 1 February 2019 - 12:51

I do have a ca-certificate update in the planning, the one from Pyro is terribly out of date.

 

Bumped in it develop to 20180409, the current version in OE master, so that will go live in 7.1-release.


Currently in use: VU+Duo 4K (2xFBC S2), Amiko Viper T2C (T2), SAB Alpha Triple HD (S2+T2), Zgemma H3.2TC ©, Zgemma H6 (fallback), VU+Zero (fallback)

Many answers to your question can be found in our new and improved wiki.

note: I do not provide support via PM !

 


Re: urllib2.urlopen and signal.alert #18 betacentauri

  • PLi® Core member
  • 4,946 posts

+233
Excellent

Posted 1 February 2019 - 17:51

Escape character is '^]'.

openpli 7.0-release gbquad4k

gbquad4k login: root
Last login: Fri Feb  1 17:44:03 CET 2019 on pts/0
root@gbquad4k:~# more /tmp/test 
from twisted.web.client import getPage
from twisted.internet import reactor

def print_data(data):
        print data
        if reactor.running:
                reactor.stop()

def errorCallback(err):
        print err
        if reactor.running:
                reactor.stop()

if __name__ == '__main__':
        d = getPage('https://www.magentasport.de/api/v2/mobile/page/52')
        d.addCallback(print_data)
        d.addErrback(errorCallback)
        reactor.run()
root@gbquad4k:~# python /tmp/test 
[Failure instance: Traceback: <class 'OpenSSL.SSL.Error'>: [('SSL routines', 'SSL23_GET_SERVER_HELLO', 'tlsv1 alert internal error')]
/usr/lib/python2.7/site-packages/twisted/internet/posixbase.py:614:_doReadOrWrite
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:205:doRead
/usr/lib/python2.7/site-packages/twisted/internet/tcp.py:211:_dataReceived
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:315:dataReceived
--- <exception caught here> ---
/usr/lib/python2.7/site-packages/twisted/protocols/tls.py:235:_checkHandshakeStatus
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1716:do_handshake
/usr/lib/python2.7/site-packages/OpenSSL/SSL.py:1456:_raise_ssl_error
/usr/lib/python2.7/site-packages/OpenSSL/_util.py:54:exception_from_error_queue
]

Website only supports TLS1.2 and TLS1.1.


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: urllib2.urlopen and signal.alert #19 betacentauri

  • PLi® Core member
  • 4,946 posts

+233
Excellent

Posted 1 February 2019 - 18:13

This works

from twisted.web.client import getPage                                                                            
from twisted.internet import reactor                                                                              
from twisted.internet import ssl as twisted_ssl                                                                   
from twisted.internet._sslverify import ClientTLSOptions                                                          
                                                                                                                  
class clientContextFactory(twisted_ssl.ClientContextFactory):                                                     
        def getContext(self):                                                                                     
                ctx = self._contextFactory(self.method)                                                           
                ClientTLSOptions('www.magentasport.de', ctx)                                                      
                return ctx                                                                                        
                                                                                                                  
                                                                                                                  
def print_data(data):                                                                                             
        print data                                                                                                
        if reactor.running:                                                                                       
                reactor.stop()                                                                                    
                                                                                                                  
def errorCallback(err):                                                                                           
        print err                                                                                                 
        if reactor.running:                                                                                       
                reactor.stop()                                                                                    
                                                                                                                  
if __name__ == '__main__':                                                                                        
        cf = clientContextFactory()                                                                               
        d = getPage('https://www.magentasport.de/api/v2/mobile/page/52', contextFactory = cf)                     
        d.addCallback(print_data)                                                                                 
        d.addErrback(errorCallback)                                                                               
        reactor.run()    

(see also here: https://forums.openp...-2#entry776809)

This was not possible in 6.2.

 

 

But I don't really like it. Is there another possibility to download data without blocking?


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04

Re: urllib2.urlopen and signal.alert #20 betacentauri

  • PLi® Core member
  • 4,946 posts

+233
Excellent

Posted 1 February 2019 - 19:28

Well, the mentioned https://github.com/r...s/httpclient.py seems really to be a good alternative.


Xtrend ET-9200, ET-8000, ET-10000, OpenPliPC on Ubuntu 12.04




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users