50 replies to this topic

[keratos]

When I have the script or code that takes the long run time... maybe I can help you.... Without having or know what you want to do it is very difficult to help.

I'm not sure it is that difficult to imagine...

 

5 functions that just manipulate data in a file. Each function does a different task, together the functions parse an M3U playlist creating channels and bouquets. The program flow is very simple, ...

 

__init__(): calls self.parse()

parse():

[keratos]

sorry , hit post too soon. Here we go...

 

 

I'm not sure it is that difficult to imagine...let me try again....

 

 

There are 5 functions that just manipulate data in a file. Each function does a different task, together the functions parse an M3U playlist creating channels and bouquets. The program flow is very very simple, ...

 

__init__(): 

   calls self.parse()

 

parse():

   calls createDict()

 

createDict(): 

   calls createList()

 

createList():

   calls createBouquets()

 

createBouquets():

   calls doHideShow()

 

doHideShow():

   is the last function

 

 

Now, we dont really need to know at this stage what each function does, . It will muddy the water, but hold that thought until you've read all this post. By the way, I will share once it is working and invite comment on how to push it upstream if needed?? Anyway, back to the plugin, ...

 

Each function (and therefore the entire plugin) blocks the main loop because each function takes around 1 minute to run.  I dont know what the best practice is for ensuring this doesnt happen. Hence my proposals above.

 

I just need to know what options exist to invoke my functions from __init__ such that they dont block the main loop. I can recode each function if needed to accomodate the recommendation. For example, you might suggest that i use some form of timer or event where each functions performs a repeatable specific task on just one specific item  - in a list/dict whatever  and keeps memory of its last/next item - on each invocation. You might however suggest that I code in some calls to some routines that I dont know of, which allows enigma loop to run / paint the components/screens?? 

 

So you see, it doest matter really what the code does, rather I am asking what methods exist in enigma2 to prevent blocking and a few lines of example on how to use them. I will then try a few and see which ones work best. I will also have gained knowledge to help others

 

PLEASE

[MiLo]

You are referring me to code that does not resemble my model.

There exactly lies the problem. You must change the model to make this work.

Rather than doing:
 

data = some_file.read(some_bytes)
process(data)

You must do (pseudo code, not actually runnable):
 

onDataAvailable(some_file).append(my_object.parsingstep1)

def parsingstep1(self, some_data):
  self.data.append(some_data)
    if self.readyForNextStep():
    ...

Edited by MiLo, 28 August 2017 - 18:30.

[keratos]

 

You are referring me to code that does not resemble my model.

There exactly lies the problem. You must change the model to make this work.

Rather than doing:
 

data = some_file.read(some_bytes)
process(data)

You must do (pseudo code, not actually runnable):
 

onDataAvailable(some_file).append(my_object.parsingstep1)

def parsingstep1(self, some_data):
  self.data.append(some_data)
    if self.readyForNextStep():
    ...

 

Yes, I know I need to change the model.

 

But your suggetion  is not the problem. As i sated earlier.  If it were i would have used the downloader class.

 

The issue is not reading a file. It is processing the file, nested for loops, math processing, use of difflib, string manipulation

 

Again, I plead, just why cant someone tell me what the available options are for event driven models. Why is there such secrecy and unwillingness to help. I just can fathom why. In all my 45+years in the software industry, from being a lecturer at UCL here in the UK , working with school kids to raise their computer literacy, and helping on forums with unix/C/C++ graphic modelling, Ive never experienced a drawn out request for help like this. All I am asking for, is the available options in enigma2 for event driven modelling through which a series of functions can be called inside which single passes of an iterator can be invokedand whereby engima2 loop ("reactor??) isnt blocked. Each call to each function can iterate (one per call) over some data.  Then I can try a few to see which ones fit best, Thats all. There can't be tens,hundreds, surely, only a handful?? Maybe a task class (Ive seen a few of these in the code base but seems to me they deal with Console and scripts?? - I could be wrong). Can anyone please help by suggesting appropriate API to call for non-blocking event driven model

Edited by keratos, 29 August 2017 - 07:35.

[keratos]

Just thought - EPGimport plugin works in the background so I will look at that and try to work out how its done - but some pointers would help me guys

[betacentauri]

Problem is that there is no documentation at all!

[keratos]

Problem is that there is no documentation at all!

Agreed! totally. Its like you could potentially work out - gvien time - the code base. But from scratch and with zero documentation, this is labour intensive. I dont know why it isnt documented. Ive searched all over the net. Nothing bar a couple of examples on how to write a very very basic plugin - you know the ones - message box, a config list. What gets me is that , well my perception, is the secrecy or "closed community" . Its almost like they dont want anyone to write plugins but then why offer the facility. This isnt in the spirit of OSS and community based development. I just cant understand why its being made so difficult. If someone asked me a coding problem on a piece of software I am competent on, authored, or admin, I would help as much as I can whilst ensuring they remain compliant with core values, good practice and so, I mean, why wouldnt you 

 

?

[keratos]

Well, looking at EPGimport has fried my head. Still need help

[betacentauri]

Devs (I'm one ) are often too lazy to write a good documentation. It's a hobby project. And a documentation needs to be maintained. That's much work. Every day/minute code can change. And an old/wrong documentation is almost as bad as no documentation.

In most cases best option is to ask devs here or to try to find a plugin which already has a solution for the problem...

[keratos]

er, a "hobby".

 

I think you are underestimating , grossly , your success with this. There must be 1000s of consumers of this "hobby project" across the forums, and probably that accounts for 20% of the actual users who install it and never visit a forum. Wow, you really are humbling yourself mate.

 

I write documentation if not at least in the code, so I can remember the design time decisions, workflow, data modelling and interface considerations  taken. But then again I tend to work alone and not part of a team because I'm the only developer in this business function. Maybe if I was part of a team, working on but one project, and working only on a small module, then I could remember aforementioned. 

 

I have asked the devs, havent I, by coming on here ??? And believe me, god, I have looked at the enigma code and tried to work through the undocumented tangle of coupled modules and packages,  but you have to remember that I am trying to figure out what is in the devs' heads aren't I, from no docs and little support.

 

I am grateful you take the time to post here, but perhaps as a dev, you could enlighten me as to what modules (.py) provide event driven models that which I can use to either time slice a function (pre-emptive interrupts) , or if enigma/python stack doesnt offer this, then some kind of function that I can call to allow engima to paint/update, or if that isnt available then a timer with callback that will invoke my callback every N seconds in which case my callback will then have to deal with restricting execution time to prevent the main loop blocking - incidentally, what is the time period after which the spinner and debug message is displayed "main loop taking longer than expected" or whatever it was??

[WanWizard]

EPGimport works the same way as described above. It reads the text file, converts its xml into an internal datastructure, then creates an event loop using callbacks to process one xml node at the time, until it reaches the end. That doesn't solve your problem if processing one dataset already takes to long, then you also need to break up your process (like MiLo already suggested).

 

Yes, that is complex, but that is what comes with writing code for a realtime environment where you don't have a higher level execution scheduler, but have to work with a simple event loop.

 

It might be simpler to make the processing a batch process (and not an enigma process), use something like Popen() to run it in the background (so the Linux scheduler can handle it), and only use an event loop to monitor progress and present feedback to the user if needed (like for example filecopy does when copying or moving a recording).

[mrvica]

Well, looking at EPGimport has fried my head. Still need help

what about EPGrefresh

https://github.com/O...c/EPGRefresh.py

is doing some job in background, I don´t use it, so can´t tell you whether it is showing spinner

[WanWizard]

EPGrefresh in itself doesn't do anything.

 

It just tunes a service for a specified time, and relies on the the standard epg readers in Enigma to capture and store EPG data on the service being tuned into.

[keratos]

I feel like banging my head on a really hard brick wall. Let me try again , for the , er .... fifth time ....

 

 

*** Please provide names of functions used by enigma2 for event handling.

***

*** Either those that I need to call                              

*** OR

*** Those that enigma invokes if provided as an attr of my class

***

*** So that the main loop isnt blocked

 

 

Phew!

 

 

 

Close to giving up here and going to another forum which would be a shame. There surely, must be, to god's sake someone who knows the code.

[hyperonex]

What I've done to send an HTTP request to a server without blocking the main loop while waiting for the request is the following

threading.Thread(target=functionNameToSendData,args=(inputs,)).start()

[WanWizard]

I'm not a Python programmer, but I can read it pretty well, and I have the theoretical background required, what helps when I need to disect the code and get the gist.

 

You can be frustrated all you want @keratos, but if you expect someone here to write you a complete code example, I think that indeed you have to move to a different forum.

 

As there is no documentation, not even in the code, you wil have to do what everyone else has to do, and what I did when I replied to your questions, put some effort in and look in the code and figure out what it does. If your programming skills don't reach that far, you're lifting above your weight, and you'll have to sort that one out first.

[betacentauri]

Handle a long file download is different to handle a long processing step. And whether you use python code for the processing is again different to handle than Linux shell code.
So we need to know what you do and what steps take so much time that the spinner occurs. Then I guess someone can help you here.

P.S. It's still a hobby. I don't earn money with it ...

[keratos]

What I've done to send an HTTP request to a server without blocking the main loop while waiting for the request is the following

threading.Thread(target=functionNameToSendData,args=(inputs,)).start()

THANK YOU
THANK YOU
THANK YOU

i shall recode to use this model

[hyperonex]

I hope it works for you. Just make sure to handle the concurrent access to your progress bar if any

Edited by hyperonex, 29 August 2017 - 21:38.

[keratos]

I hope it works for you. Just make sure to handle the concurrent access to your progress bar if any

Yes, I do have a PB and will codify so that the threads report progress to a progress thread ... 

 

until someone - or indeed myself - identifies a better way than threads