57 replies to this topic

[sousoux]

Attached is a patch which enables enigma2 to decode the Freesat EPG. This is practically completely based on the work at http://www.rst38.org.uk/vdr/

Apply with -p1 in the enigma2 directory. It also enables MHW EPG which I need as well.

Cheers

Martin

Attached Files

•  freesat.patch.v2   267.32KB   0 downloads

[pieterg]

great! You're OK with it if we want to include this in our image?
(It's either public domain or GPL I guess?)

[littlesat]

Is there also a change that it can "encode" the new CDS Nagraguide? -or- are these system completely different?

[pieterg]

completely different

[sousoux]

Absolutely no problem. The VDR stuff has a comment ... "insert GPL here". I didn't but you could. The Huffman tables seem to be in the public domain. You should keep an eye out for updates from that site since there may still be minor errors in them.

The huffman decoder really needs rewriting in C++ and the hookup to the EPG system is horrible. There is no table of SID to EPG type and PID which would be a much better way of doing it IMO.

I just hacked it together so that I can record BBC stuff simply.

I'm still somewhat confused by the enigma2 EPG. I only starting looking at this a few days ago when I got a 7025. I don't really understand it's caching / saving strategy. The EPG seems to disappear very frequently. It also appears that the timer is not monitoring the "now playing" DVB descriptor/stream which would be a good improvement since it misses the start of programs. The start before / stop after settings should not be necessary.

Have you done any work on this area?

[pieterg]

I'm still somewhat confused by the enigma2 EPG. I only starting looking at this a few days ago when I got a 7025. I don't really understand it's caching / saving strategy. The EPG seems to disappear very frequently.

It could use some improvement indeed.
Also, the current 'keep everything in memory' strategy has its limits, especially when you want to have epg for lots of different providers.
We're thinking about solutions for this.

It also appears that the timer is not monitoring the "now playing" DVB descriptor/stream which would be a good improvement since it misses the start of programs. The start before / stop after settings should not be necessary.

True, but that wouldn't work for all providers, so you wouldn't be able to rely on it, in other words, we still need to use those margins.

[pieterg]

A possible problem could be that you are sharing the seenSections / calcedSections maps with the 'normal' EIT.
And freesat channels do have at least the standard now/next, so the readers might be stopped too early.

Though the risk is theoretical, I think each non-standard epg reader should have it's own seen/calced section maps to be safe.

[pieterg]

Is there also a change that it can "encode" the new CDS Nagraguide? -or- are these system completely different?

Freesat has opted for the EIT standard, only with a custom charset/encoding (huffman), and custom pids and tables.

They use huffman to save a lot on bandwith, 8 days * 140 channels equals a huge amount of data.

Therefore, parsing and receiving the epg is really simple and straightforward, only the huffman decoding is a bit of a pain.

NagraVision on the other hand seems to have had other goals and reasons to deviate from the standard;
-they didn't compress the data, so they don't want to save bandwidth. Everything is plaintext
-however, the format doesn't follow the EIT standard, most likely they wanted to make sure standard stb's couldn't work with it, or they simply have no idea about DVB standards.

[littlesat]

@pieter,
Last weekend you discovered you were grabbing the incorrect PID. Maybe Huffman can help us now...

[pieterg]

please... Have you even tried to read what I was saying in that thread...

EDIT: besides, you're way offtopic here, this thread is about freesat.

[sousoux]

They use huffman to save a lot on bandwith, 8 days * 140 channels equals a huge amount of data.

It seems that the EPG on each channel is only for that specific channel and not for all of Freesat. The PID on Brit Shorts is different and I have not tried that as yet. Dish is covered with snow and not working at present anyway!

[pieterg]

no, my experience is that all data is on all channels.
The only thing that fails in your current version, is that it stops after 90s (enigma2 stops it after the abort timeout).
I've modified this behaviour a little, to let it run for 45 minutes.
(found that receiving all data takes about 35 minutes)

Other than that, it works great. Really nice to have program info for bbc again

[sousoux]

The only thing that fails in your current version, is that it stops after 90s (enigma2 stops it after the abort timeout).

Took me a moment to track down how it is being stopped.

What did you do?

Did you modify this line:

    if ( state == 1 && calcedSections == seenSections || state > 1 )

to

    if ( state == 1 && calcedSections == seenSections || state > 30 )

I can't figure out when calcedSections could be == seenSections at present.

Good find.

[pieterg]

state will not get higher than 2, the abortTimer only runs while state <= 1

I changed it to something like this

if ( state >= 1 && (calcedSections == seenSections || m_FreeSatTimedOut) )

and set the timeout boolean from a timer trigger.

You fill calcedSections just like the normal EIT section reader does, assuming that the section numbering uses the same rules. But it looks like seenSections never equals calcedSections, on matter how long you let it run. So I think the section numbering is not quite the same.

So for now, it just runs till the timeout kicks in.

[sousoux]

I think the reason why this is not working properly is a little more subtle.

As I understand the process when the channel is changed the EPG thread starts reading from all possible PIDs/TIDs. If no data arrives then it closes each channel after 7 secs. Data "arriving" is in the test:

time_t TM = parseDVBtime( eit_event->start_time_1, eit_event->start_time_2,    eit_event->start_time_3, eit_event->start_time_4, eit_event->start_time_5);
    time_t now = ::time(0);

    if ( TM != 3599 && TM > -1)
	   channel->haveData |= source;

As long as data is read from the channel and this test succeeds reading should continue. State only moves to true/1 if this test fails for all PIDs/TIDs.

Why doesn't this test succeed for Freesat EPG data?

[sousoux]

Interestingly enough I noticed that the time on my Dreambox was incorrect after watching the Freesat channels. (out by about 20-30 mins). Could this be related?

[pieterg]

here, it doesn't close after 7s, because it finds data correctly.
It closes after 90s, untill I told it to wait for the 45min timeout.

transponder time changing cannot really be related.

[sousoux]

here, it doesn't close after 7s, because it finds data correctly.
It closes after 90s, untill I told it to wait for the 45min timeout.

transponder time changing cannot really be related.

Sorry. I'm still trying to understand this code. I understand the state 0, 1, 2 transition now with the 7 sec and then 90 sec timing.

I'm also starting to understand the section/segment numbering scheme in the EIT. With my limited understanding it seems to be that the test ...

seenSections == calcedSections

... should actually be ...

seenSections.size() == calcedSections.size()

EDIT: Hmm, seenSections and calcedSections are defined as sets not a maps. Why is the define called tidMap?????!!!! grrr. So seenSections == calcedSections is fine but compares section numbers as well. I'll have to try running this again.

This makes more sense since calcedSections has theoretical section numbers calculated from the last_section and last_segment_section fields added to it. calcedSections.size() stabilizes quite quickly for me.

seenSections.size() == calcedSections.size() comes true for me after about 55 minutes. For the first 10 minutes I see no duplicate sections. For the next 5 minutes I see some. At this point I have about half the sections. It takes 40 minutes to get the rest.

There is currently no scheme to store the section version number in the map so if a section changes during this time (which is quite long after all) it won't get updated. If you change channel of course everything starts over again.

I might add the version number thingy in the map. In a way the map should be maintained for all Freesat channels and not discarded after a channel change. I think that if the section version number was stored in the map and the map persisted across all Freesat channels then you could cut down on a lot of event id lookups.

Thinking out loud ...

[pieterg]

seenSections.size() == calcedSections.size() comes true for me after about 55 minutes. For the first 10 minutes I see no duplicate sections. For the next 5 minutes I see some. At this point I have about half the sections. It takes 40 minutes to get the rest.

OK, so I just didn't wait long enough
I was monitoring the total size, and after 35minutes it didn't grow anymore.
So I decided to use 45 minutes as a timeout. No wonder I never completed the map.

I might add the version number thingy in the map. In a way the map should be maintained for all Freesat channels and not discarded after a channel change. I think that if the section version number was stored in the map and the map persisted across all Freesat channels then you could cut down on a lot of event id lookups.

Yes, but would it really save cpu?
We'd still have to update calcedSections every now and then, so we probably still have to clear those when zapping.
And the section maps consume quite some ram I think, so we shouldn't keep them around on a channel without freesat epg.

[sousoux]

OK, so I just didn't wait long enough
I was monitoring the total size, and after 35minutes it didn't grow anymore.
So I decided to use 45 minutes as a timeout. No wonder I never completed the map.

Looks like it can take easily up to an hour.

Yes, but would it really save cpu?
We'd still have to update calcedSections every now and then, so we probably still have to clear those when zapping.
And the section maps consume quite some ram I think, so we shouldn't keep them around on a channel without freesat epg.

As far as I understand the version number it refers to the whole of a sub table. That is a unique table id + service id for an EIT. So you need a map with the version stored against that combination (quite a small map). If you see a version number change then you clear the calcedSections set out of all the sections with that table id and service id. That should then cause the new sections to be looked at.

If you look at what happens when you switch channel and start pulling in the same data and calling eEPGCache::sectionRead each time I think you save quite a few cycles. There are a lot of map.finds and so on in eEPGCache::sectionRead.

The calcedSections + seenSections for Freesat is ~12000 (number of sections I saw) x 4 (__u32 set member) x 2 (seen and calced) bytes = 96K. Is 96K a lot? You have more experience of the platform.