Jump to content


Photo

eepgcache and title searching


  • Please log in to reply
203 replies to this topic

Re: eepgcache and title searching #181 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 4 February 2012 - 12:30

Oh, and about writing the info file (or files in general);

+ if( !(::write( fd, data, data_len + INFO_SIZE)) )
+ eDebug("eit write error (%m)");

a raw write may return less than the size, (e.g. if a driver decides it's more convenient to handle blocks of max 4k for instance) which means you have to retry with the remainder.
Also, errno EINTR can occur in any system call, when a signal was sent to the process (e2 uses some signals), in which case the system call will have to be restarted.
I often use a write wrapper function to handle incomplete writes and EINTR situations.
Or use a FILE and let the c library handle systemcall quirks internally.

(actually, I would even avoid the temporary buffer allocation, and just write the data field by field. But that's just a matter of personal taste, number of syscalls versus fear of heap fragmentation)

Re: eepgcache and title searching #182 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 4 February 2012 - 13:12

Oh, and about writing the info file (or files in general);

+ if( !(::write( fd, data, data_len + INFO_SIZE)) )
+ eDebug("eit write error (%m)");

a raw write may return less than the size, (e.g. if a driver decides it's more convenient to handle blocks of max 4k for instance) which means you have to retry with the remainder.
Also, errno EINTR can occur in any system call, when a signal was sent to the process (e2 uses some signals), in which case the system call will have to be restarted.
I often use a write wrapper function to handle incomplete writes and EINTR situations.
Or use a FILE and let the c library handle systemcall quirks internally.

(actually, I would even avoid the temporary buffer allocation, and just write the data field by field. But that's just a matter of personal taste, number of syscalls versus fear of heap fragmentation)

Is there a write wrapper function already available that can be included and called?

Re: eepgcache and title searching #183 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 4 February 2012 - 14:22

A few (I think even duplicated...), but they are in socket related code, as sockets are most likely to suffer from incomplete writes (and reads).

We have some in eSocketBase (select, writeAll), and in eServerSocket (bind, listen, accept)
but nothing global, which can be used by everybody...

Other users of raw syscalls, like for instance the filepush thread, handle incomplete reads, writes and EINTR's internally.

Re: eepgcache and title searching #184 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 4 February 2012 - 14:42

A few (I think even duplicated...), but they are in socket related code, as sockets are most likely to suffer from incomplete writes (and reads).

We have some in eSocketBase (select, writeAll), and in eServerSocket (bind, listen, accept)
but nothing global, which can be used by everybody...

Other users of raw syscalls, like for instance the filepush thread, handle incomplete reads, writes and EINTR's internally.

Ill take a look, but I think the solution can go in as long as it has the string passed as a const. This is because eit data writing was happening the same way ( agreed not safe and should be fixed ) so we are no worse then what currently exists.

Re: eepgcache and title searching #185 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 4 February 2012 - 15:02

Yes, there have been a lot of non-unix(/linux) developers trying their hands on e2 it seems ;)

Re: eepgcache and title searching #186 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 4 February 2012 - 15:35

Yes, there have been a lot of non-unix(/linux) developers trying their hands on e2 it seems ;)

I will make another patch for this, but maybe we should make this a generic thing in another location? Any thoughts on the location? base?

Re: eepgcache and title searching #187 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 4 February 2012 - 16:02

yes, probably best in base.

But how do you feel about writing everything directly to file, instead of memcopying it into a temporary buffer first?

Re: eepgcache and title searching #188 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 4 February 2012 - 16:20

yes, probably best in base.

But how do you feel about writing everything directly to file, instead of memcopying it into a temporary buffer first?

I don't really have anything against it, may be better since we don't waste much memory to do the write.
I don't think it would be too efficient to write on the id and length of the descriptor fields, which are only 3 bytes in total right now.
Do you know what performance impact there is for calling write so often, especially on small pieces of data? I am not sure does ::write only flush to disk at certain points?

Re: eepgcache and title searching #189 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 4 February 2012 - 16:34

write doesn't force a flush, unless the file was opened with O_SYNC.
(In fact, close won't even flush)

Re: eepgcache and title searching #190 MiLo

  • PLi® Core member
  • 14,054 posts

+298
Excellent

Posted 6 February 2012 - 16:54

Sorry to bump in this late...

We could just modify the .meta file, and add the extra info there. To detect old/new formats, simply use tags in the line or something that new images can readily detect, and old images will just discard as extra unknown data.

If you are planning to introduce a completely new "meta" file, it would be smarter to have it include all data that was already in the meta file (and drop the meta file). The movielist now has to open and read at least 2 files to fetch the required information, hunting for yet another small file isn't going to help performance. If the info file includes both the EIT and META information, it will improve performance, since most time is now spend searching for the file (parsing the horribly inefficient text file takes next to nothing compared to the IO involved).
Real musicians never die - they just decompose

Re: eepgcache and title searching #191 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 6 February 2012 - 18:13

Sorry to bump in this late...

We could just modify the .meta file, and add the extra info there. To detect old/new formats, simply use tags in the line or something that new images can readily detect, and old images will just discard as extra unknown data.

If you are planning to introduce a completely new "meta" file, it would be smarter to have it include all data that was already in the meta file (and drop the meta file). The movielist now has to open and read at least 2 files to fetch the required information, hunting for yet another small file isn't going to help performance. If the info file includes both the EIT and META information, it will improve performance, since most time is now spend searching for the file (parsing the horribly inefficient text file takes next to nothing compared to the IO involved).

I guess that is possible does anyone have any experience with the meta file as it comes up in a few places and I could use a hand with this change

Re: eepgcache and title searching #192 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 6 February 2012 - 20:35

You mean converting the whole meta file to a tlv-like format (in order to allow multiline descriptions)?

Re: eepgcache and title searching #193 MiLo

  • PLi® Core member
  • 14,054 posts

+298
Excellent

Posted 7 February 2012 - 16:13

Anything that will *reduce* the number of files to open is fine. The best of all would be a file that can be memory-mapped while being written by the recording in progress. That way, the recorder can update it while running and the playback or list can just use the new information as it comes in, without jumping through hoops (*) as it does now. Preferably, i'd like to see the information from .meta, .cuts and .eit in a single file, that can be read while being written.

A very simple setup would be to just use fixed-size blocks in the file, e.g. the first 1k for length, title, name, serviceref, then 1k for the cut points, 4k for the long description, etc. Though that sounds wastefull, the ~32 bytes .cuts file is consuming 4k diskspace anyway, so you'll actually use less in the end.


(*) Example "jumping through hoops" artefacts: The file size is often "0" until you let the bar pass it once, and it gets updated every pass of the bar. Another is that you often lose the propgram markers in the cuts file when you play back the recording while in progress, and you don't get to see those markers until the recording is completed.

Edited by MiLo, 7 February 2012 - 16:16.

Real musicians never die - they just decompose

Re: eepgcache and title searching #194 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 7 February 2012 - 21:06

Anything that will *reduce* the number of files to open is fine. The best of all would be a file that can be memory-mapped while being written by the recording in progress. That way, the recorder can update it while running and the playback or list can just use the new information as it comes in, without jumping through hoops (*) as it does now. Preferably, i'd like to see the information from .meta, .cuts and .eit in a single file, that can be read while being written.

A very simple setup would be to just use fixed-size blocks in the file, e.g. the first 1k for length, title, name, serviceref, then 1k for the cut points, 4k for the long description, etc. Though that sounds wastefull, the ~32 bytes .cuts file is consuming 4k diskspace anyway, so you'll actually use less in the end.


(*) Example "jumping through hoops" artefacts: The file size is often "0" until you let the bar pass it once, and it gets updated every pass of the bar. Another is that you often lose the propgram markers in the cuts file when you play back the recording while in progress, and you don't get to see those markers until the recording is completed.

If we combine all files, we still have to keep reading and parsing of old files for backwards compatibility
By combining all and keeping backwards compatibility we increase the complexity and the chances that new more complicated errors will be introduced.
Current text parsing formats of the meta file do not lend themselves easily for parsing multiline text entries (of course thats not a big problem but should be taken into consideration)

Ideally a new more efficient meta data class should be created to address all issues and combine all data files.

I am willing to work on this but I would like to work on this in a collaborative manner as this is not the only thing I am working on.

Re: eepgcache and title searching #195 ims

  • PLi® Core member
  • 13,761 posts

+214
Excellent

Posted 8 February 2012 - 16:29

will be solved compatibility for work with files on others images with OpenWebif too ?
Kdo nic nedělá, nic nezkazí!

Re: eepgcache and title searching #196 awx

  • Senior Member
  • 297 posts

+17
Neutral

Posted 8 February 2012 - 16:54

will be solved compatibility for work with files on others images with OpenWebif too ?

If you make the meta file contain all the data than on other images you will simply have a loss of information but should not have anything break.

Re: eepgcache and title searching #197 halflife

  • Senior Member
  • 59 posts

0
Neutral

Posted 27 February 2012 - 19:39

sorry guys I'm realy noob, I have a problem with latest pli on et9000 and have this error and gsod


FATAL: epgcache.cpp:236 ASSERTION pos <= 4108 FAILED!
getResolvedKey config.plugins.crashlogautosubmit.sendAnonCrashlog failed !!

I see some fixes on this topic, but I dont know how to install it (tar.gz). and, if it works, why it is not included in daily update? I'm just updated image, but it still crashed enigma

Re: eepgcache and title searching #198 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 27 February 2012 - 19:41

probably some garbage in your epgcache

init 4
rm /hdd/epg.dat
init 3

Re: eepgcache and title searching #199 halflife

  • Senior Member
  • 59 posts

0
Neutral

Posted 27 February 2012 - 20:16

I was trying many times to delete all epg files and download it again with crossepg, but gsod every time

Re: eepgcache and title searching #200 pieterg

  • PLi® Core member
  • 32,766 posts

+245
Excellent

Posted 27 February 2012 - 20:19

make sure you delete the epg.dat file when e2 is not running.

Does it also crash when you do not download epg?
crossepg might be importing corrupt data.


12 user(s) are reading this topic

0 members, 12 guests, 0 anonymous users