eepgcache and title searching
Re: eepgcache and title searching #181
Posted 4 February 2012 - 12:30
+ if( !(::write( fd, data, data_len + INFO_SIZE)) )
+ eDebug("eit write error (%m)");
a raw write may return less than the size, (e.g. if a driver decides it's more convenient to handle blocks of max 4k for instance) which means you have to retry with the remainder.
Also, errno EINTR can occur in any system call, when a signal was sent to the process (e2 uses some signals), in which case the system call will have to be restarted.
I often use a write wrapper function to handle incomplete writes and EINTR situations.
Or use a FILE and let the c library handle systemcall quirks internally.
(actually, I would even avoid the temporary buffer allocation, and just write the data field by field. But that's just a matter of personal taste, number of syscalls versus fear of heap fragmentation)
Re: eepgcache and title searching #182
Posted 4 February 2012 - 13:12
Is there a write wrapper function already available that can be included and called?Oh, and about writing the info file (or files in general);
+ if( !(::write( fd, data, data_len + INFO_SIZE)) )
+ eDebug("eit write error (%m)");
a raw write may return less than the size, (e.g. if a driver decides it's more convenient to handle blocks of max 4k for instance) which means you have to retry with the remainder.
Also, errno EINTR can occur in any system call, when a signal was sent to the process (e2 uses some signals), in which case the system call will have to be restarted.
I often use a write wrapper function to handle incomplete writes and EINTR situations.
Or use a FILE and let the c library handle systemcall quirks internally.
(actually, I would even avoid the temporary buffer allocation, and just write the data field by field. But that's just a matter of personal taste, number of syscalls versus fear of heap fragmentation)
Re: eepgcache and title searching #183
Posted 4 February 2012 - 14:22
We have some in eSocketBase (select, writeAll), and in eServerSocket (bind, listen, accept)
but nothing global, which can be used by everybody...
Other users of raw syscalls, like for instance the filepush thread, handle incomplete reads, writes and EINTR's internally.
Re: eepgcache and title searching #184
Posted 4 February 2012 - 14:42
Ill take a look, but I think the solution can go in as long as it has the string passed as a const. This is because eit data writing was happening the same way ( agreed not safe and should be fixed ) so we are no worse then what currently exists.A few (I think even duplicated...), but they are in socket related code, as sockets are most likely to suffer from incomplete writes (and reads).
We have some in eSocketBase (select, writeAll), and in eServerSocket (bind, listen, accept)
but nothing global, which can be used by everybody...
Other users of raw syscalls, like for instance the filepush thread, handle incomplete reads, writes and EINTR's internally.
Re: eepgcache and title searching #185
Re: eepgcache and title searching #186
Re: eepgcache and title searching #187
Re: eepgcache and title searching #188
Posted 4 February 2012 - 16:20
I don't really have anything against it, may be better since we don't waste much memory to do the write.yes, probably best in base.
But how do you feel about writing everything directly to file, instead of memcopying it into a temporary buffer first?
I don't think it would be too efficient to write on the id and length of the descriptor fields, which are only 3 bytes in total right now.
Do you know what performance impact there is for calling write so often, especially on small pieces of data? I am not sure does ::write only flush to disk at certain points?
Re: eepgcache and title searching #189
Re: eepgcache and title searching #190
Posted 6 February 2012 - 16:54
We could just modify the .meta file, and add the extra info there. To detect old/new formats, simply use tags in the line or something that new images can readily detect, and old images will just discard as extra unknown data.
If you are planning to introduce a completely new "meta" file, it would be smarter to have it include all data that was already in the meta file (and drop the meta file). The movielist now has to open and read at least 2 files to fetch the required information, hunting for yet another small file isn't going to help performance. If the info file includes both the EIT and META information, it will improve performance, since most time is now spend searching for the file (parsing the horribly inefficient text file takes next to nothing compared to the IO involved).
Re: eepgcache and title searching #191
Posted 6 February 2012 - 18:13
I guess that is possible does anyone have any experience with the meta file as it comes up in a few places and I could use a hand with this changeSorry to bump in this late...
We could just modify the .meta file, and add the extra info there. To detect old/new formats, simply use tags in the line or something that new images can readily detect, and old images will just discard as extra unknown data.
If you are planning to introduce a completely new "meta" file, it would be smarter to have it include all data that was already in the meta file (and drop the meta file). The movielist now has to open and read at least 2 files to fetch the required information, hunting for yet another small file isn't going to help performance. If the info file includes both the EIT and META information, it will improve performance, since most time is now spend searching for the file (parsing the horribly inefficient text file takes next to nothing compared to the IO involved).
Re: eepgcache and title searching #192
Re: eepgcache and title searching #193
Posted 7 February 2012 - 16:13
A very simple setup would be to just use fixed-size blocks in the file, e.g. the first 1k for length, title, name, serviceref, then 1k for the cut points, 4k for the long description, etc. Though that sounds wastefull, the ~32 bytes .cuts file is consuming 4k diskspace anyway, so you'll actually use less in the end.
(*) Example "jumping through hoops" artefacts: The file size is often "0" until you let the bar pass it once, and it gets updated every pass of the bar. Another is that you often lose the propgram markers in the cuts file when you play back the recording while in progress, and you don't get to see those markers until the recording is completed.
Edited by MiLo, 7 February 2012 - 16:16.
Re: eepgcache and title searching #194
Posted 7 February 2012 - 21:06
If we combine all files, we still have to keep reading and parsing of old files for backwards compatibilityAnything that will *reduce* the number of files to open is fine. The best of all would be a file that can be memory-mapped while being written by the recording in progress. That way, the recorder can update it while running and the playback or list can just use the new information as it comes in, without jumping through hoops (*) as it does now. Preferably, i'd like to see the information from .meta, .cuts and .eit in a single file, that can be read while being written.
A very simple setup would be to just use fixed-size blocks in the file, e.g. the first 1k for length, title, name, serviceref, then 1k for the cut points, 4k for the long description, etc. Though that sounds wastefull, the ~32 bytes .cuts file is consuming 4k diskspace anyway, so you'll actually use less in the end.
(*) Example "jumping through hoops" artefacts: The file size is often "0" until you let the bar pass it once, and it gets updated every pass of the bar. Another is that you often lose the propgram markers in the cuts file when you play back the recording while in progress, and you don't get to see those markers until the recording is completed.
By combining all and keeping backwards compatibility we increase the complexity and the chances that new more complicated errors will be introduced.
Current text parsing formats of the meta file do not lend themselves easily for parsing multiline text entries (of course thats not a big problem but should be taken into consideration)
Ideally a new more efficient meta data class should be created to address all issues and combine all data files.
I am willing to work on this but I would like to work on this in a collaborative manner as this is not the only thing I am working on.
Re: eepgcache and title searching #195
Re: eepgcache and title searching #196
Re: eepgcache and title searching #197
Posted 27 February 2012 - 19:39
FATAL: epgcache.cpp:236 ASSERTION pos <= 4108 FAILED!
getResolvedKey config.plugins.crashlogautosubmit.sendAnonCrashlog failed !!
I see some fixes on this topic, but I dont know how to install it (tar.gz). and, if it works, why it is not included in daily update? I'm just updated image, but it still crashed enigma
Re: eepgcache and title searching #198
Re: eepgcache and title searching #199
Re: eepgcache and title searching #200
12 user(s) are reading this topic
0 members, 12 guests, 0 anonymous users