203 replies to this topic

[pieterg]

Oh, and about writing the info file (or files in general);

+ if( !(::write( fd, data, data_len + INFO_SIZE)) )
+ eDebug("eit write error (%m)");

a raw write may return less than the size, (e.g. if a driver decides it's more convenient to handle blocks of max 4k for instance) which means you have to retry with the remainder.
Also, errno EINTR can occur in any system call, when a signal was sent to the process (e2 uses some signals), in which case the system call will have to be restarted.
I often use a write wrapper function to handle incomplete writes and EINTR situations.
Or use a FILE and let the c library handle systemcall quirks internally.

(actually, I would even avoid the temporary buffer allocation, and just write the data field by field. But that's just a matter of personal taste, number of syscalls versus fear of heap fragmentation)

[awx]

Oh, and about writing the info file (or files in general);

+ if( !(::write( fd, data, data_len + INFO_SIZE)) )
+ eDebug("eit write error (%m)");

a raw write may return less than the size, (e.g. if a driver decides it's more convenient to handle blocks of max 4k for instance) which means you have to retry with the remainder.
Also, errno EINTR can occur in any system call, when a signal was sent to the process (e2 uses some signals), in which case the system call will have to be restarted.
I often use a write wrapper function to handle incomplete writes and EINTR situations.
Or use a FILE and let the c library handle systemcall quirks internally.

(actually, I would even avoid the temporary buffer allocation, and just write the data field by field. But that's just a matter of personal taste, number of syscalls versus fear of heap fragmentation)

Is there a write wrapper function already available that can be included and called?

[pieterg]

A few (I think even duplicated...), but they are in socket related code, as sockets are most likely to suffer from incomplete writes (and reads).

We have some in eSocketBase (select, writeAll), and in eServerSocket (bind, listen, accept)
but nothing global, which can be used by everybody...

Other users of raw syscalls, like for instance the filepush thread, handle incomplete reads, writes and EINTR's internally.

[awx]

A few (I think even duplicated...), but they are in socket related code, as sockets are most likely to suffer from incomplete writes (and reads).

We have some in eSocketBase (select, writeAll), and in eServerSocket (bind, listen, accept)
but nothing global, which can be used by everybody...

Other users of raw syscalls, like for instance the filepush thread, handle incomplete reads, writes and EINTR's internally.

Ill take a look, but I think the solution can go in as long as it has the string passed as a const. This is because eit data writing was happening the same way ( agreed not safe and should be fixed ) so we are no worse then what currently exists.

[pieterg]

Yes, there have been a lot of non-unix(/linux) developers trying their hands on e2 it seems

[awx]

Yes, there have been a lot of non-unix(/linux) developers trying their hands on e2 it seems

I will make another patch for this, but maybe we should make this a generic thing in another location? Any thoughts on the location? base?

[pieterg]

yes, probably best in base.

But how do you feel about writing everything directly to file, instead of memcopying it into a temporary buffer first?

[awx]

yes, probably best in base.

But how do you feel about writing everything directly to file, instead of memcopying it into a temporary buffer first?

I don't really have anything against it, may be better since we don't waste much memory to do the write.
I don't think it would be too efficient to write on the id and length of the descriptor fields, which are only 3 bytes in total right now.
Do you know what performance impact there is for calling write so often, especially on small pieces of data? I am not sure does ::write only flush to disk at certain points?

[pieterg]

write doesn't force a flush, unless the file was opened with O_SYNC.
(In fact, close won't even flush)

[MiLo]

Sorry to bump in this late...

We could just modify the .meta file, and add the extra info there. To detect old/new formats, simply use tags in the line or something that new images can readily detect, and old images will just discard as extra unknown data.

If you are planning to introduce a completely new "meta" file, it would be smarter to have it include all data that was already in the meta file (and drop the meta file). The movielist now has to open and read at least 2 files to fetch the required information, hunting for yet another small file isn't going to help performance. If the info file includes both the EIT and META information, it will improve performance, since most time is now spend searching for the file (parsing the horribly inefficient text file takes next to nothing compared to the IO involved).

[awx]

Sorry to bump in this late...

We could just modify the .meta file, and add the extra info there. To detect old/new formats, simply use tags in the line or something that new images can readily detect, and old images will just discard as extra unknown data.

If you are planning to introduce a completely new "meta" file, it would be smarter to have it include all data that was already in the meta file (and drop the meta file). The movielist now has to open and read at least 2 files to fetch the required information, hunting for yet another small file isn't going to help performance. If the info file includes both the EIT and META information, it will improve performance, since most time is now spend searching for the file (parsing the horribly inefficient text file takes next to nothing compared to the IO involved).

I guess that is possible does anyone have any experience with the meta file as it comes up in a few places and I could use a hand with this change

[pieterg]

You mean converting the whole meta file to a tlv-like format (in order to allow multiline descriptions)?

[MiLo]

Anything that will *reduce* the number of files to open is fine. The best of all would be a file that can be memory-mapped while being written by the recording in progress. That way, the recorder can update it while running and the playback or list can just use the new information as it comes in, without jumping through hoops (*) as it does now. Preferably, i'd like to see the information from .meta, .cuts and .eit in a single file, that can be read while being written.

A very simple setup would be to just use fixed-size blocks in the file, e.g. the first 1k for length, title, name, serviceref, then 1k for the cut points, 4k for the long description, etc. Though that sounds wastefull, the ~32 bytes .cuts file is consuming 4k diskspace anyway, so you'll actually use less in the end.


(*) Example "jumping through hoops" artefacts: The file size is often "0" until you let the bar pass it once, and it gets updated every pass of the bar. Another is that you often lose the propgram markers in the cuts file when you play back the recording while in progress, and you don't get to see those markers until the recording is completed.

Edited by MiLo, 7 February 2012 - 16:16.

[awx]

Anything that will *reduce* the number of files to open is fine. The best of all would be a file that can be memory-mapped while being written by the recording in progress. That way, the recorder can update it while running and the playback or list can just use the new information as it comes in, without jumping through hoops (*) as it does now. Preferably, i'd like to see the information from .meta, .cuts and .eit in a single file, that can be read while being written.

A very simple setup would be to just use fixed-size blocks in the file, e.g. the first 1k for length, title, name, serviceref, then 1k for the cut points, 4k for the long description, etc. Though that sounds wastefull, the ~32 bytes .cuts file is consuming 4k diskspace anyway, so you'll actually use less in the end.


(*) Example "jumping through hoops" artefacts: The file size is often "0" until you let the bar pass it once, and it gets updated every pass of the bar. Another is that you often lose the propgram markers in the cuts file when you play back the recording while in progress, and you don't get to see those markers until the recording is completed.

If we combine all files, we still have to keep reading and parsing of old files for backwards compatibility
By combining all and keeping backwards compatibility we increase the complexity and the chances that new more complicated errors will be introduced.
Current text parsing formats of the meta file do not lend themselves easily for parsing multiline text entries (of course thats not a big problem but should be taken into consideration)

Ideally a new more efficient meta data class should be created to address all issues and combine all data files.

I am willing to work on this but I would like to work on this in a collaborative manner as this is not the only thing I am working on.

[ims]

will be solved compatibility for work with files on others images with OpenWebif too ?

[awx]

will be solved compatibility for work with files on others images with OpenWebif too ?

If you make the meta file contain all the data than on other images you will simply have a loss of information but should not have anything break.

[halflife]

sorry guys I'm realy noob, I have a problem with latest pli on et9000 and have this error and gsod


FATAL: epgcache.cpp:236 ASSERTION pos <= 4108 FAILED!
getResolvedKey config.plugins.crashlogautosubmit.sendAnonCrashlog failed !!

I see some fixes on this topic, but I dont know how to install it (tar.gz). and, if it works, why it is not included in daily update? I'm just updated image, but it still crashed enigma

[pieterg]

probably some garbage in your epgcache

init 4
rm /hdd/epg.dat
init 3

[halflife]

I was trying many times to delete all epg files and download it again with crossepg, but gsod every time

[pieterg]

make sure you delete the epg.dat file when e2 is not running.

Does it also crash when you do not download epg?
crossepg might be importing corrupt data.