121 replies to this topic

[DimitarCC]

In addition c++ have std::wstring that is specially for wide character strings (unicode) that have c_str as well that can do the trick.....

Ofcource if here is better expert in c++ let he says his opinion

[WanWizard]

I'm afraid I have to pass it ion, as I don't get it.

 

Python does

self.picload.startDecode(self.filelist[self.index])

and I checked,

self.filelist[self.index]

is an <str> object.

 

However, startDecode() is defined as

RESULT startDecode(const char *filename, int x=0, int y=0, bool async=true);

which wants bytes or a bytearray.

 

However, if I call it with a bytes value, it crashes with an argument error saying it expects a "char *".

 

So this is clearly above my paygrade...

[DimitarCC]

However, startDecode() is defined as

RESULT startDecode(const char *filename, int x=0, int y=0, bool async=true);

which wants bytes or a bytearray.
(

cost char *filename is not a bytes by definition. Char is a struct in c++ that represent 1 byte character. And with utf8 character you pass on 2 byte character and ofcource it doesnt like it.

[Huevos]

Isn't your problem that you read or copy the files from a Windows system without proper characterset conversion?

Are you serious that a filename conversion is needed?

 

IMS reported this months ago. https://forums.openp...evelop-crashes/

 

At the time I gave a working solution for this problem, but it seems no one was interested.

Edited by Huevos, 15 October 2023 - 09:28.

[luisteraar]

https://forums.openp...evelop-crashes/

 

I have no permission to open that forum

 what is the problem ?

[DimitarCC]

Read above... Umlauts in filenames crash enigma...

[luisteraar]

https://forums.openp...evelop-crashes/

 

I have no permission to open that forum

 what is the problem ?

 

Read above... Umlauts in filenames crash enigma...

Easy to correct.

Copy byte to byte to new string check each byte if it

is real ascii.

if not real ascii replace umlaut with ascii code without umlaut and

add a "e' after it.

Edited by luisteraar, 15 October 2023 - 10:48.

[DimitarCC]

The question is in this way is the patch going to be correct and is enigma going to find the file

[luisteraar]

https://forums.openp...evelop-crashes/

 

I have no access to that forum

Mod please give me access so i can check the

patch .

 

I'am C oriented and opening such a file is no problem

.

Edited by luisteraar, 15 October 2023 - 11:38.

[Dimitrij]

# filename fixes in eServiceReference and MovieList (fixes enigma crash)
https://github.com/OpenViX/enigma2/commit/2c74d42983c6969c0d2ec87b3c48622ba0ff3a45
https://github.com/OpenViX/enigma2/commit/6efb844aba07357c3285e120bde4dc048e2bbad8
https://github.com/OpenViX/enigma2/commit/93fa129f5555bef5e24442d1654d28656359f5e2
https://github.com/OpenViX/enigma2/commit/d9f180b06300586bf7f1e9fa34eed310937e2bef
https://github.com/OpenViX/enigma2/commit/69bb2ba48b5376c5f303b46fd399d8bf53cf28c7

# similar fixes for file_eraser and Trashcan
https://github.com/OpenViX/enigma2/commit/29a92e82ce71df88b6d48af0968df2833cab022b
https://github.com/OpenViX/enigma2/commit/ab86bf81616e75aa3928a82be90d6a91bc5aa220
https://github.com/OpenViX/enigma2/commit/9bb758215a1d7996be4c4a4f880ed9ab1d5758ab
https://github.com/OpenViX/enigma2/commit/69bb2ba48b5376c5f303b46fd399d8bf53cf28c7
https://github.com/OpenViX/enigma2/commit/9624cf8a1df2eca5a11b6810b63f9b972e408076

 

[luisteraar]

OK thanks

if (PyBytes_Check($input)) {

                $1 = PyBytes_AsString($input);

        } else {

                $1 = PyBytes_AsString(PyUnicode_AsEncodedString($input, "utf-8", "surrogateescape"));

 

 

methode is crear

if not real ascii convert it 

[birdman]

And it is nothing new. Windows uses the ISO-8859-1 characterset (at least in the Latin part of the world), Linux (and most of the rest of the world) uses UTF-8. Which means that if you have non-ASCII text (in files, or file names), it needs to be converted.

 

That is incorrect.

 

Linux filename are (in Python terms) bytes, not str. Any Python code dealing with filenames has to treat the filename as bytes otherwise it can fall over.

 

A filename can be any series of bytes (except NUL) in any order.

The fact that this may be displayed as though it is a utf8-string is application dependent.

Edited by birdman, 15 October 2023 - 13:33.

[birdman]

Read above... Umlauts in filenames crash enigma...

No. bytes that are not valid utf8 sequences crash enigma2.

An umlaut (Unicode U+0308, 0xCC 0x88, or U+00A8, 0xC2 0xA8) would be OK. A iso-8858-1 character that includes an umlaut in its display would not be.

Edited by birdman, 15 October 2023 - 13:41.

[birdman]

 

methode is crear

if not real ascii convert it 

 

Wrong.

 

If it is a filename it should always be a bytes stream.

[WanWizard]

At the time I gave a working solution for this problem, but it seems no one was interested.

 

As I wrote they are not a solution.

 

OpenVIX crashes too on the file in question, so it isn't fixed there too...
 

[WanWizard]

Linux filename are (in Python terms) bytes, not str. Any Python code dealing with filenames has to treat the filename as bytes otherwise it can fall over.

 

A filename can be any series of bytes (except NUL) in any order.

The fact that this may be displayed as though it is a utf8-string is application dependent.

 

You need to read this in the context of the problem, no point having an abstract and therefore not relevant discussion.

 

In E2, the files in a directory are enumerated in a list of serviceref objects, in which the path is an std::string, which is set from readdir() output. This is how the original sequence of bytes result from the filesystem ends up in a Python str object.

 

This creates a sort of catch-22:

• the string can't be handled in python without causing a crash, in case the string doesn't contain utf-8
• it is not easy to convert the string if you don't know the original encoding (although you can guess using chardet, which OpenVIX has done, and I also implemented, but not committed yet)
• you can't alter the path in the serviceref itself, as that is also used to access the file, any conversion of that variable causes file access to fail

ePicLoad() accepts the path as a char pointer, so imho the entire charset business should be transparent for that part of the code.

 

From a Python perspective, it crashes on

self.picload.startDecode(self.filelist[self.index])

where

self.filelist[self.index]

is an str object containing the binary representation of the filename (with the non-utf8 code point for the u umlaut).

 

If this is converted to bytes (without loss), the startDecode() call triggers an exception

in method 'ePicLoad_startDecode', argument 2 of type 'char const *'

which means passing the string to startDecode() isn't the problem, something in the C code of ePicLoad is, as this call never returns.
 

[Huevos]

 

https://forums.openp...evelop-crashes/

 

I have no permission to open that forum

 what is the problem ?

 

Read above... Umlauts in filenames crash enigma...

Easy to correct.

Copy byte to byte to new string check each byte if it

is real ascii.

if not real ascii replace umlaut with ascii code without umlaut and

add a "e' after it.

 

That is not going to work because once you modify the name you won't be able to access it on the filesystem.

[Huevos]

I don't know what part of what Birdman says is not understandable. You just tell python to read the file from disk as a bytestream.

[Huevos]

 

Linux filename are (in Python terms) bytes, not str. Any Python code dealing with filenames has to treat the filename as bytes otherwise it can fall over.

 

A filename can be any series of bytes (except NUL) in any order.

The fact that this may be displayed as though it is a utf8-string is application dependent.

 

You need to read this in the context of the problem, no point having an abstract and therefore not relevant discussion.

 

In E2, the files in a directory are enumerated in a list of serviceref objects, in which the path is an std::string, which is set from readdir() output. This is how the original sequence of bytes result from the filesystem ends up in a Python str object.

 

This creates a sort of catch-22:

• the string can't be handled in python without causing a crash, in case the string doesn't contain utf-8
• it is not easy to convert the string if you don't know the original encoding (although you can guess using chardet, which OpenVIX has done, and I also implemented, but not committed yet)
• you can't alter the path in the serviceref itself, as that is also used to access the file, any conversion of that variable causes file access to fail

ePicLoad() accepts the path as a char pointer, so imho the entire charset business should be transparent for that part of the code.

 

From a Python perspective, it crashes on

self.picload.startDecode(self.filelist[self.index])

where

self.filelist[self.index]

is an str object containing the binary representation of the filename (with the non-utf8 code point for the u umlaut).

 

If this is converted to bytes (without loss), the startDecode() call triggers an exception

in method 'ePicLoad_startDecode', argument 2 of type 'char const *'

which means passing the string to startDecode() isn't the problem, something in the C code of ePicLoad is, as this call never returns.
 

 

And that is exactly what our fix for movie selection handled. In the cpp code.

[WanWizard]

This one you mean?

https://github.com/OpenViX/enigma2/commit/2c74d42983c6969c0d2ec87b3c48622ba0ff3a45

isn't that a bit of a hack, calling a python function from C, instead of fixing it on the python side so no invalid data is passed in the first place?

 

I was looking for a cleaner solution, but my C knowledge is too limited.

 

BTW: if you have fixed it, why does OpenVIX still crash on this image? It doesn't give a green screen, but there is some exeception somewhere, as you need to kill E2 to get control of the box back after you've pressed "ok" on the image in the movie list.