[OS-mio] Crash when folder contains filenames with Windows encoding

[birdman]

name is str in getItemDisplayName. But what is IMPORTANT here, "filename" is still intact inside str, it's just in different format.

A Python str is an array of Unicode points. How did you get a filename in there that might not have been valid utf8 in the first place?

 

[Huevos]

A Python str is an array of Unicode points. How did you get a filename in there that might not have been valid utf8 in the first place?

I don't know what you mean "How did you". We are working on the code in the repo. We haven't done anything to create that behaviour. It was already there.

 

[ocean]

I got it working! Attatched all modifications I currently have and files are playing fine! (service.cpp function is copy/pasted, sorry)

Also have very good news. It's NOT required to change any python functions to use bytes. Escaped characters in str can be handled using typemaps.

 

[Huevos]

Can the copy/paste be simplified?

 

[Huevos]

It's NOT required to change any python functions to use bytes. Escaped characters in str can be handled using typemaps.

Does that mean we don't need the changes in MovieList at all?

 

[ocean]

Does that mean we don't need the changes in MovieList at all?

MovieList still has to be changed. I posted change earlier today and zip has this change included.

 

[ocean]

Can the copy/paste be simplified?

Yes, but I don't have it. I tried few things, but it did not compile. What works in file_eraser.cpp does not work in service.cpp

 

[Huevos]

MovieList still has to be changed. I posted change earlier today and zip has this change included.

We already committed this: https://github.com/OpenViX/enigma2/commit/6efb844aba07357c3285e120bde4dc048e2bbad8

What about the copy/paste? Is there a nicer way to do that? https://github.com/OpenViX/enigma2/commit/e3474fc6ae016b563ad0e56073f27c17b8d13785

Edit: sorry, it seems we posted at more or less the same time.

 

[ocean]

We already committed this: https://github.com/OpenViX/enigma2/commit/6efb844aba07357c3285e120bde4dc048e2bbad8

Yes, but that change also affected MovieSelection.py, because there is "from Components.MovieList import getItemDisplayName".

movielist.py change was only intended for 3 cases "data.txt" in movielist.py, so I changed movielist.py little.

What about the copy/paste? Is there a nicer way to do that?

I also googled, but did not find obvious ways to do that. It seems it's not possible to call constructor from outside.

Also it should be possible to add these typemaps to original functions, then these extra char* functions would not be needed. (So far attempt has been unsuccessful)

 

[Huevos]

So the problem with the copy/paste version is we now have duplicate code which would need maintaining in synchronisation. Bit of a nightmare.

eServiceReference::eServiceReference(const char* string2)
{
	std::string string(string2);
	eDebug("[eServiceReference][char]");
	eServiceReference(string);
}

| service/service.cpp: In constructor 'eServiceReference::eServiceReference(const char*)':
| service/service.cpp:121:26: warning: unnecessary parentheses in declaration of 'string' [-Wparentheses]
|   121 |         eServiceReference(string);
|       |                          ^~~~~~~~
| service/service.cpp:121:26: note: remove parentheses
|   121 |         eServiceReference(string);
|       |                          ^~~~~~~~
|       |                          -      -
| service/service.cpp:121:27: error: conflicting declaration 'eServiceReference string'
|   121 |         eServiceReference(string);
|       |                           ^~~~~~

So looks like it doesn't understand we are calling the function.

Is this problem caused because someone thought it would be a good idea to give the class and the function the same name?

 

[Huevos]

Ok, playing a bit more...

In iservice.h:

	eServiceReference(int type, int flags, const std::string &path)
		: type(type), flags(flags), path(path)
	{
		memset(data, 0, sizeof(data));
		number = 0;
	}
+	[B]void eServiceReferenceBase(const std::string &string);[/B]
	eServiceReference(const std::string &string);
	eServiceReference(const char* string2);
	std::string toString() const;
	std::string toCompareString() const;

In service.cpp:

	}
	return res;
}

-[B]eServiceReference::eServiceReference(const std::string &string)[/B]
+[B]void eServiceReference::eServiceReferenceBase(const std::string &string)[/B]
{
	const char *c = string.c_str();
	int pathl = 0;

Then add this in service.cpp:

eServiceReference::eServiceReference(const std::string &string)
{
	eDebug("[eServiceReference][std]");
	eServiceReferenceBase(string);
}

eServiceReference::eServiceReference(const char* string2)
{
	std::string string(string2);
	eDebug("[eServiceReference][char]");
	eServiceReferenceBase(string);
}

 

[birdman]

I don't know what you mean "How did you". We are working on the code in the repo. We haven't done anything to create that behaviour. It was already there.

It's an anonymous you. I could have written "How did one".

I'm intrigued as to how bytes end up in a str given things like:

 >>> bstr = b'\x86\x87\x88'
>>> ustr = bstr.decode() 
Traceback (most recent call last): 
  File "<stdin>", line 1, in <module> 
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x86 in position 0: invalid start byte

 

[ocean]

I'm intrigued as to how bytes end up in a str given things like:

 >>> bstr = b'\x86\x87\x88'
>>> ustr = bstr.decode() 
Traceback (most recent call last): 
  File "<stdin>", line 1, in <module> 
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x86 in position 0: invalid start byte

bstr = b'\x86\x87\x88'
ustr = bstr.decode('UTF-8', 'surrogateescape')

Ofcourse this is python only string format. When this string is sent to SWIG, it sees escaped characters and throws exception. But they can be converted back to bytes using typemap.

 

[Huevos]

Ok, playing a bit more...

In iservice.h:

	eServiceReference(int type, int flags, const std::string &path)
		: type(type), flags(flags), path(path)
	{
		memset(data, 0, sizeof(data));
		number = 0;
	}
+	[B]void eServiceReferenceBase(const std::string &string);[/B]
	eServiceReference(const std::string &string);
	eServiceReference(const char* string2);
	std::string toString() const;
	std::string toCompareString() const;

In service.cpp:

	}
	return res;
}

-[B]eServiceReference::eServiceReference(const std::string &string)[/B]
+[B]void eServiceReference::eServiceReferenceBase(const std::string &string)[/B]
{
	const char *c = string.c_str();
	int pathl = 0;

Then add this in service.cpp:

eServiceReference::eServiceReference(const std::string &string)
{
	eDebug("[eServiceReference][std]");
	eServiceReferenceBase(string);
}

eServiceReference::eServiceReference(const char* string2)
{
	std::string string(string2);
	eDebug("[eServiceReference][char]");
	eServiceReferenceBase(string);
}

By the way, I forgot to say, this does actually work.

 

[Huevos]

So I have added this, based on ocean's pattern:
https://github.com/OpenViX/enigma2/commit/2c74d42983c6969c0d2ec87b3c48622ba0ff3a45

 

[Huevos]

MovieList still has to be changed. I posted change earlier today and zip has this change included.

So what goes wrong without the MovieList change now we have the new code in iservice.h/service.cpp? Can you test with the old MovieList.py and new cpp code please?

 

[birdman]

bstr = b'\x86\x87\x88'
ustr = bstr.decode('UTF-8', 'surrogateescape')

Ofcourse this is python only string format. When this string is sent to SWIG, it sees escaped characters and throws exception. But they can be converted back to bytes using typemap.

Seems a convoluted (and hence error prone) way to keep data when it is always actually a bytes format.

 

[Huevos]

Seems a convoluted (and hence error prone) way to keep data when it is always actually a bytes format.

We are just working with what python/SWIG developers dropped in our laps.

 

[twol]

Seems a convoluted (and hence error prone) way to keep data when it is always actually a bytes format.

Huevos is correct its what the python 3 Developers have given us, bless them!
Also there are bytes and bytes - with Unicode you have be careful with text files and their manipulation, as a unicode character can in theory occupy upto 4 bytes in utf-8, so memory manipulation can typically occupy 2 x memory compared to python 2 and be slower with older python 2 methods of manipulation.

 

[ocean]

So what goes wrong without the MovieList change now we have the new code in iservice.h/service.cpp? Can you test with the old MovieList.py and new cpp code please?

Without movielist changes it crashes same way as in post #1.

data.txt is inside larger object/structure, it's not a function parameter. Typemaps can only handle function parameters.

It should also be easy to test yourself too. Take any videofile, check that it plays fine. Edit filename to contain b'\xe4' for example.