[OS-mio] Crash when folder contains filenames with Windows encoding

[ocean]

Crash happens in elistboxcontent.cpp, line 920

const char *string = (PyUnicode_Check(pstring)) ? PyUnicode_AsUTF8(pstring) : "<not-a-string>";

"pstring" contains escaped characters and PyUnicode_AsUTF8(pstring) fails. Changed it to this bellow and it doesn't crash anymore. (Note: there are atleast 5 similar checks in this file, possibly they all should be fixed same way)

const char *string = (PyUnicode_Check(pstring)) ? PyBytes_AsString(PyUnicode_AsEncodedString(pstring, "utf-8", "surrogateescape")) : "<not-a-string>";

But that only fixes the crash. GUI expects UTF-8. Movielist change "decode(encoding)" is still needed to display latin-1 characters correctly.

 

[Huevos]

How do you trigger this crash?

Are the current changes in enigma repo of service.cpp working for you?

 

[twol]

Crash happens in elistboxcontent.cpp, line 920

const char *string = (PyUnicode_Check(pstring)) ? PyUnicode_AsUTF8(pstring) : "<not-a-string>";

"pstring" contains escaped characters and PyUnicode_AsUTF8(pstring) fails. Changed it to this bellow and it doesn't crash anymore. (Note: there are atleast 5 similar checks in this file, possibly they all should be fixed same way)

const char *string = (PyUnicode_Check(pstring)) ? PyBytes_AsString(PyUnicode_AsEncodedString(pstring, "utf-8", "surrogateescape")) : "<not-a-string>";

But that only fixes the crash. GUI expects UTF-8. Movielist change "decode(encoding)" is still needed to display latin-1 characters correctly.

Is this crash still from movielist or another location?
if another issue how did you get there

 

[ocean]

How do you trigger this crash?

That is the original crash without any changes to movielist. (You asked to test with old movielist)

Are the current changes in enigma repo of service.cpp working for you?

Yes, service.cpp changes are working fine.

 

[Huevos]

@ocean, what are you using to FTP the file to the STB?

Is it possible that FTP is interfering with the filename. Windows is supposed to save filenames as UTF16.

 

[ocean]

Is it possible that FTP is interfering with the filename. Windows is supposed to save filenames as UTF16.

No, it's not only FTP issue. If files are saved to USB drive in Windows. When you plug that USB drive to box, it has same issues.

But with current changes in repo it's already working. Just include latest movielist changes I posted earlier, then it doesn't affect anything in movieselection.py

 

[birdman]

But that only fixes the crash. GUI expects UTF-8. Movielist change "decode(encoding)" is still needed to display latin-1 characters correctly.

Linux filenames have no concept of Latin-1 (or of iso8859-1, Windows-1252, utf8 etc.) They are just an array of bytes.
The filename has to be kept (and passed around) as bytes while any display has to use a utf8 representation of this.

 

[Huevos]

Linux filenames have no concept of Latin-1 (or of iso8859-1, Windows-1252, utf8 etc.) They are just an array of bytes.
The filename has to be kept (and passed around) as bytes while any display has to use a utf8 representation of this.

Gordon, we are talking about GUI display. We have to feed something to the GUI that it understands.

 

[birdman]

Gordon, we are talking about GUI display. We have to feed something to the GUI that it understands.

Yes. But you also need to retain the original name in bytes in order to pass it to the filesystem.
If you start with a filename you should only ever need to decode it (in some lossless way) to produce a display value.
If you start with a display string (such as setting up a timer/Autotimer) then you only ever need to encode it to get a filename.

So if you ever find yourself encoding and decoding the same value then something has gone wrong in the logic.

 

[Huevos]

The current code works so far but open to any improvements you suggest.

 

[ocean]

@Huevos Could you still add some fix for RedirectOutput.py? There are debug prints, one in navigation.py, line 110, print("[Navigation] playing ref", ref and ref.toString())

If you don't like try/except, maybe change this:

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")

To this in RedirectOutput.py?

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")
		else:
			data = data.encode("UTF-8", "ignore").decode()

Everything else seems working for me.

 

[Huevos]

@Huevos Could you still add some fix for RedirectOutput.py? There are debug prints, one in navigation.py, line 110, print("[Navigation] playing ref", ref and ref.toString())

If you don't like try/except, maybe change this:

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")

To this in RedirectOutput.py?

		if isinstance(data, bytes):
			data = data.decode(encoding="UTF-8", errors="ignore")
		else:
			data = data.encode("UTF-8", "ignore").decode()

Everything else seems working for me.

That crash was caused by something nasty in RecordTimer.py...

< 4037.8571> 17:38:38.4681 Traceback (most recent call last):
< 4037.8572> 17:38:38.4681 File "/usr/lib/enigma2/python/StartEnigma.py", line 224, in processDelay
< 4037.8581> 17:38:38.4690 callback(*retval)
< 4037.8581> 17:38:38.4691 File "/usr/lib/enigma2/python/Screens/InfoBarGenerics.py", line 3333, in recordQuestionCallback
< 4037.8585> 17:38:38.4694 File "/usr/lib/enigma2/python/Screens/InfoBarGenerics.py", line 3253, in startInstantRecording
< 4037.8588> 17:38:38.4697 File "/usr/lib/enigma2/python/RecordTimer.py", line 1275, in record
< 4037.8591> 17:38:38.4701 File "/usr/lib/enigma2/python/timer.py", line 229, in addTimerEntry
< 4037.8594> 17:38:38.4704 File "/usr/lib/enigma2/python/timer.py", line 268, in calcNextActivation
< 4037.8597> 17:38:38.4706 File "/usr/lib/enigma2/python/timer.py", line 363, in processActivation
< 4037.8601> 17:38:38.4710 File "/usr/lib/enigma2/python/RecordTimer.py", line 1009, in doActivate
< 4037.8607> 17:38:38.4717 File "/usr/lib/enigma2/python/RecordTimer.py", line 676, in activate
< 4037.8611> 17:38:38.4720 File "/usr/lib/enigma2/python/RecordTimer.py", line 485, in log_tuner
< 4037.8614> 17:38:38.4723 File "/usr/lib/enigma2/python/RecordTimer.py", line 284, in log
< 4037.8617> 17:38:38.4726 File "/usr/lib/enigma2/python/Tools/RedirectOutput.py", line 16, in write
< 4037.8620> 17:38:38.4729 TypeError: in method 'ePythonOutput', argument 1 of type 'char const *

What exactly is that bad output?

 

[Huevos]

Python:

ePythonOutput(self.line, self.level)

C++:

void ePythonOutput(const char *, int lvl = lvlDebug);

const char * ... shouldn't we be feeding that bytes?

 

[twol]

Python:

ePythonOutput(self.line, self.level)

C++:

void ePythonOutput(const char *, int lvl = lvlDebug);

const char * ... shouldn't we be feeding that bytes?

Thats in eerror.h but its also in eerror.cpp

 void ePythonOutput(const char *string, int lvl)

Which confuses me - why in both

 

[ocean]

... shouldn't we be feeding that bytes?

It's possible to change that to take bytes. But do you want log files to be other than UTF-8? In receiver menu there is option to view log, it most likely crashes if log is not UTF-8

 

[ocean]

What exactly is that bad output?

That no longer happens and problem was debug print in traschan.py, it contained filename. I don't know why traceback said "RecordTimer.py", nothing was wrong there.

Current debug log crash is this and it just tries to print "playing.." and filename

<  1151.2274> 19:04:31.2122 Traceback (most recent call last):
<  1151.2275> 19:04:31.2123   File "/usr/lib/enigma2/python/StartEnigma.py", line 224, in processDelay
<  1151.2282> 19:04:31.2130     callback(*retval)
<  1151.2283> 19:04:31.2131   File "/usr/lib/enigma2/python/Screens/InfoBar.py", line 223, in movieSelected
<  1151.2291> 19:04:31.2139   File "/usr/lib/enigma2/python/Screens/InfoBar.py", line 228, in openMoviePlayer
<  1151.2295> 19:04:31.2143   File "/usr/lib/enigma2/python/StartEnigma.py", line 314, in open
<  1151.2301> 19:04:31.2149     dlg = self.current_dialog = self.instantiateDialog(screen, *arguments, **kwargs)
<  1151.2302> 19:04:31.2150   File "/usr/lib/enigma2/python/StartEnigma.py", line 251, in instantiateDialog
<  1151.2306> 19:04:31.2155     return self.doInstantiateDialog(screen, arguments, kwargs, self.desktop)
<  1151.2307> 19:04:31.2155   File "/usr/lib/enigma2/python/StartEnigma.py", line 274, in doInstantiateDialog
<  1151.2312> 19:04:31.2160     dlg = screen(self, *arguments, **kwargs)
<  1151.2312> 19:04:31.2161   File "/usr/lib/enigma2/python/Screens/InfoBar.py", line 289, in __init__
<  1151.2315> 19:04:31.2164   File "/usr/lib/enigma2/python/Navigation.py", line 110, in playService
<  1151.2318> 19:04:31.2166   File "/usr/lib/enigma2/python/Tools/RedirectOutput.py", line 16, in write
<  1151.2322> 19:04:31.2170 TypeError: in method 'ePythonOutput', argument 1 of type 'char const *'
Additional information:
Wrong number or type of arguments for overloaded function 'ePythonOutput'.

 

[birdman]

That crash was caused by something nasty in RecordTimer.py...?

Trying to log something.

The logging code should really trap this in case the caller doesn't (as they clearly all do not).

 

[birdman]

Current debug log crash is this and it just tries to print "playing.." and filename

And that goes via a print statement to the log. If filename is not valid utf8 (and there is no reason at all why it has to be unless the logging code has turned it into valid Unicode) then you'll get a crash.
We reached here earlier in the thread.