[OS-mio] Crash when folder contains filenames with Windows encoding

ocean · Aug 28, 2022

Huevos said:
Is it possible to check with the current code in the repo:

I'll test later..

Huevos · Aug 28, 2022

ocean said:
I said it could possible to add overloaded function, not to modify existing one.

I did try compile image from source and there were no errors, but I don't think my changes were included to image. There are no instructions how to compile enigma2 from local folder. Maybe I needed to clean something too. I modified ENIGMA2_URI in site.conf to point to git:///home/.. Could someone add some instructions to github enigma2 page?

My initial cpp and h are attached if you like to test those. Don't know if they compile or not.

Also added debug prints to original function where filename is empty or not found. I think you can commit 2 debug lines if they work, erase shouldn't normally be called with empty or non existing file.

If you get debug prints working you should see in log what filename contains in case it does not erase.

Just for your curiosity:

Code:

| components/file_eraser.cpp: In member function 'void eBackgroundFileEraser::erase(const char*)':
| components/file_eraser.cpp:87:21: error: declaration of 'std::string filename' shadows a parameter
|    87 |         std::string filename(str(filename));
|       |                     ^~~~~~~~
| components/file_eraser.cpp:85:47: note: 'const char* filename' previously declared here
|    85 | void eBackgroundFileEraser::erase(const char* filename)
|       |                                   ~~~~~~~~~~~~^~~~~~~~
| components/file_eraser.cpp:87:30: error: 'str' was not declared in this scope; did you mean 'std'?
|    87 |         std::string filename(str(filename));
|       |                              ^~~
|       |                              std
| components/file_eraser.cpp:85:47: warning: unused parameter 'filename' [-Wunused-parameter]
|    85 | void eBackgroundFileEraser::erase(const char* filename)
|       |                                   ~~~~~~~~~~~~^~~~~~~~
| In file included from ../lib/components/file_eraser.h:5,
|                  from components/file_eraser.cpp:1:
| ../lib/base/message.h: In instantiation of 'eFixedMessagePump<T>::eFixedMessagePump(eMainloop*, int, const char*) [with T = eBackgroundFileEraser::Message]':
| components/file_eraser.cpp:16:2:   required from here
| ../lib/base/message.h:187:51: warning: unused parameter 'mt' [-Wunused-parameter]
|   187 |         eFixedMessagePump(eMainloop *context, int mt, const char *name):
|       |                                               ~~~~^~
| In file included from ../lib/service/service.h:6,
|                  from ../lib/dvb/idvb.h:12,
|                  from ../lib/dvb/dvb.h:8,
|                  from components/scan.cpp:1:
| ../lib/service/iservice.h: In member function 'virtual void iServiceInformation::getAITApplications(std::map<int, std::__cxx11::basic_string<char> >&)':
| ../lib/service/iservice.h:432:69: warning: unused parameter 'aitlist' [-Wunused-parameter]
|   432 |         virtual void getAITApplications(std::map<int, std::string> &aitlist) {};
|       |                                         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~
| make[4]: *** [Makefile:2703: components/file_eraser.o] Error 1
| make[4]: *** Waiting for unfinished jobs....

Huevos · Aug 28, 2022

ocean said:
Could someone add some instructions to github enigma2 page?

Example of building the enigma2 binary. Modify paths/machine to suit what is on your server.

Paths:

Code:

enigma2 git location = /home/username/6.2/builds/developer/vusolo4k/tmp/work/vusolo4k-oe-linux-gnueabi/enigma2/enigma2-5.2+gitAUTOINC+f3fe89ab92-r0/git
enigma2 binary location = /home/username/6.2/builds/developer/vusolo4k/tmp/work/vusolo4k-oe-linux-gnueabi/enigma2/enigma2-5.2+gitAUTOINC+f3fe89ab92-r0/package/usr/bin/enigma2

Run at command line:

Code:

cd /home/username/6.2/builds/openvix/developer/vusolo4k
MACHINE=vusolo4k
MACHINEBUILD=vusolo4k
. env.source
bitbake -f -c clean enigma2
bitbake -f -c unpack enigma2

[B][COLOR="#FF0000"](transfer your modified files to "enigma2 git location")[/COLOR][/B]

bitbake -c package enigma2

[B][COLOR="#FF0000"](retrieve enigma2 binary from "enigma2 binary location")...[/COLOR][/B]

twol · Aug 28, 2022

ocean said:
I said it could possible to add overloaded function, not to modify existing one.

I did try compile image from source and there were no errors, but I don't think my changes were included to image. There are no instructions how to compile enigma2 from local folder. Maybe I needed to clean something too. I modified ENIGMA2_URI in site.conf to point to git:///home/.. Could someone add some instructions to github enigma2 page?

My initial cpp and h are attached if you like to test those. Don't know if they compile or not.

Also added debug prints to original function where filename is empty or not found. I think you can commit 2 debug lines if they work, erase shouldn't normally be called with empty or non existing file.

If you get debug prints working you should see in log what filename contains in case it does not erase.

I have run out of time to run test ...I had
initial crash but that was my issue....
modded c++ files attached...... yes I changed filename for convert

LraiZer · Aug 28, 2022

ocean said:
I did try compile image from source and there were no errors, but I don't think my changes were included to image. There are no instructions how to compile enigma2 from local folder. Maybe I needed to clean something too. I modified ENIGMA2_URI in site.conf to point to git:///home/..

I think different people have different ways to rebuild enigma2. Here is one way!

In the build-envoronment site.conf file comment out to not INHERIT "rm_work" before building.

DL_DIR = "/home/LraiZer/sources"
#INHERIT += "rm_work"

This leaves the enigma2 source code after a build to allow editing and recompile .cpp changes.

EG.
/home/lraizer/openvix/build-enviroment/builds/openvix/release/inihdp/tmp/work/xpeedlx3-oe-linux/enigma2/enigma2-6.2+gitAUTOINC+0cd13deda6-r0/temp

Edit your changes to the enigma2 .cpp source code.

Then execute the script "run.do_compile" file, this will recompile engima2 binary only with the files that have been edited, so is very fast.

Then copy the recompiled enigma2 binary to /tmp on the set top box and run it from there in debug mode to test.

For speedy edit, test, edit, test.. this process only takes about 60 seconds to edit, recompile and transfer the new enigma2 binary to STB /tmp if you put the commands to run the script and ftp binary to STB in a build.sh script

ocean · Aug 28, 2022

twol said:
modded c++ files attached...... yes I changed filename for convert

Ok, thanks, modified files look good, just needed to use diffent variable. I was not sure if it was required to cast "char *" to string, not used C++ in 10 years.

If it still doesn't like bytes, then not sure what would be the correct type for bytes?! Also could add function with different name "erase2", then error message atleast must be different

birdman · Aug 28, 2022

Huevos said:
1) it works with utf8 filenames that contain single byte chars > 127.
2) it works with utf8 filenames that contain multi byte chars.

Filenames are not utf8. They are an array of arbitrary bytes.
Until we get away from that viewpoint there will be problems.

Huevos · Aug 28, 2022

birdman said:
Filenames are not utf8. They are an array of arbitrary bytes.
Until we get away from that viewpoint there will be problems.

They are from a particular charset converted to bytes.

Huevos · Aug 28, 2022

LraiZer said:
the recompiled enigma2 binary to /tmp on the set top box and run it from there in debug mode to test.

Can you give more detail on this step?

twol · Aug 28, 2022

ocean said:
Ok, thanks, modified files look good, just needed to use diffent variable. I was not sure if it was required to cast "char *" to string, not used C++ in 10 years.

If it still doesn't like bytes, then not sure what would be the correct type for bytes?! Also could add function with different name "erase2", then error message atleast must be different

Will try tomorrow….. if current doesn,t work will try erase2 idea.
you may have not used C++ for 10 years, but there are very few people with any C++ knowledge left to help with Enigma2 changes, so your input is invaluable.

LraiZer · Aug 28, 2022

Huevos said:
Can you give more detail on this step?

The enigma2 work folder does not get removed after a build since the commented out #INHERIT += "rm_work"
Copy the recompiled engima2 binary you find in the enigma2 work folder to Set Top Box /tmp folder and chmod 755 executable.

Then run and monitor in a test enviromant on STB via telnet (i use HyperTerminal with logging)
to stop current enigma2 and run the enigma2 test binary instead:

stop current enigma2
init 4

start the enigma2 binary in /tmp with debug level 4
ENIGMA_DEBUG_LVL=4 /tmp/enigma2

stop the test engima2 binary test
CTRL+C

restart original engima2 as normal
init 3

birdman · Aug 29, 2022

Huevos said:
They are from a particular charset converted to bytes.

They are not.
I can put any bytes I want (except /and NUL) in any order I wish to into a file-name.

birdman · Aug 29, 2022

twol said:
Will try tomorrow….. if current doesn,t work will try erase2 idea.
you may have not used C++ for 10 years, but there are very few people with any C++ knowledge left to help with Enigma2 changes, so your input is invaluable.

Not sure that any C++ code changes should be needed. I'd expect that to just be handing a filename name around, not trying to (wrongly) decode/encode it.

twol · Aug 29, 2022

@Ocean, Huevos

so this is what I see when sending a byte string to the C++ code ( trying erase b'/media/hdd/movie/.Trash/20220827 1404 - ITV HD - ITV News & Weather.ts' )
Q: should a Null be added to the end of the byte string?? ------>since tried that didn't make difference
so finds both C++ functions, but dislikes the byte string going from python to C++, os stat accepts the string.
Maybe something to do with SWIG and the way our python.h or other python modules in E2 are defined?

Code:

07:56:02.2879 [RecordTimer] prepare ok, waiting for begin
07:56:02.2897 [RecordTimer] activating state 2 (Running)
07:56:02.2901 [Trashcan] probing folders
07:56:02.2935 [Notifications] AddPopup, id = None
07:56:02.2979 [Trashcan] looking in trashcan /media/hdd/movie/.Trash
07:56:02.3058 [Trashcan] /media/hdd/movie/.Trash: Size: 347,121,183
07:56:02.3063 [Trashcan] trying erase b'/media/hdd/movie/.Trash/20220827 1404 - ITV HD - ITV News & Weather.ts'
07:56:02.3102 [Screen] Warning: Skin is missing element 'icon' in <class 'Screens.MessageBox.MessageBox'>(A recording has been started:
Breakfast).07:56:02.3103
07:56:02.3117 [Pixmap] setPixmapNum(1) failed! defined pixmaps: []
07:56:02.3123 [MessageBox] Timeout set to 3 seconds.
07:56:02.3165 [eDVBServiceRecord] Recording to /media/hdd/movie/20220829 0756 - BBC One HD - Breakfast.ts...
07:56:02.3173 [eDVBServiceRecord] start recording...
07:56:02.3175 [eDVBServiceRecord] RECORD: have 1 video stream(s) (19c9), and 1 audio stream(s) (19ca), and the pcr pid is 19c9, and the text pid is ffffffff
07:56:02.3179 [eDVBServiceRecord] ADD PID: 0000
07:56:02.3181 [eDVBServiceRecord] ADD PID: 0004
07:56:02.3183 [eDVBServiceRecord] ADD PID: 19c9
07:56:02.3184 [eDVBServiceRecord] ADD PID: 19ca
07:56:02.3199 [setIoPrio] realtime level 7 ok
07:56:02.3200 [eFilePushThreadRecorder] THREAD START
07:56:02.3210 [RecordTimer] start recording on tuner: I
07:56:02.3331 [eInputDeviceInit] 0 160 (352) 1
07:56:02.3335 [eRCDeviceInputDev] emit: 0
07:56:02.3344 [InfoBarGenerics] Key: 352 (Break) KeyID='KEY_OK' Binding='('OK',)'.
07:56:02.3375 [Task] >>> Error: [<Components.Task.FailedPostcondition object at 0xb0622370>]
07:56:02.3381 [Task] job Components.Task.Job name=Cleaning Trashes #tasks=1 completed with [<Components.Task.FailedPostcondition object at 0xb0622370>] in Components.Task.Task name=Cleaning Trashes
07:56:02.3400 [Task] unrecoverable task failure
07:56:02.3404  Cleaning Trashes
Error: [Failure instance: Traceback: <class 'TypeError'>: in method 'eBackgroundFileEraser_erase', argument 2 of type 'char const *'
Additional information:
Wrong number or type of arguments for overloaded function 'eBackgroundFileEraser_erase'.
Possible C/C++ prototypes are:
eBackgroundFileEraser::erase(std::string const &)
eBackgroundFileEraser::erase(char const *)

/usr/lib/python3.10/threading.py:1009:_bootstrap_inner
/usr/lib/python3.10/threading.py:946:run
/usr/lib/python3.10/site-packages/twisted/_threads/_threadworker.py:47:work
/usr/lib/python3.10/site-packages/twisted/_threads/_team.py:182:doWork
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:244:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:260:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
/usr/lib/enigma2/python/Tools/Trashcan.py:200:work
]07:56:02.3431
07:56:02.3435 [Task] not retrying job.
07:56:03.9599 [eDVBDemux] open demux /dev/dvb/adapter0/demux0
07:56:03.9601 [eDVBDemux][eDVBSectionReader] DMX_SET_FILTER pid=18
07:56:05.3470 [MessageBox] Timeout!
07:56:05.3613 [Screen] Warning: Skin is missing element 'icon' in <class 'Screens.MessageBox.MessageBox'>(Cleaning Trashes
Error: [Failure instance: Traceback: <class 'TypeError'>: in method 'eBackgroundFileEraser_erase', argument 2 of type 'char const *'
Additional information:
Wrong number or type of arguments for overloaded function 'eBackgroundFileEraser_erase'.
Possible C/C++ prototypes are:
eBackgroundFileEraser::erase(std::string const &)
eBackgroundFileEraser::erase(char const *)

/usr/lib/python3.10/threading.py:1009:_bootstrap_inner
/usr/lib/python3.10/threading.py:946:run
/usr/lib/python3.10/site-packages/twisted/_threads/_threadworker.py:47:work
/usr/lib/python3.10/site-packages/twisted/_threads/_team.py:182:doWork
--- <exception caught here> ---
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:244:inContext
/usr/lib/python3.10/site-packages/twisted/python/threadpool.py:260:<lambda>
/usr/lib/python3.10/site-packages/twisted/python/context.py:117:callWithContext
/usr/lib/python3.10/site-packages/twisted/python/context.py:82:callWithContext
/usr/lib/enigma2/python/Tools/Trashcan.py:200:work
]).07:56:05.3614

ocean · Aug 29, 2022

twol said:
Maybe something to do with SWIG and the way our python.h or other python modules in E2 are defined?

Yes, that seems to be the issue. Somehow needs to tell SWIG it's okay to pass bytes to these functions.

Probably there are other projects with same issues after moving to py3, maybe worth to google errormessages and see if others have solved this

twol · Aug 29, 2022

So this is what I get if I use str() on byte string in python, it appears to execute... but fails as cannot find the file

13:56:26.7858 [Trashcan] root = b'/media/hdd/movie/.Trash'
13:56:26.7859 [Trashcan] name = b'20220829 1054 - BBC One HD - Morning Live.ts.sc'
13:56:26.7860 [Trashcan] fn = b'/media/hdd/movie/.Trash/20220829 1054 - BBC One HD - Morning Live.ts.sc'
13:56:26.7861 [Trashcan] trying erase b'/media/hdd/movie/.Trash/20220829 1054 - BBC One HD - Morning Live.ts.sc'
13:56:26.7861 [eBackgroundFileEraser] filename b'/media/hdd/movie/.Trash/20220829 1054 - BBC One HD - Morning Live.ts.sc' not found: Function not implemented
13:56:26.7863 [Trashcan] /media/hdd/movie/.Trash: Size now: 0

where fn is byte string before and fn1 after ---> fn1 = str(fn)
then I specifically call the routine now erase2 in C++ code

Code:

						st = stat(fn)						
						fn1 = str(fn)
						print("[Trashcan] fn = ", fn)
						print("[Trashcan] trying erase", fn1)
						enigma.eBackgroundFileEraser.getInstance().erase2(fn1)

wondering if we need this if (rename(filename.c_str(), delname.c_str())<0) with the .c_str

ocean · Aug 29, 2022

twol said:
if I use str() on byte string in python, it appears to execute

I don't think doing str() to bytes results anything useful.

However there are questions like this: https://github.com/swig/swig/issues/752

Not sure if adding that "#define SWIG_PYTHON_STRICT_BYTE_CHAR" somewhere would help

twol · Aug 29, 2022

ocean said:
I don't think doing str() to bytes results anything useful.

However there are questions like this: https://github.com/swig/swig/issues/752

Not sure if adding that "#define SWIG_PYTHON_STRICT_BYTE_CHAR" somewhere would help

I would agree except its passed to the C++ routine whereas previously the call failed

Also seen that post, but issue is it would apply all over and not sure we want to do that

Huevos · Aug 29, 2022

ocean said:
Not sure if adding that "#define SWIG_PYTHON_STRICT_BYTE_CHAR" somewhere would help

There are a lot of strings passed in and out of the c++ code. I think adding that would affect everything.

Huevos · Aug 29, 2022

LraiZer said:
The enigma2 work folder

So I am here:
/home/openvix/6.1/builds/openvix/developer/vuultimo4k/tmp/work/vuultimo4k-oe-linux-gnueabi/enigma2/enigma2-6.2+gitAUTOINC+6ff9cbe887-r0

I commented out #INHERIT += "rm_work"

Ran run.do_compile.

What folder is the rebuilt binary now in?

[OS-mio] Crash when folder contains filenames with Windows encoding

Member

Administrator

Administrator

Moderator

Attachments

V.I.P

Member

Moderator

Administrator

Administrator

Moderator

V.I.P

Moderator

Moderator

Moderator

Member

Moderator

Member

Moderator

Administrator

Administrator

Attachments

Similar threads

OpenViX Feeds Status