[OS-mega] Drive free space vanishing

[speedygonzalez]

After having flashed the box with the Willo build (which was on 4.2.030), I found that my 1TB internal hdd (supplied very recently with the box by WoS) freespace went from 99% to 33%, after having left it on for only a few days for just a few recordings.

In particular many other users are also reporting the problem, including at least one user (aceone in Techkings) running Openatv 6, see:
http://www.world-of-satellite.com/showthread.php?55917-What-is-eating-my-HDD-space
and

http://www.techkings.org/threads/is-the-edision-os-mega-the-right-one-for-me.119032/#post-680916

I have tried many tests and potential fixes, but the problem persists:
1) Redirecting /hdd/movie and timeshift to a 16GB USB stick formatted in ext4
2) Flashed 5.0.002
3) Turning off timeshift
4) Turning off swap
5) Performing a reinitialisation of the hdd, in ext4, via the menu
6) Performing a file system check on the hdd, via the menu. Like a restart this was however effective in returning the lost space, but the problem did not go away

I have found that:
a) Executing fsck -alR on the hdd, and indeed from root, via puTTY, showed no sizeable file that could possibly have explained the lost space
b) Executing fsck or hard restart / reboot from the menu are effective in reclaiming the lost space
c) Recorded programme file sizes are as expected, timeshift file sizes (that are visible by Filezilla or ls-a) do not appear excessive
d) Recording programmes, allowing timeshift to be on, or pressing the pause button on the remote all give rise to hdd freespace lost
e) On the other hand hdd freespace does not change (between restarts e.g.) by leaving the box on without recording programmes as long as timeshift is disabled and not pressing the pause button
f) It is of course not conclusive, from just reading posts and asking other kind users to do tests on their machines, but I have seen no evidence yet that any user is free from this hdd freespace loss phenomenon, if they record and delete programmes, or have timeshift on.
g) Disabling timeshift might have lowered the rate of freespace loss. At one stage I was experiencing ~8GB of loss an hour.

Many thanks for the help in advance!

Cheers!

 

[Andy_Hazza]

Try setting up without a pre done build. Do a fresh flash of ViX 5.0.1 & setup from scratch and enable logs and see what happens.


Sent from my iPhone using Tapatalk

 

[birdman]

See other thread:

http://www.world-of-satellite.com/s...g-my-HDD-space&p=442776&viewfull=1#post442776

 

[speedygonzalez]

Try setting up without a pre done build. Do a fresh flash of ViX 5.0.1 & setup from scratch and enable logs and see what happens.

Andy I have set up a clean 5.0.001 build from scratch without restore as you suggested.

I had 942.78GB free, then I recorded two BBC HD programmes, in total around one hour, which according to Filezilla were just over 2.2GB, but freespace dropped to 911.16GB!

I always read freespace off Menu / Information / Devices

Then I deleted those programmes, waited and waited and yet freespace did not increase at all, remaining steadfast at 911.16GB. The fact that no increase of freespace at all is unexpected, because in previous builds the freespace always increase after programme deletion, but not enough. The difference "might" have been caused by the fact that I stopped those recordings by the red button in the PVR menu, rather than letting those programmes finish recording, but either way the symptom is still not justifiable.

Then I executed the command

find /media/hdd -type f -size +50000k -exec ls -alh {} \;

which member ccs here kindly suggested earlier in the other thread - as expected from my previous exploration, the biggest files found were the recordings, all totalling nothing like the 30GB+ that is missing...

Then I restarted, now freespace is 940.11GB. The difference with the 942.78GB might have been caused by my setting Logging up while the programmes were recording.

I noticed, from monitoring by Filezilla, that recording file size changed only every 10:00 minutes. It would mean the system writes into a temporary workspace in the intervening time and only copy/overwrite the recording at those instances. It is conceivable that the system fails to release those temporary workspace properly.

Now having restarted, logging has started. I will record something, let it finish normally, and see what comes up.

I will also try to figure out how best to run the script:

#!/bin/sh
#
cd /proc
for l in [0-9]*/fd/*; do
( /bin/ls -l $l | grep '(deleted)$') 2>/dev/null
done

birdman kindly provided me with in the other thread.

As I mentioned in the original bug report this morning, a user running Openatv 6 is having the same problem, which probably means the problem is probably not affected by the build, but is something far more fundamental / low level within the system.

Cheers!

 

[ccs]

Don't forget that deleted programmes go into .trash for n days, which is configurable.

The script can be run via telnet, by simply issuing each command, line by line, starting with cd /proc.

(I tried it, and it found no files on my box.)

 

[speedygonzalez]

Don't forget that deleted programmes go into .trash for n days, which is configurable.

The script can be run via telnet, by simply issuing each command, line by line, starting with cd /proc.

(I tried it, and it found no files on my box.)

Thanks for letting me know. I have now disabled trash and restarted.

Cheers!

 

[speedygonzalez]

Folks, I have done more tests as discussed above.

In case there is any confusion, timeshift has never been enabled for this clean 5.0.001 build, and I did not touch the pause button.

Trash is now disabled, which I established to be the case after deleting a programme - incidentally now without trash on after programme deletion the freespace does increase (like I have been experiencing before this build), and the increase took its time (over a few minutes) - another phenomenon which is also consistent with my previous builds. so I suspect my previous builds had trash disabled.

Unfortunately, the increase in freespace after programme deletion is again coming up very short. The hdd freespace gobbling monster is alive and well!!

After rebooting and before recording, freespace was 911.16GB.

After recording the programme, the size of which I established via both Filezilla and PuTTY to be 2GB, freespace was 889.81GB, i.e. over 21GB gone.

I then ran the script shown above that Birdman kindly wrote for me, it came up with nothing.

After deleting the programme with the red button in PVR, I waited the few minutes for freespace to stabilize (which incidentally coincided with the programme files no longer visible by Filezilla), freespace became 891.74GB. Essentially over 19GB vanished

I am sure if I restart, I will get the lost freespace back. But I am going to wait, so that I do not destroy useful evidence that might help hunting this bug.

I have attached the log, unfortunately, it is gobbledegook to me...

Cheers!

 

[bbbuk]

Some commands which may help:

Displays disc usage on flash as well HDD and/or USB

df -mh

Puts box to sleep, unmounts HDD, and forces a check of disc (ensure you don't have recording or timeshift in operation - /dev/sda1 may differ also)

init 4
umount /dev/sda1
e2fsck -p -c -f /dev/sda1

With regards to debug log, the following sticks out for me:

There were a lot of the following messages...

[eDVBRecordFileThread] Waiting for I/O to complete
[eFilePushThreadRecorder] Warning: All write buffers busy

and also I found this:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do

Forgive me as I haven't read any previous threads over this but is your HDD powered itself or via USB?

 

[birdman]

Observations on the log.

(Unfortunately it has no timestamps - they've been turned back on for future release - so we can't work out when things happened in relation to others).

There are three recordings mentioned in the log.

[RecordTimer] Record RecordTimerEntry(name=\86Building Dream Homes\87, begin=Tue Mar 14 13:57:00 2017, serviceref=1:0:19:1B1C:802:2:11A0000:0:0:0:, justplay=0, isAutoTimer=0)
[RecordTimer] Record RecordTimerEntry(name=ITV Racing Live: Cheltenham..., begin=Tue Mar 14 12:57:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False)[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False}

The \86 \87 characters are "Start of selected Area" and "End of selected Area". Odd (possibly).


The presence of the first two is a little odd, given that the log appears to show enigma2 stating up at 15:56:46.
The first doesn't seem to have activated a recording.
The second does, as does the third.

There are many entries like this (which isn't good):

[eFilePushThreadRecorder] Warning: All write buffers busy
[eDVBRecordFileThread] Waiting for I/O to complete

Then this shows up as one recording stops:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do.

Plus there are lots of reports about the EPGcache, like this:

[eEPGCache] FS subtable (60280000) version changed (12) now/next (1)
[eEPGCache] FS subtable (60280000) version changed (12) now/next (1)

which may, or may not, be significant.

However, there is definitely something amiss with your storage system.
Are you using a local HDD, or a remote file-server?

 

[speedygonzalez]

The hdd is a Toshiba 1TB drive (MQ01ABD1), supplied inside the box, by WoS, a couple of weeks ago.

Cheers!

 

[speedygonzalez]

Some commands which may help:

Displays disc usage on flash as well HDD and/or USB

df -mh

Puts box to sleep, unmounts HDD, and forces a check of disc (ensure you don't have recording or timeshift in operation - /dev/sda1 may differ also)

init 4
umount /dev/sda1
e2fsck -p -c -f /dev/sda1

With regards to debug log, the following sticks out for me:

There were a lot of the following messages...

[eDVBRecordFileThread] Waiting for I/O to complete
[eFilePushThreadRecorder] Warning: All write buffers busy

and also I found this:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do

Forgive me as I haven't read any previous threads over this but is your HDD powered itself or via USB?

Observations on the log.

(Unfortunately it has no timestamps - they've been turned back on for future release - so we can't work out when things happened in relation to others).

There are three recordings mentioned in the log.


The \86 \87 characters are "Start of selected Area" and "End of selected Area". Odd (possibly).


The presence of the first two is a little odd, given that the log appears to show enigma2 stating up at 15:56:46.
The first doesn't seem to have activated a recording.
The second does, as does the third.

There are many entries like this (which isn't good):


Then this shows up as one recording stops:

[eFilePushThreadRecorder] *read error* (Bad file descriptor) - aborting thread because i don't know what else to do.

Plus there are lots of reports about the EPGcache, like this:
which may, or may not, be significant.

However, there is definitely something amiss with your storage system.
Are you using a local HDD, or a remote file-server?

I am sorry bbbuk and birdman, I don't think I am adequately knowledgeable or competent to comment, but am of course more than happy to answer any question if I can. The only programme, that was recorded in this session immediately after a restart to ensure a "clean start" with trash can disabled just before 4pm, was this:

[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False}

The other recordings are old and there are other old recordings not listed for whatever reason. I am happy to delete them all if and when it helps.

bbbuk what I do know, is if I do a file system check on the hdd, then all the lost space will be recovered.

 

[bbbuk]

bbbuk what I do know, is if I do a file system check on the hdd, then all the lost space will be recovered.

I'm starting to wonder if your missing space is because of (what looks like) a storage problem especially if your missing space appears back again once HDD is checked/corrected.

Do you, by any chance, have another HDD you could try using instead just to test if that HDD is blame for missing space and the other (what looks like) storage issues?

 

[speedygonzalez]

I'm starting to wonder if your missing space is because of (what looks like) a storage problem especially if your missing space appears back again once HDD is checked/corrected.

Do you, by any chance, have another HDD you could try using instead just to test if that HDD is blame for missing space and the other (what looks like) storage issues?

I haven't got a free hdd at the moment, but I do have a 32GB usb stick, but it might be too small? And do I have to rip the internal hdd out?

A data point that might suggest the issue lies elsewhere, is that users at the other thread here indicate their different disks (of 320GB, 500GB and 1TB) are having the same issue?

Is there any other similar angle/tool applicable, apart from birdman's script he kindly provided and reproduced above for "file deleted but not closed", that might be able to identify the "orphaned" hdd space either when it was still being created/filled, or after it was created/discarded?

I guess I am wondering if the space was ever a "file", because if it was, shouldn't it have been spotted by e.g. ls -alR during recording, e.g., while it was still in use and before it was deleted?

I therefore also wonder how many ways are there to "take possession" of hdd space in linux, and what code can spot them all.

My apologies in advance, if my train of thought makes no sense...

Cheers!

 

[kokojnr]

I'm starting to wonder if your missing space is because of (what looks like) a storage problem especially if your missing space appears back again once HDD is checked/corrected.

Do you, by any chance, have another HDD you could try using instead just to test if that HDD is blame for missing space and the other (what looks like) storage issues?

Either the HDD or the Receiver, because this issue seems to affect only users of EDISION MEGA, including myself.

Sent from my Moto G (4) using Tapatalk

 

[judge]

Still don't get the issue? or the paranoia about HDD space being used up?

 

[birdman]

I guess I am wondering if the space was ever a "file", because if it was, shouldn't it have been spotted by e.g. ls -alR during recording, e.g., while it was still in use and before it was deleted?

It's perfectly legal to open a new file and immediately delete it. Since you still have a file-handle to it it still exists in the file system and you can write to and read from it. Several things do this for security (if you cant access the file-handle, you can't access any of the data) and simplicity of tidying up (once the process dies - for what ever reason - the filespace gets freed).

One thing you could try (perhaps you already have?) is just restarting the GUI, rather than restarting the system.

 

[birdman]

The only programme, that was recorded in this session immediately after a restart to ensure a "clean start" with trash can disabled just before 4pm, was this:

[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False}

Not according to the log, which has:

[RecordTimer] Record RecordTimerEntry(name=ITV Racing Live: Cheltenham..., begin=Tue Mar 14 12:57:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False)
[RecordTimer] activating state 1
[RecordTimer] Found enough free space to record
[RecordTimer] Filename calculated as: '/media/hdd/movie/20170314 1257 - ITV London HD - ITV Racing Live_ Cheltenham___'

and

[RecordTimer] Record RecordTimerEntry(name=\86Celebrity Home Secrets\87, begin=Tue Mar 14 16:27:00 2017, serviceref=1:0:19:5208:812:2:11A0000:0:0:0::ITV London HD, justplay=0, isAutoTimer=False)
...
[RecordTimer] activating state 1
[RecordTimer] Found enough free space to record
[RecordTimer] Filename calculated as: '/media/hdd/movie/20170314 1627 - ITV London HD - Celebrity Home Secrets'

Although since the restart appears to have been at 15:56 it's possible that the first one is a recording that was actually running when you restarted the box, so you should have had a prompt about that - (how do you restart your box?) - and when enigma2 restarted it found that it had a timer recording that hadn't yet reached it's end time.

 

[speedygonzalez]

It's perfectly legal to open a new file and immediately delete it. Since you still have a file-handle to it it still exists in the file system and you can write to and read from it. Several things do this for security (if you cant access the file-handle, you can't access any of the data) and simplicity of tidying up (once the process dies - for what ever reason - the filespace gets freed).

One thing you could try (perhaps you already have?) is just restarting the GUI, rather than restarting the system.

Thanks for the explanation regarding the file handling. I have just restarted the GUI, it makes no difference to the lost hdd space.

Not according to the log, which has:

and

Although since the restart appears to have been at 15:56 it's possible that the first one is a recording that was actually running when you restarted the box, so you should have had a prompt about that - (how do you restart your box?) - and when enigma2 restarted it found that it had a timer recording that hadn't yet reached it's end time.

Birdman you are good! You are absolutely right regarding the 2nd recording, because now looking at Filezilla I can see the Cheltenham racing one finishing at 4:35pm, I must have programmed it earlier and forgot it was there, and it never crossed my mind that the recording would automatically restart in later sessions (programme officially ran from 12:57 to 16:35 according to the timer record), and it therefore continued to record for about 36 minutes, consuming 2.2GB, after the restart during the logged session. Sorry for forgetting - I was picking random HD channels to make test records, and lost track of them...

There were earlier 'Cheltenham racing' recordings but they were manually deleted before the restart. So for completeness/correctness, 4.2GB associated with 2 actual programmes were recorded during the logged session. I deleted one of the programmes, 'Celebrity Home Secrets', 2GB in size, to find hdd space of around 17GB remained missing.

I have always restarted the box by Menu / Standby and Restart / Restart

Is the code that should have released the lost hdd space likely part of Enigma/OpenVix, or lower level within the system (e.g. in machine code, or part of the disc controller firmware)?

Cheers!

 

[ccs]

What does the telnet command df -T produce?

There is still the possibility that the hdd is faulty, looking thru' earlier posts.

 

[ljmramst]

Still don't get the issue? or the paranoia about HDD space being used up?

Having only 32% space left on a 500gb hdd (Seagate installed by WOS) after only setting up epg and making a test recording of approx 2 minutes. The only way to retrieve lost space is to cut power every time.

That's my problem in a nutshell and why I started the other thread. Speedygonzalez stated on that thread he opened this one as he believed it was the proper way to gain official support.

I wouldn't call it 'paranoia' when the box is virtually useless as a recording device in its current state.