epgcache overlap problems

[adm]

I assume that with more than one tuner that the EPG will be populated automatically by the use of other tuner in the background (if not actively being used for a recording). Is this correct? How often does it automatically fetch the EPG, and when? On coming out of deep standby or some set time? For example, if away on holiday for two or three weeks and the box doesn’t wake up from deep standby for a recording will this feature automatically wake up the box to fetch the EPG or does the user have to set up some form of wake up timer so that this feature works reliably?

Currently I have a repeat timer that wakes up the box for a IEPG recording at a time that I don’t usually use the box. The location for the recording is the trash folder and the box remains on IEPG for at least 10 minutes. This seems to work well except very occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced) BUT only for those 10 minutes at the ends of the listing. I have Freesat EPG disabled but I do have EIT enabled because I also have terrestrial tuners and want over the air EPG for Freeview.

 

[adm]

except very occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced) BUT only for those 10 minutes at the ends of the listing.

An example
The double entry on BBC ONE (SD and HD) for "Election" and "BBC News". Note: even though my bouquet is labelled fav_freesat that is just a name I haven't changed when I changed to Sky FTA in ABM and disabled Freesat EPG in the appropriate settings.

 

[BrokenUnusableAccount]

occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced) BUT only for those 10 minutes at the ends of the listing. I have Freesat EPG disabled but I do have EIT enabled because I also have terrestrial tuners and want over the air EPG for Freeview.

Me too.
Never seems to happen with Freeview, just Sky EPG.

Presumably it's the now/next disagreeing with IEPG 1 data from some time ago, but I don't really know.

 

[adm]

... except very occasionally I get an entry in the Sky FTA EPG data that appears to have conflicting information as though the program start/end times have been changed by 5 to 10 minutes and the original EPG (IEPG?) data has been overwritten (rather than replaced).

An example
The double entry on BBC ONE (SD and HD) for "Election" and "BBC News". Note: even though my bouquet is labelled fav_freesat that is just a name I haven't changed when I changed to Sky FTA in ABM and disabled Freesat EPG in the appropriate settings.

View attachment 61320

Hmm. since installing 5.4.005. disabling my daily timer for IEPG and using opentvzapper instead I’m now seeing a LOT more overwritten EPG entries. Just checking before posting this, and after my box has been on for 3 hours, I have 12 channels showing the problem (in a bouquet with 40 channels) whereas before I would only see the occasional one of two, and not every day.

As I've mentioned before I also have EIT enabled to obtain terrestrail EPG data as I have both types of tuners in my box.

 

[adm]

Seeing as how this just tunes the IEPG the same as using epgrefresh or directly I can't see how this would produce a different result.

I agree it should not make any difference but I wasn't using epgrefresh.
I was using a repeat timer to switch to IEPG at around 6:30am each day. It was a record timer but with the file directed directly to the trash folder. The timer ended by putting the box into standby and 10 minutes later a power timer put the box back into deep standby. This sequence ensured that the EPG was saved to disk

Did you disable Freesat?

Yes

Try disabling now and next.

I'm will experiment with a few EIT settings over the next week.

Also I don't understand how you think this is connected to terrestrial.

As I understand it, the EIT gets you the 7 day EPG on Freeview AND for now/next on satellite. As there is only a single option to enable EIT in the configuration menu then I’m assuming that enabling EIT will give both terrestrial AND satellite functionality for a box with the two types of tuner. I already know that disabling EIT kills to ability to get an over the air EPG on terrestrial Freeview.

This problem also raises another question. If I had 12 overwriten EPG entries this morning is the original EPG obtained from IEPG a few hours ago or the overwritten information obtained from maybe now/next the correct information? It one case on the taking pictures channel the film in the title EPG had been changed and another film was broadcast. Now/Next gave the correct film title so even with an overwrite of the start/stop timing other things are not overwritten, such as the description.
Is the IEPG data from sky so unreliable that it requires 12 off corrections from now/next with the same hour or two?

 

[Huevos]

Whatever process you used to zap the box to IEPG is not relevant. What happened afterwards may be relevant. I.e. does the save to epg.dat and then sleep do some sort filtering of the data?

 

[adm]

Whatever process you used to zap the box to IEPG is not relevant. What happened afterwards may be relevant. I.e. does the save to epg.dat and then sleep do some sort filtering of the data?

I downloaded 5.4.005 and installed then deleted the repeat timer I had for IEPG and enabled opentvzapper. I then deleted the EPG cache before rebooting the box to see if opentvzapper was working. The Sky FTA EPG data was populated soon afterwards.

So, this mornings' observations are with only opentvzapper running for IEPG on 28.2E (and EIT enabled) so unlikely to be my previous setup.

My box was switched on from deep standby at approx 8:30 am and the overwrite of the EPG data for 12 channels noticed just before I posted at around 1pm. Even if the EPG data from yesterday was read from the hard disk on switch on opentvzapper would have kicked in shortly afterwards to update the (I)EPG data. I have two satellite tuners and no recordings were taking place at anytime during the morning.

I realise that opentvzapper is just a background channel changer (zapper) but something is causing the EPG data in RAM cache to be overwritten, and not very well. I realise also that I have seen this problem, but to a much lesser extent, before opentvzapper and even possibly before epgrefresh.

Although a different method of obtaining over the air EPG I don’t recall seeing the problem on the Freeview terrestrial EPG which also suggests the problem is not how my box saves EPG to disk on shut down nor how it reads EPG from disk after startup from deep standby, and possibly not related to how the EPG is updated if a single method is used (only EIT in the case of Freeview). My box is in deep standby every night.

As I said in a previous post, I will disable EIT or now/Nex EIT to see if the problem goes away. This may take a week or so to reach any conclusions.

 

[Huevos]

Give me a current example.

 

[adm]

Give me a current example.

I cannot do this at the current time. After changing some of the EIT configuration this afternoon I deleted the EPG cache and ran opentvzapper manually to get new EPG data. I haven't seen the problem since. However, in the past I may go a few days or more before seeing any overwriting of EPG data. This morning the problem was not just one or two channels at any one time as seen before but this time the EPG data for 12 channels had problems. Possibly just an unrelated coincidence after installing 5.4.005?

 

[bbbuk]

Could this issue be caused by corrupt EPG data in cache?

CrossEPG looks like it handles cache. Don't think EPGrefresh did (when I first looked at its code when it first came on the scene) and quick look at OpenTVZapper doesn't look like it handles cache really (but could be wrong as was a quick look).

If there is a potential for corrupt EPG cache then could it coded into OpenTVZapper that it deletes EPG (cache) before it downloads the EPG. Could it then save EPG data once its done? This could enabled/disabled by a menu option to handle cache. So if enabled it would delete EPG cache etc before download etc and if disabled (default as is now) it wouldn't do anything. User choice?

 

[BrokenUnusableAccount]

I downloaded 5.4.005 and installed then deleted the repeat timer I had for IEPG and enabled opentvzapper. I then deleted the EPG cache before rebooting the box to see if opentvzapper was working. The Sky FTA EPG data was populated soon afterwards.

So, this mornings' observations are with only opentvzapper running for IEPG on 28.2E (and EIT enabled) so unlikely to be my previous setup.

My box was switched on from deep standby at approx 8:30 am and the overwrite of the EPG data for 12 channels noticed just before I posted at around 1pm. Even if the EPG data from yesterday was read from the hard disk on switch on opentvzapper would have kicked in shortly afterwards to update the (I)EPG data. I have two satellite tuners and no recordings were taking place at anytime during the morning.

I realise that opentvzapper is just a background channel changer (zapper) but something is causing the EPG data in RAM cache to be overwritten, and not very well. I realise also that I have seen this problem, but to a much lesser extent, before opentvzapper and even possibly before epgrefresh.

Although a different method of obtaining over the air EPG I don’t recall seeing the problem on the Freeview terrestrial EPG which also suggests the problem is not how my box saves EPG to disk on shut down nor how it reads EPG from disk after startup from deep standby, and possibly not related to how the EPG is updated if a single method is used (only EIT in the case of Freeview). My box is in deep standby every night.

As I said in a previous post, I will disable EIT or now/Nex EIT to see if the problem goes away. This may take a week or so to reach any conclusions.

Maybe with Freeview there is a simple means in the received data (perhaps the EIT number mentioned in another thread) for the EPG store to automatically delete old EPG data when it gets changed.
While with IEPG data 1 perhaps there isn't such a simple way to detect that a newly received piece of EPG data replaces an old piece so both get stored. Presumably exactly identical records aren't stored twice. If the only way is to look for something like overlapping start and end times it may not be implemented, or not implemented well enough.

 

[ccs]

@adm

The attachment in post #11 looks like the duplicate occurred at midnight (ie a day boundary), and the entry you highlighted was the first programme for "today" as it was when you took the snapshot.

Do the others you've seen fit the same scenario?

(Hope it makes sense. )

 

[adm]

@adm

The attachment in post #11 looks like the duplicate occurred at midnight (ie a day boundary), and the entry you highlighted was the first programme for "today" as it was when you took the snapshot.

Do the others you've seen fit the same scenario?

(Hope it makes sense. )

Until now I haven't had a systematic approach to the problem - it's just a minor annoyance that I've noticed at various times.

Currently I'm running the box with certain EPG settings disabled and checking what is happening with the 28.2E EPG data. If the problem disappears I will re-instate the disabled configurations to see if the problems re-appear. As the problem was previously not seen every day, now a few days with a clear EPG doesn’t mean that the change in EPG setting has necessarily cured the problem. It may take some time to come to meaningful conclusions.

 

[adm]

@adm

The attachment in post #11 looks like the duplicate occurred at midnight (ie a day boundary), and the entry you highlighted was the first programme for "today" as it was when you took the snapshot.

Do the others you've seen fit the same scenario?

(Hope it makes sense. )

Other instances do not have the same scenario. Where I had 12 occurances at the same time all the problems relaeted to programs starting late morning.

 

[ccs]

.... I think we need to see when it's happening, with enough notice to be able to check if (as in post #11) the time ever was adjusted by 5 minutes.

 

[adm]

The first problem seen since i changed settings

Settings


01:35 to 2:45
Problem seen







The entry for "The Professionals" should be from 1:35 to 2:45 and not overwritten by Auto Mundial (which was the program before The Professionals AND on my other box that doesn't show the problem River Monsters is not present in the EPG.

Note: the box has been on continiously for approx 18 hours.

My other box is a Zgemma H9S with a single satelitte tuner, with opentvzapper disabled and with a repeating timer set to switch the box from deep standby to the IEPG channel each morning at 6:30am

the setting for ths box is



And the EPG is shown correctly for ITV4 at the same time as the above




I'm now going to delete the EPG cache and change a setting to see if the problem re-appers in the next day or so.

 

[adm]

After a bit of investigation I do believe the problem to be both EIT now/next and IEPG populating the same EPG data base.

If I disable all modes for getting data from IEPG, then delete the EPG cache the following setting will populate (or not) the (grid) EPG with now/next data



So, if both Enable EIT EPG and Enable OpenTV EPG are both configured to yes the (grid) EPG for Sky FTA will be populated with now/next data.

Now, if the IEPG channel is also selected then the (grid) EPG data will be updated from this second source, as well as now/next

You could argue that for SKY FTA the Enable EIT EPG option can be turned off. This would give IEPG as the single source of EPG data and/or any Internet download as the single source of data.

However, I want Enable EIT EPG to be on to get the over the air terrestrial EPG for Freeview!

People with only satellite tuners may never see the problem if EIT EPG is configured to be off BUT the question arises about having an up to date EPG for Sky FTA. If IEPG or a Internet download is only updated once a day or every 6 hours the problem I found the other day with 12 TV channels having incorrect or overwritten data may indicate that now/next is possibly more up to date with respect to program changes.

The show EIT now/next in info bar option only seems to make a difference to the information shown in the Info Bar. In some cases in the table above where now/next doesn’t populate the EPG cache the now/next information is correctly shown in the Info bar.

 

[Huevos]

The column of the table that explain what each yes/no means is missing.

 

[adm]

The column of the table that explain what each yes/no means is missing.

The first row in the table is the headings for each column, the next 8 rows show all the possible settings for the three configurable menu items.
The last column in the table is the result - the observation about now/next data appearing in the EPG cache.

It shows that in
menu -> setup -> EPG -> settings
that option " Show EIT now/next in infobar" is a don't care as it is not controlling what appers in the EPG cache, and seen in the grid EPG

Only the combination of the options "Enable EIT data" and "Enable OpenTV EPG" both being set to "Yes" result in now/next appering in the EPG cache.

"Enable EIT data" and "Enable OpenTV EPG" both being set to "Yes" is the setting I need for over the air EPG for both satellite (Sky FTA) and terrestrial (Freeview) population.

 

[Huevos]

So it turns out that previously epgcache OpenTV reader was using the event start time as the event_id. LraiZer has provided a patch to cure this which needs testing. https://github.com/OpenViX/enigma2/commit/0731baaaa16d7985cb42b003d3ec50dd5631b871

If you want a binary to use for testing I can provide an ARM version.