[Zgemma H7] Weird EPG errors. EPG Refresh. Recent versions of OpenViX e.g. 5.4.008.

[BrokenUnusableAccount]

Try this out and see how it fares.

I will, unless I realise it's wrong before tonight.

Everyone: I made the debug log files available at the bottom of message #99.
Since I seem to be incapable of explaining what I think is going on without confusing myself and getting it wrong it might be best to look at them and try and understand that way if you want to understand.

They're logs from starting with an empty EPG and manually zapping to IEPG data 1 for 3+ minutes.

 

[LraiZer]

Quick initial thoughts on this? Thanks for debugging this Brian, are you saying we may only need to decrement the mjd date if start seconds are sent greater than 1 day and the bugus times will then be correct with no adjustment required to the sent start seconds?

// if start seconds are sent greater than 1 day (86400)
// we need to decrement startTimeBcd by 2 days (172800)

if (startSeconds > 86400)
	startTimeBcd -= 172800;

 

[BrokenUnusableAccount]

Quick initial thoughts on this? Thanks for debugging this Brian, are you saying we may only need to decrement the mjd date if start seconds are sent greater than 1 day and the bugus times will then be correct with no adjustment required to the sent start seconds?

// if start seconds are sent greater than 1 day (86400)
// we need to decrement startTimeBcd by 2 days (172800)

if (startSeconds > 86400)
	startTimeBcd -= 172800;

No.
Some of what I posted late last night, well I was living up to my handle. I was confused.

Today I'm saying as far as I can see

		if (startSecond >= 86400)
		{
			startSecond -= (0x20000 - 86400);
			--startMjd; // Interpret weird time beyond 86400 seconds as previous day
		}

should always get you back to sanity.

I guess maybe this would do it too:

if (startSeconds >= 86400)
	startTimeBcd -= 0x20000;

To be more cautious you could instead just throw away any record where startSecond >= 86400.

 

[ccs]

You can maintain old EPG data for as long as 12 hours in epg/settings.

It might at least help seeing where the phantoms are coming from without staying up too late.

 

[birdman]

It may not be programs spanning midnight being the problem but 11pm - note the +1hr

This, of course, means that in one timezone they are in one day whilst in a different timezone they are in a different day.
So the sent EPG contains a "standard" Mjd, requiring a -ve seconds offset?

 

[Joe_90]

The start_time in the schedule would/should always be coded as UTC. Any timezone (or DST) offset should be coded in the TOT in order to allow the software calculate and present the data correctly in the local time in the EPG viewed by the user.

 

[birdman]

The start_time in the schedule would/should always be coded as UTC. Any timezone (or DST) offset should be coded in the TOT in order to allow the software calculate and present the data correctly in the local time in the EPG viewed by the user.

That's what I think too. But I also suspect that someone provides the data and someone else converts this into what gets transmitted, and it's not impossible that somewhere in between there is a mismatch of intent. Particularly as I doubt the original data uses Mjd format.

 

[LraiZer]

I guess maybe this would do it too:

if (startSeconds >= 86400)
	startTimeBcd -= 0x20000;

This correctly fixed the many bogus events i was seeing on the Gemporia Craft channel. Looks good to me, thanks! Maybe you should submit a patch with your fix?

 

[BrokenUnusableAccount]

This correctly fixed the many bogus events i was seeing on the Gemporia Craft channel. Looks good to me, thanks! Maybe you should submit a patch with your fix?

Okay. I will.

 

[abu baniaz]

Thanks. Added here
https://github.com/OpenViX/enigma2/commit/2045f5ba782ae4c862f4fffa29539180660df1a6

Pull request submitted to PLi
https://github.com/OpenPLi/enigma2/pull/2889

 

[Joe_90]

That's what I think too. But I also suspect that someone provides the data and someone else converts this into what gets transmitted, and it's not impossible that somewhere in between there is a mismatch of intent. Particularly as I doubt the original data uses Mjd format.

The DVB Standard requires the MJD format. I extracted that info and posted it earlier in the thread:

start_time: This 40-bit field contains the start time of the event in Universal Time, Co-ordinated (UTC) and Modified
Julian Date (MJD) (see annex C). This field is coded as 16 bits giving the 16 LSBs of MJD followed by 24 bits coded as
6 digits in 4-bit Binary Coded Decimal (BCD). If the start time is undefined (e.g. for an event in a NVOD reference
service) all bits of the field are set to "1".
EXAMPLE 1:
93/10/13 12:45:00 is coded as "0xC079124500".
duration: A 24-bit field containing the duration of the event in hours, minutes, seconds. format: 6 digits,
4-bit BCD = 24 bit.
EXAMPLE 2:
01:45:30 is coded as "0x014530".

 

[birdman]

The DVB Standard requires the MJD format. I extracted that info and posted it earlier in the thread:

Precisely.
But the original data provider will not be providing the time in Mjd format, so some other entity will be doing a conversion somewhere.
That conversion code may be doing odd things with timezone manipulation. We don't know, but it could explain what is being seen.

 

[adm]

Precisely.
But the original data provider will not be providing the time in Mjd format, so some other entity will be doing a conversion somewhere.
That conversion code may be doing odd things with timezone manipulation. We don't know, but it could explain what is being seen.

Is this the correct fix? Where does the incorrect start seconds data come from?

As far as I’m aware the ghost/duplicate entries are for programs that have been previously broadcast and on the day they were broadcast must have had the correct EPG data suggesting the correct startMJD and correct startseconds. Where has the incorrect startseconds now come from? Probably not from the broadcaster as others have reported that other means for fetching the IEPG data don’t show these types of problem.

Also, in my experience, when the problem occurs a re-fetch of the IEPG doesn’t correct the problem UNLESS the EPG cache (and epg.dat file) are first deleted, and often by stopping Enigma to accomplish the deletion. If the EPG cache and epg.dat are deleted and the EPG data then populates from the IEPG then the EPG appears to be problem free suggesting that raw IEPG data wasn’t the cause of an incorrect startseconds value.

 

[ronand]

The raw data is fine. It's just some entries got incorrectly processed by the reader to 36 hours later than they should. By the time you notice the incorrect entries it looks they come from past programs.

 

[adm]

The raw data is fine. It's just some entries got incorrectly processed by the reader to 36 hours later than they should.

That's my point. If the raw data is fine the bug is before the point where the value of startseconds exceeding the number of seconds in a day is being used to establish the start time.
The bug fix above is at a point where the incorrect (startseconds) data is being used and not where it is being incorrectly assigned.

 

[Joe_90]

Precisely.
But the original data provider will not be providing the time in Mjd format, so some other entity will be doing a conversion somewhere.
That conversion code may be doing odd things with timezone manipulation. We don't know, but it could explain what is being seen.

I'm not following your train of thought. What other entities/conversions are involved? The raw DVB table information would be created by Redbee (Freesat) or whoever is acting as the media company for Sky and is placed in the satellite or terrestrial datastream. If the data on the stream was in error, then Freesat receivers or Sky boxes would have the same problems of ghost data, perhaps?

 

[birdman]

That's my point. If the raw data is fine the bug is before the point where the value of startseconds exceeding the number of seconds in a day is being used to establish the start time.
The bug fix above is at a point where the incorrect (startseconds) data is being used and not where it is being incorrectly assigned.

So you are suggesting that the problem is in the data being written out to epg.dat since it goes away when that data is first deleted?

But if that were the case why does it only affect data from OpenTV and why does fixing the OpenTV-specific code fix the problem?

Does anyone actually have the raw data to confirm that it is fine?

EDIT: Hmmmm, I see what you mean.
#
The DVB standard note above mentions:

93/10/13 12:45:00 is coded as "0xC079124500".

but this section of code is getting that 6-BCD digit field as a 16-bit field of "2s units".
So something else has been doing that.

 

[LraiZer]

OpenTV is not DVB standard.

Test channel: Gemporia Craft
Test program: Sewing Street / Yarn Lane REPEATS

Without the patch this channel still continues to have bogus events for this program after deleting epg.dat and without performing any saving/loading of the epg.dat file

With the patch this frequently bogus program is now fixed, I've added a widget to the skin to also show the event ID's so you can match the corrections made in the screenshots.

These are not old programs that can fixed by deleting epg.dat and clearing cache as some suggest, these are current/future programs that appear wrongly offset by 2^17. I've seen up to 6 future bogus events for this program on this channel over last few days.

 

[Huevos]

Gordon, your right the code makes no logical sense.

startSecond having a value greater than 86400 is just an indicator that the data is broken.

And the acquired knowledge is when the data is broken it is always 0x20000 seconds too far in the future so the fix is to just subtract that.

The data in startMjd must be broken too, because, the maximum value startSecond could ever have is 0x20000 and this is always the value that needs subtracting irespective of how much over 86400 the value is.

 

[adm]

So you are suggesting that the problem is in the data being written out to epg.dat since it goes away when that data is first deleted?
.

No I'm suggesting possibly more than one problem

Consider this scenario.

I start with no EPG data having stopped Enigma and deleted the epg.dat file and the RAM cache.

The box now gets its EPG data via IEPG/OpenTV

The box has EPG data for days A, B, C, D, E, F and G

Because of the “bug” some information from day A gets inserted into day B or day C, 36 hours later – but I don’t notice this at the time. The bug would also created holes in the EPG for day A. The same source data cannot establish two different time slots 36 hours apart. I haven’t noticed holes in the EPG for the current day, especially as they would be very noticeable with so may ghost/double entries in the future which should have created the equivalent number of holes.

On day B I turn on my box and notice the ghost/duplicate entries in the EPG. I now decide to update the source data from IEPG/OpenTV.

I’m now fetching data for days B, C, D, E, F, G and H – day A is already history so buggy data from day A cannot create ghost/duplicate data in the EPG for day B. [Edit: or can it]

However, the ghost/duplicate entries in day B remains! The EPG is not being overwritten with anything new for day B

Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correct. The content of this forced EPG data must be identical to that of the forced EPG request issued before the epg.dat and epg cache deletion but then it didn’t result in the EPG data being corrected.

Doesn’t this suggest that not only is there a bug in the way the start time is (was) being calculated in that the a ghost time is generated 36 hours in the future but also a bug that prevents more up to date (and correct) source data from the broadcaster also updating a previously corrupted EPG cache?

As has been pointed out if branded sky and freesat boxes don’t have this problem, nor does fetching IEPG in a different way, then the broadcasters are not sending the equivalent of a startsecond value greater than the number of seconds in a day. Startseconds being such a large value is the bug and the fix should be to find out why rather than compensating for the problem at a later stage in processing. Bugs fixed in the wrong place have a nasty habit of returning with other symptoms.