[Zgemma H7] Weird EPG errors. EPG Refresh. Recent versions of OpenViX e.g. 5.4.008.

[birdman]

Gordon, your right the code makes no logical sense.

Hence the need for a comment associated with the fix.

And the raw data is not fine. It's being sent over the air in this "perverted" format by OpenTV.

 

[adm]

These are not old programs that can fixed by deleting epg.dat and clearing cache as some suggest, these are current/future programs that appear wrongly offset by 2^17. I've seen up to 6 future bogus events for this program on this channel over last few days.

If it is a current or future program incorrectly assigned to a future timeslot offset by 36 hours do you also see a hole in the EPG where it should have occurred?

 

[birdman]

Because of the “bug” some information from day A gets inserted into day B or day C, 36 hours later – but I don’t notice this at the time. The bug would also created holes in the EPG for day A. The same source data cannot establish two different time slots 36 hours apart. I haven’t noticed holes in the EPG for the current day, especially as they would be very noticeable with so may ghost/double entries in the future which should have created the equivalent number of holes.

However, let's assume that when the data is first broadcast by OpenTV it is correct.
It ends up in your EPG.
At some future point (while the programme is still running?) this field becomes corrupted.
Now you have a "new" bogus future event.

On day B I turn on my box and notice the ghost/duplicate entries in the EPG. I now decide to update the source data from IEPG/OpenTV.

I’m now fetching data for days B, C, D, E, F, G and H – day A is already history so buggy data from day A cannot create ghost/duplicate data in the EPG for day B.

It can't newly create them. But the refetch will not delete items - it (might) change events with the same eventID, but the bogus programme has now finished, so you will not get any update for it. Hence it remains in place.

Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correct

Because now you have removed the bogus entry and there is now no data to recreate it.

 

[BrokenUnusableAccount]

If it is a current or future program incorrectly assigned to a future timeslot offset by 36 hours do you also see a hole in the EPG where it should have occurred?

In all the cases I noticed it should have occurred in the past so no hole seen.

 

[adm]

In all the cases I noticed it should have occurred in the past so no hole seen.

I haven't noticed any holes either and I mostly use the grid view of the EPG where holes would be very noticeable. At some time some correct information must have populated these potential holes.

I don’t fully understand how the EPG data is managed on the boxes but my understanding is that the over the air Freesat EPG comes directly from the broadcast stream when tuned to, say, BBC1 as does the Sky FTA EPG when tuned into the IEPG channel.

The over the air data received by an Enigma box over air must be correct or else the same problem would be seen on Freesat and Sky branded boxes. It must be some software running on the Enigma box that is wrongly interpreting the broadcast EPG data and creating values of startseconds that are not sensible.

 

[Huevos]

In all the cases I noticed it should have occurred in the past so no hole seen.

Gemporia Craft has holes.

 

[Huevos]

It can't newly create them. But the refetch will not delete items - it (might) change events with the same eventID, but the bogus programme has now finished, so you will not get any update for it. Hence it remains in place.

As explained by LraiZer modified entries can't be removed because eventData type is only 8 bit so impossible for it to hold the identifier of OpenTV (0x4000).

https://www.world-of-satellite.com/...iX-e-g-5-4-008&p=511282&viewfull=1#post511282

 

[LraiZer]

Without the patch being applied first, the fix for updating old events was just removing more correct events that were overlapped by the bogus events, resulting in the event update fix making the first issue 10 times worse.

128 is an old removed free source type so i guess you could temp just change OPENTV=16384 to that as a bypass hack until update patch is fully tested and working?

 

[BrokenUnusableAccount]

Gordon, you're right the code makes no logical sense.

As I tried to explain in #103, #105 (note: in #105 I wrote 249 when I should have written 239)

They are kind of treating the field I've called startSecond as a signed time.
However it doesn't roll from meaning a +ve number to meaning a -ve number in the usual place (when the MSB flips to a 1).
Instead it seems it should be treated as a signed negative number if it's greater than or equal to 86400.

Then just as when you interpret a normal negative signed 17 bit number as unsigned the result is 0x20000 too high.

 

[LraiZer]

Just for the curious, here is a debug data dump log of the current bogus events being fixed.
Data cycle appears to currently be running at 139 seconds to populate the epg

 

[Huevos]

Without the patch being applied first, the fix for updating old events was just removing more correct events that were overlapped by the bogus events, resulting in the event update fix making the first issue 10 times worse.

128 is an old removed free source type so i guess you could temp just change OPENTV=16384 to that as a bypass hack until update patch is fully tested and working?

Could you make a patch for this please?

 

[bbbuk]

The over the air data received by an Enigma box over air must be correct or else the same problem would be seen on Freesat and Sky branded boxes. It must be some software running on the Enigma box that is wrongly interpreting the broadcast EPG data and creating values of startseconds that are not sensible.

Delete epg.dat and epg cache at this point and request a forced EPG data fetch and the EPG data for day B is now correct. The content of this forced EPG data must be identical to that of the forced EPG request issued before the epg.dat and epg cache deletion but then it didn’t result in the EPG data being corrected.

This is what I was trying to get at earlier by saying that I don't notice these issues with Sky UK when using CrossEPG to grab EPG data! Have previously seen when I switched from CrossEPG to EPGRefresh/OpenTV.

It's also looks like CrossEPG does some work-around maybe with epg cache as it creates a new instance of EPG cache.

Doesn't this, at least for Sky UK, potentially mean that broadcast data is correct and yes there could be multiple issues and EPG cache being other issue?

 

[BrokenUnusableAccount]

This is what I was trying to get at earlier by saying that I don't notice these issues with Sky UK when using CrossEPG to grab EPG data! Have previously seen when I switched from CrossEPG to EPGRefresh/OpenTV.

It's also looks like CrossEPG does some work-around maybe with epg cache as it creates a new instance of EPG cache.

Doesn't this, at least for Sky UK, potentially mean that broadcast data is correct and yes there could be multiple issues and EPG cache being other issue?

I did look quickly at the OpenTV part of CrossEPG and as far as I could see the relevant part of the code looked very like that in OpenViX's opentv.cpp.
No obvious sign of any means for CrossEPG to already be correct that I noticed.

 

[BrokenUnusableAccount]

As I tried to explain in #103, #105 (note: in #105 I wrote 249 when I should have written 239)

They are kind of treating the field I've called startSecond as a signed time.
However it doesn't roll from meaning a +ve number to meaning a -ve number in the usual place (when the MSB flips to a 1).
Instead it seems it should be treated as a signed negative number if it's greater than or equal to 86400.

Then just as when you interpret a normal negative signed 17 bit number as unsigned the result is 0x20000 too high.

Let me put that another way just in case some people still can't see it.

I think that startSecond (including the 0 LSB that's been added) is the bottom 17 bits from a larger (say 32 bit) signed integer that is only allowed to take on even values between -44672 and 86398.

 

[birdman]

They are kind of treating the field I've called startSecond as a signed time.

Which seems and odd thing to do.
Would you ever consider that I'm writing this at -11:00 on Monday?

It sounds more like and artefact of the way they process service suppliers data rather than being intentional. It's noticeable that it only seems to affect programmes which wrap between days at midnight. That's midnight UK, so for many Western European countries these do not span two days - they are completely within the second day.

But since the OpenTV spec isn't published we can only ever speculate, and just have work with whatever is sent out.

 

[BrokenUnusableAccount]

Which seems and odd thing to do.
Would you ever consider that I'm writing this at -11:00 on Monday?

Perhaps the day is the day the program finishes?
Perhaps the day part is not allowed to be before today?

It sounds more like an artefact of the way they process service suppliers data rather than being intentional. It's noticeable that it only seems to affect programmes which wrap between days at midnight. That's midnight UK, so for many Western European countries these do not span two days - they are completely within the second day.

But since the OpenTV spec isn't published we can only ever speculate, and just have work with whatever is sent out.

But presumably genuine Sky boxes know what to do with it.

 

[BrokenUnusableAccount]

While we have developer types in here can anyone tell me what's going on in opentv.cpp a few lines underneath where I was working?

		uint8_t *otvt = new uint8_t[titleLength];
		memcpy(otvt, buffer + 9, titleLength);
		crc32 = opentv_crc(otvt, titleLength);
		delete [] otvt;

WT* is it necessary to allocate memory to make a copy before working out the crc and then release it afterwards?
If you look at the definition of opentv_crc further up it can't modify it's input data.
It doesn't make any sense to me.

 

[LraiZer]

WT* is it necessary to allocate memory to make a copy before working out the crc and then release it afterwards?
If you look at the definition of opentv_crc further up it can't modify it's input data.
It doesn't make any sense to me.

Wasn't this also used and deleted somewhere instead in the early code creation during debug testing and then that code was removed and just a quick temporary delete added back here and then all this was forgot about?

Looks like you've got something else you'll have to fix :thumbsup:

 

[LraiZer]

Could you make a patch for this please?

[PATCH] New ENIGMA_EPG_V8 file structure

I think this patch should open up all the cache management code that was previously being blocked.
Will needs a lot of monitoring and testing over a period of time to see what it actually does or breaks!

 

[Huevos]

Just for the curious, here is a debug data dump log of the current bogus events being fixed.
Data cycle appears to currently be running at 139 seconds to populate the epg

If you inspect the stream can you see these erroneous values?