[Zgemma H7] Weird EPG errors. EPG Refresh. Recent versions of OpenViX e.g. 5.4.008.

[Huevos]

[PATCH] New ENIGMA_EPG_V8 file structure

I think this patch should open up all the cache management code that was previously being blocked.
Will needs a lot of monitoring and testing over a period of time to see what it actually does or breaks!

Thanks. That is now up on our repo for testing. :thumbsup:

 

[Joe_90]

I'm ready to test when software is available

 

[LraiZer]

If you inspect the stream can you see these erroneous values?

These erroneous values? e3 e0

They are in the stream buffer data that i output in the log file.

e3 << 9 = 116224
e0 << 1 = 448
116224 + 448 = 116672
1615766400 + 116672 = 1615883072
1615883072 - 131072 = 1615752000

BOGUS: Sewing Street / Yarn Lane REPEATS
 mjd=59288(bcd:1615766400) bcd=1615883072 bcdfix=1615752000 sec=116672(E3 E0) duration=43200
b5
22
e3
e0
54
60
6b

 

[Huevos]

Is it possible to get CRID from this stream? Simon is planning some code for programming series link.

 

[birdman]

There does seem to be some end-user documentation for an OpenTV API here:

https://opentv.nagra.com/sites/default/files/api/medialive_os_api/5.2.4/CCOM/index.html

Not sure how complete this is, but I can't find any reference to CRID data in it.

 

[Huevos]

There does seem to be some end-user documentation for an OpenTV API here:

https://opentv.nagra.com/sites/default/files/api/medialive_os_api/5.2.4/CCOM/index.html

Not sure how complete this is, but I can't find any reference to CRID data in it.

Definitely has it though because Sky boxes use it.

 

[birdman]

Definitely has it though because Sky boxes use it.

Good point...
So we'll need another hack to use it.

 

[ronand]

No spurious epg entries today on build 5.004.009.005 (as expected). Good work Brian on tracking down that gremlin.

 

[ccs]

No spurious epg entries today on build 5.004.009.005 (as expected). Good work Brian on tracking down that gremlin.

..... and @something_fishy who spotted the 36 hours 24 minutes link.

 

[BrokenUnusableAccount]

If anyone is able to check this change to OpenTV EPG doesn't cause problems with other services (other than Sky UK) that use OpenTV EPG, that would be good.

 

[LraiZer]

I only have 28.2E to test on. so cant test on others, but here is some code ideas for your pondering

Interpretation of the data sent as a negative number, not a positive:

		// HACK: startSecond is detected as being from the previous 
		// mjd date when the h:m:s is sent as greater than 1 day.
		// when this occurs, shifted two's complement has a negative 
		// sign bit of a signed integer, and is not a positive number.

		int32_t startSecond = UINT16(&buffer[2]) << 1;

		// if h:m:s is sent as greater than 1 day in seconds,
		// first bit is a negative sign bit without padding.

		if (startSecond >= 86400)
			startSecond = 0xFFFE0000 | (startSecond & 0x1FFFF);

		startTimeBcd = ((startMjd - 40587) * 86400) + startSecond;

 

[Huevos]

I only have 28.2E to test on. so cant test on others, but here is some code ideas for your pondering

Interpretation of the data sent as a negative number, not a positive:

		// HACK: startSecond is detected as being from the previous 
		// mjd date when the h:m:s is sent as greater than 1 day.
		// when this occurs, shifted two's complement has a negative 
		// sign bit of a signed integer, and is not a positive number.

		int32_t startSecond = UINT16(&buffer[2]) << 1;

		// if h:m:s is sent as greater than 1 day in seconds,
		// first bit is a negative sign bit without padding.

		if (startSecond >= 86400)
			startSecond = 0xFFFE0000 | (startSecond & 0x1FFFF);

		startTimeBcd = ((startMjd - 40587) * 86400) + startSecond;

Is this tested code on 28.2E?

On another note, do you know about CRID on this transport stream? It would be a nice addition now that Simon is adding code in eEPGCache to process it.
https://github.com/OpenViX/enigma2/...955f8d690ba180059716f8f64bd2366de9dcd0d44b0c0

 

[LraiZer]

Is this tested code on 28.2E?

Only tested by me so far based on how i now interpret the data can be a positive or a negative signed integer.
It correctly adjusted about 10 yesterday inc. all the regular occurring ones on our test channel Gempora Craft.

On another note, do you know about CRID on this transport stream? It would be a nice addition now that Simon is adding code in eEPGCache to process it.
https://github.com/OpenViX/enigma2/...955f8d690ba180059716f8f64bd2366de9dcd0d44b0c0

I dont see CRID, as far as i am aware it uses URI as defined:

DVB original_network_id = 1 (0x1)
DVB transport_stream_id = 10 (0xa)
DVB service_id = 100 (0x64)
DVB event_id = 1000 (0x3e8)
Unique eventId string = 0001000a006403e8

DVB original_network_id = 1 (0x1)
DVB transport_stream_id = 10 (0xa)
DVB service_id = 100 (0x64)
Unique serviceId string = "0001000a0064"

event.serviceId = “serviceId_XYZ”
event.uri = “tv://channel.serviceId_XYZ”
event.eventID = eventId_XYZ
event.seriesID = seriesLink (we can now proccess)
event.episodeID = ??? (cant see this in the data)

.sourceURL = event.uri

 

[bbbuk]

Looking at OpenTV.nagra.com website, it references "seriesId" and "episodeId" as @LraiZer mentions...

https://www.google.co.uk/search?q=site%3Ahttps%3A%2F%2Fopentv.nagra.com+seriesid&ei=G1VgYJ3tKouU8gLegLiABg&oq=site%3Ahttps%3A%2F%2Fopentv.nagra.com+seriesid&gs_lcp=Cgdnd3Mtd2l6EANQ1NYBWPDrAWD_7AFoAXAAeACAAewCiAGQEJIBBzQuMy40LjGYAQCgAQGqAQdnd3Mtd2l6wAEB&sclient=gws-wiz&ved=0ahUKEwjd5pfV3tLvAhULilwKHV4ADmAQ4dUDCA0&uact=5

https://docs.nagra.com/docs/os/524/series-linking-tutorial

 

[BrokenUnusableAccount]

I only have 28.2E to test on. so cant test on others, but here is some code ideas for your pondering

Interpretation of the data sent as a negative number, not a positive:

		// HACK: startSecond is detected as being from the previous 
		// mjd date when the h:m:s is sent as greater than 1 day.
		// when this occurs, shifted two's complement has a negative 
		// sign bit of a signed integer, and is not a positive number.

		int32_t startSecond = UINT16(&buffer[2]) << 1;

		// if h:m:s is sent as greater than 1 day in seconds,
		// first bit is a negative sign bit without padding.

		if (startSecond >= 86400)
			startSecond = 0xFFFE0000 | (startSecond & 0x1FFFF);

		startTimeBcd = ((startMjd - 40587) * 86400) + startSecond;

I'm not entirely sure it's any clearer now that we have a comment by the original code.
Though it does kind of process things in a more logical order.
You could write 0xFFFE0000 as (-0x20000) though that shouldn't be necessary since int32_t should be guaranteed to be 32 bits.

 

[BrokenUnusableAccount]

Looking at OpenTV.nagra.com website....

This must be documentation for some piece of software that takes in EPGs and things and produces the "secret" OpenTV data that Sky transmit.
I certainly can't find much information in there to help with interpreting the raw transmitted data (such as EPG).
Perhaps all the APIs and things described are how the TV companies transfer their EPG data, and so on, to Sky.
(but it does, as bbbuk pointed out, give you a good idea of what's in there somewhere)

 

[BrokenUnusableAccount]

Gemporia Craft has holes.

For a while after using the fix I think I wasn't seeing any holes in Gemporia Craft.

But now I am seeing holes, and also weird program ending times of the form xx:x7 just before the holes.

All other channels seem fine though.

I think the most liekly explanation is that Gemporia Craft are doing something weird.

 

[LraiZer]

Just thinking aloud

Gemporia Craft take the piss with 20 hour events = 72000 seconds duration. I guess you could start your debug by checking opentv code for uint16_t getDuration limits of 65535. Maybe there could also be an upper int18_t limit on a (startSecond+duration) that also triggers startSecond as a negative, even when startSecond is not greater than 1 day?

 

[LraiZer]

For a while after using the fix I think I wasn't seeing any holes in Gemporia Craft.

But now I am seeing holes, and also weird program ending times of the form xx:x7 just before the holes.

All other channels seem fine though.

I think the most liekly explanation is that Gemporia Craft are doing something weird.

I've fixed the issue on OpenPLi where xx.x7 was seen for 20 hours (72000 seconds) events.
This was just a wrongly defined upper uint16_t limitation which subtracting 65535 from 72000 giving a false duration of 1 hour 47 minutes.

The issue of the holes now again on Gemporia Craft is that most events they send are sent correctly as GMT +0, but the ones that appear to be causing the issue are the new 20 hour events that are sent with an 1 hour too late startTime. I can only assume this is related to BST +1 change last week that they have not done correctly and so may only be a temporary issue once they realize their mistake?