[Zgemma H7] Weird EPG errors. EPG Refresh. Recent versions of OpenViX e.g. 5.4.008.

[ccs]

If anyone is looking for actual programme schedule changes, New: Unforgotten has moved from 21:00 today to 21:00 tomorrow. (Already showing on freeview)

 

[BrokenUnusableAccount]

I'm really struggling to understand some of the stuff in opentv.cpp and epgcache.cpp.
For instance the end of line 160 in opentv.cpp.

startTimeBcd = (((startMjd - 40587) * 86400) + (UINT16(&buffer[2]) << 1));

What is function UINT16?
I've written C before but what does "&buffer[2]" do?
& is the "address off" function but the result is treated as if it was a number not an address (it's shifted left)!?

 

[Joe_90]

Can't tell you about the UINT stuff, but the calculation before the "+" seems to be the number of seconds in the integer days elapsed since 1/1/1970. (startTimeBcd is the startMjd (current day number since 17 November 1858) less the number of days to the unix epoch (1/1/1970) multiplied by the number of seconds in a day. Presumably the &buffer[2] contains a number of seconds to establish the start time in BCD?

It's odd that the time errors in the EPG times are offset 36 hours and 24 minutes ahead, i.e. 2184 minutes which is 0x888 or binary 100010001000. Seems like some error in data manipulation, maybe?

 

[adm]

I'm really struggling to understand some of the stuff in opentv.cpp and epgcache.cpp.
For instance the end of line 160 in opentv.cpp.

startTimeBcd = (((startMjd - 40587) * 86400) + (UINT16(&buffer[2]) << 1));

What is function UINT16?
I've written C before but what does "&buffer[2]" do?
& is the "address off" function but the result is treated as if it was a number not an address (it's shifted left)!?

I come from a hardware language background (VHDL) so I may be mistaken in my observation but ….

I would be more worried that a 16 bit signed 2s complement value (startMJd) multiplied by a 17 bit number (86400 or hexadecimal 1_5180) would result in a 33 bit number but startTimeBcd appears to be defined only as signed 2s complement 32 bit number. There doesn’t seem to be any sanity checking for under/over flows in the number ranges which may make a large positive number plus 1 = a negative number etc.
Also where have those arbitrary values of 40587 and 86400 come from?

However, this thread is possibly the wrong place to disscuss code in what may be an unrelated module to that causing the double entries

Edit

Tony has probably answered the 40587 and 86400 question.
Does the BCD and the end of the startTimeBcd indicate that the result needs to be in a binary coded decimal format?

 

[birdman]

What is function UINT16?

At a guess, a cast to an unsigned 16-bit integer.

I've written C before but what does "&buffer[2]" do?
& is the "address off" function but the result is treated as if it was a number not an address (it's shifted left)!?

&buffer[2] is the location of the 3rd element of buffer.
It's the same as buffer +2.
It's possible that UINT16 takes an address and uses it as a pointer to an unsigned 16-bit integer, whence shifting would be OK.

So the secret is to determine what UNT16 actually is.

 

[birdman]

Also where have those arbitrary values of 40587 and 86400 come from?

86400 isn't arbitrary - it's the number of seconds in a day.
The 40587 is probably something to do with a different base date. Such as something that started in ~1859, which rings a bell somewhere as something Microsoft/DEC used...
But using usefully-tagged parameters would help

EDIT: Tracked down the bell...

Modified Julian Date (MJD) A continuous measure in days since midnight at the start of 17 November 1858. Based on UTC.

And a bit more from:

 https://www.secret-bases.co.uk/wiki/Julian_day_number

The Modified Julian Date (MJD) was introduced by the Smithsonian Astrophysical Observatory in 1957 to record the orbit of Sputnik via an IBM 704 (36-bit machine) and using only 18 bits until August 7, 2576. MJD is the epoch of VAX/VMS and its successor OpenVMS, using 63-bit date/time, which allows times to be stored up to July 31, 31086, 02:48:05.47. The MJD has a starting point of midnight on November 17, 1858 and is computed by MJD = JD - 2400000.5

 

[bbbuk]

As this doesn't seem to happen with CrossEPG (that I have noticed), what is difference between using EPG sources from CrossEPG and OpenTV/EPGRefresh ?

I'm aware CrossEPG does something with cache unlike OpenTV/EPGRefresh but fundamentality is there any other difference?

Also, for comparison reasons are many of the FTA channels exactly same between FreeSat and Sky? ie is Channel 4 HD via FreeSat exactly same frequency as Sky UK?

 

[Joe_90]

The FTA channels are exactly the same between Freesat and Sky. There is no CH4 HD on Freesat's EPG, but it's FTA on Sky's EPG, so you have to pull the info from there.

 

[ccs]

Has anyone tried viewing the OpenWebIf version of the epg when the overlaps are showing via the gui?

I don't think it'll prove anything, but it might come up with a clue.

Also, does epg search find the phantom programmes?

 

[Joe_90]

Can't tell you about the UINT stuff, but the calculation before the "+" seems to be the number of seconds in the integer days elapsed since 1/1/1970. (startTimeBcd is the startMjd (current day number since 17 November 1858) less the number of days to the unix epoch (1/1/1970) multiplied by the number of seconds in a day. Presumably the &buffer[2] contains a number of seconds to establish the start time in BCD?

It's odd that the time errors in the EPG times are offset 36 hours and 24 minutes ahead, i.e. 2184 minutes which is 0x888 or binary 100010001000. Seems like some error in data manipulation, maybe?

Maybe one of the devs could look at the date manipulation in the OpenTVreader code which is triggered by OpenTVzapper? I looked through the EPG this morning and found examples of events from before midnight Sunday being transposed into the EPG before midday today. It happened on about 10 channels that I looked at quickly from BBC2 onwards. Not all channels affected but not limited to BBC/ITV/C4. The offset seems to be consistent at 2184 minutes which @something_fishy identified. 2184 decimal is 0x888. Looks like a bit mask error.

Here is some further information on the structure of the start_time field in the DVB EIT definition. I don't know if the proprietary data in the Sky EPG information conforms to this standard but I would guess that it would:

start_time: This 40-bit field contains the start time of the event in Universal Time, Co-ordinated (UTC) and Modified
Julian Date (MJD) (see annex C). This field is coded as 16 bits giving the 16 LSBs of MJD followed by 24 bits coded as
6 digits in 4-bit Binary Coded Decimal (BCD). If the start time is undefined (e.g. for an event in a NVOD reference
service) all bits of the field are set to "1".
EXAMPLE 1:
93/10/13 12:45:00 is coded as "0xC079124500".
duration: A 24-bit field containing the duration of the event in hours, minutes, seconds. format: 6 digits,
4-bit BCD = 24 bit.
EXAMPLE 2:
01:45:30 is coded as "0x014530".

 

[birdman]

Has anyone tried viewing the OpenWebIf version of the epg when the overlaps are showing via the gui?

It's possible to get a raw dump of the data using the OpenWebIF API.

https://dream.reichholf.net/e2web/

 

[something_fishy]

Has anyone tried viewing the OpenWebIf version of the epg when the overlaps are showing via the gui?

At least one does (I don't know about the others as the phantoms are now in the past in many cases)

Also, does epg search find the phantom programmes?

A search on the phantom finds other instances on other channels.

Bates Motel (BBC1HD @23:25-00:10) should have a phantom on Thursday ay 11:49. There is nothing in the guide yet, lets see if this changes tomorrow when the original has moved into the past.

 

[BrokenUnusableAccount]

I don't think I'm getting any "phantoms" if I run EPG Refresh daily at 08:37 instead of my usual 03:37.

But I will continue to experiment running it manually after midnight.

 

[BrokenUnusableAccount]

At a guess, a cast to an unsigned 16-bit integer.
&buffer[2] is the location of the 3rd element of buffer.
It's the same as buffer +2.
It's possible that UINT16 takes an address and uses it as a pointer to an unsigned 16-bit integer, whence shifting would be OK.

So the secret is to determine what UNT16 actually is.

Following through the chain of .h files including other .h files it seems UINT16 is as you say, taking an address and returning the 16 bit value pointed to by it.
But it swaps the byte order if necessary depending on the architecture of the targeted system.

 

[birdman]

But it swaps the byte order if necessary depending on the architecture of the targeted system.

A lot of network code does that (through macros).
Usually what comes down the wire is Big Endian (as it was probably developed on a Sun Microsystems box), but it has to be manipulated locally in native mode.
True of any binary transfer of data, really.

 

[BrokenUnusableAccount]

Do any legitimate EPG entries ever start at xx:x4 or xx:x9 ?
I don't think I've ever seen any.

 

[ccs]

Do any legitimate EPG entries ever start at xx:x4 or xx:x9 ?
I don't think I've ever seen any.

Sky ITV London Weather is in some schedules starting at 15:59 today - I haven't sky so don't see it on ViX, but there are others on freeview ...

 

[something_fishy]

Bates Motel (BBC1HD @23:25-00:10) should have a phantom on Thursday ay 11:49. There is nothing in the guide yet, lets see if this changes tomorrow when the original has moved into the past.

There is now a phantom entry for Bates Motel at 11:49-12:34 on Thursday on BBC1 (amongst others).

As far as I can see the phantoms are only created when the program crosses midnight and are in the past. Starts or ends on midnight don't trigger it (so news channels are unaffected) and it appears insensitive to start time:
CourtTV has a phantom "Death of George Floyd Murder Trial at 09:24-13:24 today (repeated from Monday night/Tuesday morning) and on Thursday (repeated from Tuesday night/Wednesday morning).

 

[adm]

I don't think I'm getting any "phantoms" if I run EPG Refresh daily at 08:37 instead of my usual 03:37.

But I will continue to experiment running it manually after midnight.

A one off test at a set time for a day or two isn't necessarily going to provide any meaningful results.

Without changing any settings, in the past 3 days I’ve only seen one occurrence of the problem but this doesn’t mean that tomorrow, or within a week, I will access the EPG and see duplicates on half a dozen channels.

 

[BrokenUnusableAccount]

A one off test at a set time for a day or two isn't necessarily going to provide any meaningful results.
Without changing any settings, in the past 3 days I’ve only seen one occurrence of the problem but this doesn’t mean that tomorrow, or within a week, I will access the EPG and see duplicates on half a dozen channels.

The dependence on time of day I scan is not a one off thing I've noticed. I mentioned it earlier in this thread.
In particular results just before and just after midnight are very different.