[ViX_Misc] Autotimers and Description Uniqueness

[spanner123]

This is not an indicator of repeats, plus...
If I do a search for "World War II: The Last Heroes" it comes up with 43 occurrences. Now consider that I've missed the first broadcast and now want to record the repeat - how do I stop it recording the many other repeats.

A note: this is one occurrence where the EPG data is different on Freeview and Satellite. On satellite the description for part 2 contains the word “theyre” and for terrestrial (Freeview ) the description contains the word “they’re”

In most cases the check for uniqueness works perfectly because each unique episode has a different episode title or description, and the repeat has the same title/description. The check sees that the description is the same for one broadcast occurrence and another broadcast occurrence (not necessarily the first broadcast and the first repeat) and it discards the second occurrence.
In this case ALL the 14(?) different episodes and 12 (?) repeats have exactly the SAME character description for the first 150 characters. The only difference is at the very end of the description the series/episode number is given as, say, S1, Ep3 and this is what identifies the different episodes and is the only indicator of it being a repeat showing.

I just set an autotimer for World War II and it found 5 events. Episodes 2 - 6 with no repeats!

 

[adm]

I just set an autotimer for World War II and it found 5 events. Episodes 2 - 6 with no repeats!

The example of World War II is a red herring in this context. In this case World War II does work for the autotimer unique filters becase each episode description is different. If I had applied a Autotimer filter to the EPGs I too would have got 5 timer events BUT 38 events rejected because they were repeats of the 5 timer events. If World War II only had generic descriptions then you would have seen the problem in that the autotimer would have either set a timer for one episode or all 43 episodes depending on the filter option.
The 43 occurences I quoted were just from the search of the EPG[1](terrestrail and satellite) just to demonstrate that a semi colon was not an indicator of repeats nor could it used alone to limit recordings to a 1 off repeat, say, if you had missed the original transmission.

[1] press yellow (EPG Search) when in the grid view EPG

 

[Willo3092]

I didn't mean that a colon indicated repeats, just that the recordings I seemed to have problems with have a colon in the title.
I just picked the World War II programme at random as an example.

One that I cannot for the life of me get to work without recording loads of duplicates is:

The Curious Life and Death Of: Lizzie Borden on Smithsonian

I have changed the match title to The Curious Life and Death Of and set to be unique in any service/recording and uniqueness in title and all descriptions.

It has picked up episode 2 which is The Curious Life and Death Of: A Rolling Stone but it is still recording duplicates for both episodes.

 

[adm]

I didn't mean that a colon indicated repeats, just that the recordings I seemed to have problems with have a colon in the title.
I just picked the World War II programme at random as an example.

One that I cannot for the life of me get to work without recording loads of duplicates is:

The Curious Life and Death Of: Lizzie Borden on Smithsonian

I have changed the match title to The Curious Life and Death Of and set to be unique in any service/recording and uniqueness in title and all descriptions.

It has picked up episode 2 which is The Curious Life and Death Of: A Rolling Stone but it is still recording duplicates for both episodes.

If you set the option to “title and short description” it should work because the descriptions for each episode are different.

I can confirm your results which I believe means “the uniqueness in title and all descriptions” option is not working at all - it does nothing. In this case all the repeated episodes have identical descriptions. The "uniqueness in title and all descriptions" is not detecting true uniqueness and as a result the autotimer is again setting timers for all occurrences of the same program.

The "uniqueness in title and all descriptions" cannot detect minor differences in descriptions – when all episodes have the same generic description but only differ by a minor variation by a unique series or episode number.

The "uniqueness in title and all descriptions" cannot detect identical descriptions – when each episode has a different description but where any repeated broadcast has the same description as the original.

The “title and short description” falls over when every episode has the same generic description that only varies with a series/episode indication at the end. The test appears not to be detecting different series/episode information.

The “title and all descriptions” falls over when every episode has the same generic description that only varies with a series/episode indication at the end AND when descriptions are completely identical (not unique). The test appears not to be detecting the same series/episode information nor the same description.

 

[birdman]

I have a very vague recollection that there used to be another option of checking a long description but this crashed the software as often there was no long description to check. Perhaps a fix for this bug has resulted in unexpected behaviour ????????

My recollection is that the concept of a short and a long description exists, but Freeview (UK) only ever uses one of them.

 

[birdman]

If I do a search for "World War II: The Last Heroes" it comes up with 43 occurrences. Now consider that I've missed the first broadcast and now want to record the repeat - how do I stop it recording the many other repeats.

I'd just set a manual timer for that one recording.

 

[adm]

My recollection is that the concept of a short and a long description exists, but Freeview (UK) only ever uses one of them.

The problem is on both Freeview and Freesat/Sky FTA (when the EPG data is collected over the air)

If there is only a short description to check (on either service) why do the two options for unique filtering both give different wrong results for certain programs?

Checking the short description doesn't appear take into account the series/episode data that is displayed in the description shown on screen. This is when the only difference in the description is the series/episode information.

Checking ALL decriptions surely must include the short description so why does it fail to identify descriptions that are the same? If there is only the short description that it can be checked in the "all" option why does it produce different results from the "short" filter option? Does this imply the filter is checking the short description plus random garbage if the "long" description is not populated???

 

[adm]

I'd just set a manual timer for that one recording.

I wouldn't need to as it would work with an autotimer checking for uniqueness in the short description. It wasn't the type of program with a generic description for all episodes and it wasn't program where a colon in the description signified different broadcast events. As I mentioned in post 62, this is not a program that shows the bug in the autotimer (short description) uniqueness testing. It should be exclude as a series of programs in the context of this thread.

 

[Willo3092]

These settings have given the best results so far and only records one duplicate of Rolling Stone and ignores Lizzie Borden because it's already recorded. This timer is unusual though, in that the title changes with different episodes.

 

[ccs]

My recollection is that the concept of a short and a long description exists, but Freeview (UK) only ever uses one of them.

My recollection when this was debated a long while ago is that long descriptions simply do not work so shouldn't be used.

That may not be quite how it works now.

 

[Old Codger]

Thanks for all the replies, they're coming in almost too fast for me to keep up with

In most cases the check for uniqueness works perfectly because each unique episode has a different episode title or description, and the repeat has the same title/description.

I'm not sure that the episode title works any better than the series/episode numbers if the rest of the description is generic - the main problem (I reckon) is that the description needs a certain percentage of 'uniqueness' to be treated as such. Can't be sure now, but I think that I tested a programme with only the episode title different for each episode, and that didn't work either.

Sorry i forgot to add - after selecting the various posts that you wish to quote then click "reply with quote" and you will see them all in your message and you can edit them as required.

Many thanks, that works - just too obvious for me to spot However, it only seems to work for posts on the same page, if you go on to a 2nd page, the previous ones are lost?

Did you have to change your password recently? If so log out and then log back in but make sure the little tick box close to the usename paswod box is ticked. By the Way your new password can be the same as the old password when you get the 30 day forum request to change it.

No, haven't changed my password recently. I didn't realise that you were required to change it every 30 days, and haven't (to my knowledge) received a specific request to change it, even though I registered back in mid-July. Done so now, though.
I've never used the tick box you refer to, on the basis that I don't know what it actually does.

 

[Old Codger]

If you set the option to “title and short description” it should work because the descriptions for each episode are different.

I can confirm your results which I believe means “the uniqueness in title and all descriptions” option is not working at all - it does nothing. In this case all the repeated episodes have identical descriptions. The "uniqueness in title and all descriptions" is not detecting true uniqueness and as a result the autotimer is again setting timers for all occurrences of the same program.

The "uniqueness in title and all descriptions" cannot detect minor differences in descriptions – when all episodes have the same generic description but only differ by a minor variation by a unique series or episode number.

The "uniqueness in title and all descriptions" cannot detect identical descriptions – when each episode has a different description but where any repeated broadcast has the same description as the original.

The “title and short description” falls over when every episode has the same generic description that only varies with a series/episode indication at the end. The test appears not to be detecting different series/episode information.

The “title and all descriptions” falls over when every episode has the same generic description that only varies with a series/episode indication at the end AND when descriptions are completely identical (not unique). The test appears not to be detecting the same series/episode information nor the same description.

Yes, exactly. You've described much better than I ever could, thanks for that.

The problem is on both Freeview and Freesat/Sky FTA (when the EPG data is collected over the air)

Checking the short description doesn't appear take into account the series/episode data that is displayed in the description shown on screen. This is when the only difference in the description is the series/episode information.

And possibly also when the only difference in the description is the episode title (see my previous post).

 

[Joe_90]

...
I've never used the tick box you refer to, on the basis that I don't know what it actually does.

The tick box keeps you logged in. If you don't tick it the forum logs you out after a time.

 

[Old Codger]

The tick box keeps you logged in. If you don't tick it the forum logs you out after a time.

Explains why I keep being logged out! Many thanks.

 

[BrianTheTechieSnail]

I've been looking at the source code of the AutoTimer plug-in.
For those set up to build their own images it's in /build-enviroment/builds/openvix/<BUILDTYPE>/<RECEIVER>/tmp/sysroots-components/<RECEIVER>/enigma2-plugins/usr/lib/enigma2/python/Plugins/Extensions/AutoTimer
where <BUILDTYPE> is release or developer.
and <RECEIVER> is your type of receiver, eg h7 for Zgemma H7.

I'm afraid I haven't found where it is on the internet (probably github or similar).

The comparisons of the titles and descriptions seem to be done in function checkSimilarity which starts on line 838 of the file AutoTimer.py.
It uses a function SequenceMatcher from difflib, which you can find descriptions of on the web such as https://towardsdatascience.com/sequencematcher-in-python-6b1e6f3915fc

I don't think it's really the right function in this application because, for instance, a single character difference in the middle of a description counts as a huge difference while a single character difference near the beginning or end counts only as a small difference. Thus the change from (S01:E04) to (S01:E05) right at the end of a description is seen as something to ignore.

I think we can come up with something better. I'm not sure exactly what though.

 

[adm]

The comparisons of the titles and descriptions seem to be done in function checkSimilarity which starts on line 838 of the file AutoTimer.py.
It uses a function SequenceMatcher from difflib, of which can find descriptions on the web such as https://towardsdatascience.com/sequencematcher-in-python-6b1e6f3915fc

I don't think it's really the right function in this application because, for instance, a single character difference in the middle of a description counts as a huge difference while a single character difference near the beginning or end counts only as a small difference. Thus the change from (S01:E04) to (S01:E05) right at the end of a description is seen as something to ignore.

That may explain why the "uniqueness in short description" sees everything as being identical when there is a generic description for all episodes with only the series/episode information at the end changing changing.

It doesn't explain why "uniqueness in all descriptions" sees everything as being different (including all repeats with identical information). If there is no other descriptions to add to the short description then the results of the tests should be identical unless, as I speculated earlier, garbage is being added to the search string.

Perhaps the software adding a test string of identical characters to the end of a short descriptions before the test is made would bring the (S01:E04) etc. towards the middle of the description and any such changes would have more weighting ?????

Edit: a quick read of the description -- is it missing series/episode info because they are enclosed in brackets?

 

[ccs]

These are recent commits to autotimer, with clues where it resides....

https://github.com/oe-alliance/enigma2-plugins/commits/master/autotimer/src/AutoTimer.py

You don't need to build an image to try things out, just modify AutoTimer.py, download it to the correct location and reload.

 

[BrianTheTechieSnail]

That may explain why the "uniqueness in short description" sees everything as being identical when there is a generic description for all episodes with only the series/episode information at the end changing changing.

It doesn't explain why "uniqueness in all descriptions" sees everything as being different (including all repeats with identical information). If there is no other descriptions to add to the short description then the results of the tests should be identical unless, as I speculated earlier, garbage is being added to the search string.

Yes, there is something more than I described going on. Most likely due to bugs/errors elsewhere I'm thinking?

Perhaps the software adding a test string of identical characters to the end of a short descriptions before the test is made would bring the (S01:E04) etc. towards the middle of the description and any such changes would have more weighting ?????

Edit: a quick read of the description -- is it missing series/episode info because they are enclosed in brackets?

I don't think maximum matching sequence length ratio is right for this at all, even if you somehow manage to shuffle things around first
I just haven't got my head around exactly what would be better yet.

(Oh by the way it is configured to ignore spaces, I think)

 

[BrianTheTechieSnail]

These are recent commits to autotimer, with clues where it resides....

https://github.com/oe-alliance/enigma2-plugins/commits/master/autotimer/src/AutoTimer.py

You don't need to build an image to try things out, just modify AutoTimer.py, download it to the correct location and reload.

Ah ha!
Here we are: https://github.com/oe-alliance/enigma2-plugins/blob/master/autotimer/src/AutoTimer.py

Yes AFAICS in a running image we have /usr/lib/enigma2/python/Plugins/Extensions/AutoTimer/AutoTimer.pyo which we can rename and replace with a modified AutoTimer.py, new new AutoTimer.pyo will be created the first time it's used and then you could delete the AutoTimer.py if you wished.

 

[Old Codger]

Ah ha!
Here we are: https://github.com/oe-alliance/enigma2-plugins/blob/master/autotimer/src/AutoTimer.py

There are at least 2 more versions on Github -
https://github.com/OpenPLi/enigma2-plugins/tree/master/autotimer/src/AutoTimer.py (bang up-to-date)
https://github.com/opendreambox/enigma2-plugins/tree/master/autotimer/src/AutoTimer.py (well out-of-date)