Mailing List Archive

mythfilldatabase: Ignoring unknown timestamp format: 20161231235960
Just a heads up since I'm getting a bunch of these from my daily
mythfilldatabase runs, looks like it is unhappy with the upcoming leap
second. Seems like it might also prevent it from inserting all the
other data too and not just the one bad program.

The message appears to come from
https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L170
so it seems to be a QDateTime issue.

http://doc.qt.io/qt-5/qdatetime.html says:

Note: QDateTime does not account for leap seconds.

I worked around it by inserting
| sed -e 's,"20161231235960 +0000","20161231235959 +0000",g'"
into my tv_grab wrapper script.

Ian.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On 12/16/2016 02:38 AM, Ian Campbell wrote:
> Just a heads up since I'm getting a bunch of these from my daily
> mythfilldatabase runs, looks like it is unhappy with the upcoming leap
> second. Seems like it might also prevent it from inserting all the
> other data too and not just the one bad program.
>
> The message appears to come from
> https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L170
> so it seems to be a QDateTime issue.
>
> http://doc.qt.io/qt-5/qdatetime.html says:
>
> Note: QDateTime does not account for leap seconds.
>
> I worked around it by inserting
> | sed -e 's,"20161231235960 +0000","20161231235959 +0000",g'"
> into my tv_grab wrapper script.

Um, you do realize that there's no leap second, right? Google says so.

https://cloudplatform.googleblog.com/2016/11/making-every-leap-second-count-with-our-new-public-NTP-servers.html

Time just flows 0.0014% slower in the period from 2pm on Dec 31 through
10am on Jan 1. I suppose this means that we'll all get to celebrate the
new years a little longer--though slower. Oh, and make sure you
compensate for this during the countdown to the ball drop or you'll be
embarrassed when you get to 0 before everyone else! Talk about an
awkward 0.00014-second pause...

Next thing you know someone is going to post something about some spoon,
as if that existed, too...

Mike
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On 16 December 2016 6:08:46 PM Ian Campbell <ijc@hellion.org.uk> wrote:

> Just a heads up since I'm getting a bunch of these from my daily
> mythfilldatabase runs, looks like it is unhappy with the upcoming leap
> second. Seems like it might also prevent it from inserting all the
> other data too and not just the one bad program.
>
> The message appears to come from
> https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L170
> so it seems to be a QDateTime issue.
>
> http://doc.qt.io/qt-5/qdatetime.html says:
>
> Note: QDateTime does not account for leap seconds.
>
> I worked around it by inserting
> | sed -e 's,"20161231235960 +0000","20161231235959 +0000",g'"
> into my tv_grab wrapper script.
>
> Ian.
>

Wouldn't the correct time be 20170101000000? I assume that's what the start
time of the subsequent program is assuming you are getting the incorrect
datetime value in endtime tags.



_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On Fri, 2016-12-16 at 12:35 +0000, Mark Perkins wrote:
> On 16 December 2016 6:08:46 PM Ian Campbell <ijc@hellion.org.uk>
> wrote:
>
> >
> > Just a heads up since I'm getting a bunch of these from my daily
> > mythfilldatabase runs, looks like it is unhappy with the upcoming
> > leap
> > second. Seems like it might also prevent it from inserting all the
> > other data too and not just the one bad program.
> >
> > The message appears to come from
> > https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfi
> > lldatabase/xmltvparser.cpp#L170
> > so it seems to be a QDateTime issue.
> >
> > http://doc.qt.io/qt-5/qdatetime.html says:
> >
> >     Note: QDateTime does not account for leap seconds.
> >
> > I worked around it by inserting
> >     | sed -e 's,"20161231235960 +0000","20161231235959 +0000",g'"
> > into my tv_grab wrapper script.
> >
> > Ian.
> >
>
> Wouldn't the correct time be 20170101000000? I assume that's what the start 
> time of the subsequent program is assuming you are getting the incorrect 
> datetime value in endtime tags.

I couldn't muster up the enthusiasm to care about 1s either way and
this required less brain power and characters changed.

Ian.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On 16/12/16 12:02, Michael T. Dean wrote:
> On 12/16/2016 02:38 AM, Ian Campbell wrote:
>> Just a heads up since I'm getting a bunch of these from my daily
>> mythfilldatabase runs, looks like it is unhappy with the upcoming leap
>> second. Seems like it might also prevent it from inserting all the
>> other data too and not just the one bad program.
>>
>> The message appears to come from
>> https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L170
>>
>> so it seems to be a QDateTime issue.
>>
>> http://doc.qt.io/qt-5/qdatetime.html says:
>>
>> Note: QDateTime does not account for leap seconds.
>>
>> I worked around it by inserting
>> | sed -e 's,"20161231235960 +0000","20161231235959 +0000",g'"
>> into my tv_grab wrapper script.
>
> Um, you do realize that there's no leap second, right? Google says so.
>
> https://cloudplatform.googleblog.com/2016/11/making-every-leap-second-count-with-our-new-public-NTP-servers.html
>
>
> Time just flows 0.0014% slower in the period from 2pm on Dec 31 through
> 10am on Jan 1. I suppose this means that we'll all get to celebrate the
> new years a little longer--though slower. Oh, and make sure you
> compensate for this during the countdown to the ball drop or you'll be
> embarrassed when you get to 0 before everyone else! Talk about an
> awkward 0.00014-second pause...
>

That's not correct, there is a leap second occuring on the 31st
December this year [1]. However google have their own way of dealing
with such things.

Rather than actually inserting the leap second like most ntp servers
do, google intentionally drift the time of their ntp servers, such that
once the leap second has occurred, the time is correct for the post
leap second event. This has the advantage of not needing to hand hold
systems that potentially miss handle the leap second.

That's what they are documenting in the blog

Regards
Stuart



[1] - https://datacenter.iers.org/eop/-/somos/5Rgv/latest/16


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On Fri, 2016-12-16 at 15:25 +0000, Stuart Auchterlonie wrote:
> On 16/12/16 12:02, Michael T. Dean wrote:
> >
> > On 12/16/2016 02:38 AM, Ian Campbell wrote:
> > >
> > > Just a heads up since I'm getting a bunch of these from my daily
> > > mythfilldatabase runs, looks like it is unhappy with the upcoming
> > > leap
> > > second. Seems like it might also prevent it from inserting all
> > > the
> > > other data too and not just the one bad program.
> > >
> > > The message appears to come from
> > > https://github.com/MythTV/mythtv/blob/master/mythtv/programs/myth
> > > filldatabase/xmltvparser.cpp#L170
> > >
> > > so it seems to be a QDateTime issue.
> > >
> > > http://doc.qt.io/qt-5/qdatetime.html says:
> > >
> > >      Note: QDateTime does not account for leap seconds.
> > >
> > > I worked around it by inserting
> > >      | sed -e 's,"20161231235960 +0000","20161231235959
> > > +0000",g'"
> > > into my tv_grab wrapper script.
> >
> > Um, you do realize that there's no leap second, right?  Google says
> > so.
> >
> > https://cloudplatform.googleblog.com/2016/11/making-every-leap-second-count-with-our-new-public-NTP-servers.html
> >
> >
> > Time just flows 0.0014% slower in the period from 2pm on Dec 31 through
> > 10am on Jan 1.  I suppose this means that we'll all get to celebrate the
> > new years a little longer--though slower.  Oh, and make sure you
> > compensate for this during the countdown to the ball drop or you'll be
> > embarrassed when you get to 0 before everyone else!  Talk about an
> > awkward 0.00014-second pause...
> >
>
> That's not correct,

I think he was kidding around...

Ian.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On 16 December 2016 11:06:32 PM Mark Perkins <perkins1724@hotmail.com> wrote:

> On 16 December 2016 6:08:46 PM Ian Campbell <ijc@hellion.org.uk> wrote:
>
>> Just a heads up since I'm getting a bunch of these from my daily
>> mythfilldatabase runs, looks like it is unhappy with the upcoming leap
>> second. Seems like it might also prevent it from inserting all the
>> other data too and not just the one bad program.
>>
>> The message appears to come from
>> https://github.com/MythTV/mythtv/blob/master/mythtv/programs/mythfilldatabase/xmltvparser.cpp#L170
>> so it seems to be a QDateTime issue.
>>
>> http://doc.qt.io/qt-5/qdatetime.html says:
>>
>> Note: QDateTime does not account for leap seconds.
>>
>> I worked around it by inserting
>> | sed -e 's,"20161231235960 +0000","20161231235959 +0000",g'"
>> into my tv_grab wrapper script.
>>
>> Ian.
>>
>
> Wouldn't the correct time be 20170101000000? I assume that's what the start
> time of the subsequent program is assuming you are getting the incorrect
> datetime value in endtime tags.
>
>
>
> _______________________________________________

Sure, but i am I intrigued that your listing provider (who was your
listing provider?) is ending (and maybe starting) programs at 1sec before
the hour. Do all your other programs end at xxxxxxxxxx5959 ie the ones not
around the end of the year? Or xxxxxxxxxx2959 if ending on the half hour?
Just curious.


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On Fri, 2016-12-16 at 20:29 +0000, Mark Perkins wrote:

> Sure,  but i am I intrigued that your listing provider (who was your 
> listing provider?)

It's schedules direct via tv_grab_sd_json. I've CC'd xmltv-devel, just
FtheirI I guess since as I said earlier I don't think 1s here or there
is worth worrying about (but if someone else wants to worry, fine!)

Context for xmltv-devel readers: This thread starts at http://lists.myt
htv.org/pipermail/mythtv-users/2016-December/390066.html and it appears
that something in the SD -> tv_grab_sd_json is mishandling the leap
second i.e. programs crossing midnight this year end 1s early

(independently mythfilldatabase does not accept the actual leap second
as an input, which is what I originally reported, but it seems mostly
moot since it looks like they shouldn't be present in the first place
for the given data).

> is ending (and maybe starting) programs at 1sec before 
> the hour. Do all your other programs end at xxxxxxxxxx5959 ie the ones not 
> around the end of the year? Or xxxxxxxxxx2959 if ending on the half hour? 
> Just curious.

Most programs span midnight, one of the "bad" ones from my unadjusted
data (so without the sed I mentioned earlier) was:
<programme start="20161231232500 +0000" stop="20161231235960 +0000" channel="24321">
<programme start="20170101000000 +0000" stop="20170101001500 +0000" channel="24321">

picking a random other channel at a random other time (not over midnight):
<programme start="20161223002000 +0000" stop="20161223005000 +0000" channel="25117">
<programme start="20161223005000 +0000" stop="20161223011500 +0000" channel="25117">

picking another random one with 5959 in it though:
<programme start="20161231170000 +0000" stop="20161231190000 +0000" channel="31786">
<programme start="20161231190000 +0000" stop="20170101005959 +0000" channel="31786">
<programme start="20170101010000 +0000" stop="20170101060000 +0000" channel="31786">
<programme start="20170101060000 +0000" stop="20170101080000 +0000" channel="31786">

and another:
<programme start="20161231210000 +0000" stop="20161231230000 +0000" channel="57747">
<programme start="20161231230000 +0000" stop="20170101005959 +0000" channel="57747">
<programme start="20170101010000 +0000" stop="20170101030000 +0000" channel="57747">
<programme start="20170101030000 +0000" stop="20170101060000 +0000" channel="57747">

WRT the half hour and other offsets they have a similar issue when
spanning the leap second e.g.:
<programme start="20161231210000 +0000" stop="20161231220000 +0000" channel="65161">
<programme start="20161231220000 +0000" stop="20170101002959 +0000" channel="65161">
<programme start="20170101003000 +0000" stop="20170101011500 +0000" channel="65161">
or
<programme start="20161231210000 +0000" stop="20161231233000 +0000" channel="21824">
<programme start="20161231233000 +0000" stop="20170101001459 +0000" channel="21824">
<programme start="20170105011500 +0000" stop="20170105030000 +0000" channel="21824">

So it looks like crossing the leap second has introduced a 1s quirk,
but it seems to resolve itself once programs are fully within 2017.

Ian.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
(resending after dns failure)

On Fri, Dec 16, 2016 at 3:25 PM, Stuart Auchterlonie
<stuarta@squashedfrog.net> wrote:
....
> That's not correct, there is a leap second occuring on the 31st
> December this year

And, technically, programs can end on the 60th
second of December 31st. That MythTV does
not handle a valid time (due to the QDateTime
implementation) should likely have a ticket created,
and marked as closed "wont fix" (or "upstream")
to document the issue for future generations so
that when it hits again (and it will always hit again,
perhaps in some other context) one can point
to the ticket. Since I believe only devs can open
tickets as closed, I will leave it one of the devs
interested in time.
MythTV could also decide to work to request
that their designated representatives to the next
WRC meeting (in 2023?) that will discuss the
leap second issue argues that the leap second
should be eliminated since it is so difficult to
properly deal with.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On Sat, Dec 17, 2016 at 3:12 PM, Gary Buhrmaster <gary.buhrmaster@gmail.com>
wrote:


> And, technically, programs can end on the 60th
> second of December 31st. That MythTV does
> not handle a valid time (due to the QDateTime
> implementation) should likely have a ticket created,
> and marked as closed "wont fix" (or "upstream")
> to document the issue for future generations so
> that when it hits again (and it will always hit again,
> perhaps in some other context) one can point
> to the ticket.


Hi Gary,

I personally have a differing opinion in this case.

It would appear that QT do not, and (likely without a lot of reworking)
will not soon attempt to support leap seconds as part of qdatetime.

Knowing that leap seconds have occurred since 1972, this is a particularly
bothersome limitation in a truly cross-platform toolkit.
However, I also consider that about 99% of the worlds population who uses
QT based apps will neither care about a 1 second discrepancy in some sort
of logfile, or use a QT app without any bearing on time whatsoever.

However, we have a nice little corner case. Not really even an edge case -
I fully submit that mythtv is over there in the dusty old corner of 'we
really want to work properly but can't due to circumstances out of our
control' . This doesn't sit well with me, a true believer in Postel's
Law: be conservative in what you do, be liberal in what you accept from
others.

If we know there is a problem affecting leap-seconds, that this problem
uniquely affects MythTV, and is propagated by very common scheduling
software, should MythTV not work themselves out of the corner and develop
the corner-fix for this particularly stupid (and pushed upstream to deaf
ears) bug?

MythTV should liberally attempt to catch a datetime error, validate if
'seconds=60' (starting at 0 thus index position 61) and then rework the
time to be 1 second different, potentially truncating the recording by one
paltry second.

If this bug still exists by the time the next leap-second is agreed upon by
the IERS, I will personally fault anyone who reasonably knows C++ and can
contribute to this project who hasn't attempted to patch this at the point
of currently-known pain: inside MythTV itself.

Mike
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On Sat, Dec 17, 2016 at 10:35 PM, Mike Hodson <mystica@gmail.com> wrote:

> I personally have a differing opinion in this case.

Getting time right is hard, and there are few
people willing to work through the entire process.
I look forward to your contribution to Qt to get
proper time support included. Given that for
UTC calculations this will *require* all apps to
update their (often embedded in the application)
time zone files to be updated every few months
(because whether a leap second will be added
is currently only known (around) six months in
advance, but as the earth continues to slow, it
may be added more often), and a 61 second
minute is only valid at those approved periods.
It should be an interesting exercise working
with the Qt community process (especially as
some of the commercial users of Qt want
a "ship and forget" approach). Perhaps the
answer is convert all time calculations to TAI.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On 18 December 2016 1:23:31 PM Gary Buhrmaster <gary.buhrmaster@gmail.com>
wrote:

> On Sat, Dec 17, 2016 at 10:35 PM, Mike Hodson <mystica@gmail.com> wrote:
>
>> I personally have a differing opinion in this case.
>
> Getting time right is hard, and there are few
> people willing to work through the entire process.

Without disagreeing with any of the prior posts, and whilst also agreeing
that the 61sec minute / leap second appears to be a valid time - I find it
hard to believe that Schedules Direct intended to have start / finish times
as per the examples posted (ie not rounded to the nearest minute). It seems
extraordinarily precise. Perhaps someone with a SD account can post a
message on the SD forums / raise a SD ticket for their input?

Is the end result that mythfilldatabase just fails to load the disagreeable
programs? Or does it fail to load all subsequent programs? Or does it fail
to load any programs? I can imagine that if the latter there may be many
more disgruntled posts to come in the next few days as people begin to
notice - and the quickest work around in the next 13 odd days may be a
tweak at the source with the long term actions to resolve to happen
'later'(tm).


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On Sun, 2016-12-18 at 04:30 +0000, Mark Perkins wrote:
>
> On 18 December 2016 1:23:31 PM Gary Buhrmaster <gary.buhrmaster@gmail.com> 
> wrote:
>
> > > On Sat, Dec 17, 2016 at 10:35 PM, Mike Hodson <mystica@gmail.com> wrote:
> >
> >> I personally have a differing opinion in this case.
> >
> > Getting time right is hard, and there are few
> > people willing to work through the entire process.
>
> Without disagreeing with any of the prior posts, and whilst also agreeing 
> that the 61sec minute / leap second appears to be a valid time - I find it 
> hard to believe that Schedules Direct intended to have start / finish times 
> as per the examples posted (ie not rounded to the nearest minute). It seems 
> extraordinarily precise. Perhaps someone with a SD account can post a 
> message on the SD forums / raise a SD ticket for their input?

I copied xmltv devel and one of the SD devs has responded already.

> Is the end result that mythfilldatabase just fails to load the disagreeable 
>
> programs? Or does it fail to load all subsequent programs? Or does it fail 
> to load any programs? I can imagine that if the latter there may be many 
> more disgruntled posts to come in the next few days as people begin to 
> notice - and the quickest work around in the next 13 odd days may be a 
> tweak at the source with the long term actions to resolve to happen 
> 'later'(tm).

I'm not 100% sure and I can't easily check since I applied the
workaround but it appeared to not add _any_ programs to that channel,
or maybe even at all.

I'll highlight this on the xmltv thread too.

Ian.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: mythfilldatabase: Ignoring unknown timestamp format: 20161231235960 [ In reply to ]
On Sun, 2016-12-18 at 08:46 +0000, Ian Campbell wrote:
> On Sun, 2016-12-18 at 04:30 +0000, Mark Perkins wrote:
> > 
> > On 18 December 2016 1:23:31 PM Gary Buhrmaster <gary.buhrmaster@gma
> il.com> 
> > wrote:
> > 
> > > > On Sat, Dec 17, 2016 at 10:35 PM, Mike Hodson <mystica@gmail.co
> m> wrote:
> > >
> > >> I personally have a differing opinion in this case.
> > >
> > > Getting time right is hard, and there are few
> > > people willing to work through the entire process.
> > 
> > Without disagreeing with any of the prior posts, and whilst also
> agreeing 
> > that the 61sec minute / leap second appears to be a valid time - I
> find it 
> > hard to believe that Schedules Direct intended to have start /
> finish times 
> > as per the examples posted (ie not rounded to the nearest minute).
> It seems 
> > extraordinarily precise. Perhaps someone with a SD account can post
> a 
> > message on the SD forums / raise a SD ticket for their input?
>
> I copied xmltv devel and one of the SD devs has responded already.

Actually, not an SD dev (I don't think) but the author of
tv_grab_sd_json.

This issue does not occur (due to a previous refactoring rather than
explicit fix) in the CVS version of tv_grab_sd_json.

Ian.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org