Mailing List Archive

On 09/04/2023 09:50, Ross Boylan wrote:
> A few days ago my system locked up with video problems; I was able to
> ssh in and execute shutdown. But since then I see an error like this
> in the logs, I think for every show I record:
> Apr 8 22:06:30 barley mythbackend[2592]: 2023-04-08 22:06:30.919562 E
> DB Error (Duration insert):
> Apr 8 22:06:30 barley mythbackend[2592]: Query was:
> Apr 8 22:06:30 barley mythbackend[2592]: INSERT INTO recordedmarkup
> (chanid, starttime, mark, type, data) VALUES ( 10203,
> '2023-04-09T02:54:00Z', 0, 33, 7950000);
> Apr 8 22:06:30 barley mythbackend[2592]: Bindings were:
> Apr 8 22:06:30 barley mythbackend[2592]: :CHANID=10203,
> :DATA=7950000, :STARTTIME=2023-04-09T02:54:00.000Z, :TYPE=33
> Apr 8 22:06:30 barley mythbackend[2592]: Driver error was [2/1062]:
> Apr 8 22:06:30 barley mythbackend[2592]: QMYSQL: Unable to execute query
> Apr 8 22:06:30 barley mythbackend[2592]: Database error was:
> Apr 8 22:06:30 barley mythbackend[2592]: Duplicate entry
> '10203-2023-04-09 02:54:00-33-0' for key 'PRIMARY'

In order to exclude the possibility of database corruption, could you
check/repair the database?

https://www.mythtv.org/wiki/Database_errors

> Myth 1:31.0+fixes20220227.git7e4ce1ba98-dmo0+deb11u1 from Marillat's
> repository on Debian 11/bullseye
> mariadb 1:10.5.19-0+deb11u1
> nvidia proprietary driver 470.161.03-1
> amd64 architecture

People here may not be familiar with that particular repository (I
certainly am not, but I'm also not a MythTV dev).

Please always include the output of mythbackend --version in issue reports.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sun, 9 Apr 2023 10:32:25 +0200, you wrote:

>In order to exclude the possibility of database corruption, could you
>check/repair the database?
>
>https://www.mythtv.org/wiki/Database_errors

Yes, it is likely a corrupt recordedseek table. That is the table
that is almost certain to be damaged if there is a crash or lockup or
reboot done while a program is recording. Whenever your system is not
shut down cleanly, you must always check all the database tables
before starting any new recordings. The easiest way to do that is to
just manually run the automated check and repair that should be being
done automatically by anacron every day:

/etc/cron.daily/optimize_mythdb

Use sudo to run that file.

If you do not have that file installed, you should fix that by copying
the original version, which can be found here in the Ubuntu packages:

/usr/share/doc/mythtv-backend/contrib/maintenance/optimize_mythdb.pl

The optimize_mythdb script should be run when no recordings are in
progress, so if you run it manually, you should probably shut down
mythbackend first. Once it completes and says it has fixed the
recordedseek table (and any other damaged tables), you can restart
mythbackend, and then you will need to run mythcommflag --rebuild on
any recordings made since the recordedseek table was damaged. If you
do commercial skip processing, you will also need to run that again
for those recordings, after running --rebuild. Find the mythcommflag
settings you are using in:

mythtv-setup > 1. General > Job Queue (Global) > Advert-detection
command

and run that command against each recording file. To specify one
recording file to mythcommflag, use its "--chanid" and "--starttime"
options.

Note that successfully repairing the database tables requires that you
have sufficient free space on the partition where the mythconverg
database is stored to make copies of the largest table's files. The
largest table is almost always recordedseek. Use this command to see
what the largest tables are:

sudo ls -alS /var/lib/mysql/mythconverg/ | head -n 20

and this to see how much space the recordedseek tables use:

sudo du -hc /var/lib/mysql/mythconverg/recordedseek*

Debian may put the files in different places from Ubuntu, so check the
locations.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Jan and Stephen, thank you for your advice. I ran mysqlcheck and then
(redundantly?) optimize_mythdb. The former reported OK for every
table; I'm not sure if that means it was OK before or after the run.

I restarted the backend (--version info below) and, at least for now,
all recordings show as having positive size. Which is odd, since when
I query the database directly there are just as many with
recorded.filesize=0 as before.

As root I ran mythcommflag --rebuild on one of the files. It produced
a bunch of errors like
2023-04-09 14:37:01.979810 E DB Error (delta position map insert):
Query was:
INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`)
VALUES (10501,'2023-04-08T04:59:00Z',9,0,376),(10501,'2023-04-08T04:59:00Z',9,87,5975016),(10501,'2023-04-08T04:59:00Z',9,96,6595040),(10501,'2023-04-08T04:59:00Z',9,177,12453872),(10501,'2023-04-08T04:59:00Z',9,242,17192224),(10501,'2023-04-08T04:59:00Z',9,251,17963400),(10501,'2023-04-08T04:59:00Z',9,259,18314020),(10501,'2023-04-08T04:59:00Z',9,294,20370176),(10501,'2023-04-08T04:59:00Z',9,366,24934628),(10501,'2023-04-08T04:59:00Z',9,438,29536492),(10501,'2023-04-08T04:59:00Z',9,510,34433328),(10501,'2023-04-08T04:59:00Z',9,560,37335296),(10501,'2023-04-08T04:59:00Z',9,611,40210944),(10501,'2023-04-08T04:59:00Z',9,647,42266536),(10501,'2023-04-08T04:59:00Z',9,648,42378208),(10501,'2023-04-08T04:59:00Z',9,651,42579368),(10501,'2023-04-08T04:59:00Z',9,685,44738172),(10501,'2023-04-08T04:59:00Z',9,711,46408176),(10501,'2023-04-08T04:59:00Z',9,765,49665276),(10501,'2023-04-08T04:59:00Z',9,792,51293356),(10501,'2023-04-08T04:59:00Z',9,812,52881392),(10501,'2023-04-08T04:59:00Z',9,833,53866700),(10501,'2023-04-08T04:59:00Z',9,858,55807236),(10501,'2023-04-08T04:59:00Z',9,874,56442676),(10501,'2023-04-08T04:59:00Z',9,899,57998000),(10501,'2023-04-08T04:59:00Z',9,924,59380928),(10501,'2023-04-08T04:59:00Z',9,946,60735656),(10501,'2023-04-08T04:59:00Z',9,970,62145844),(10501,'2023-04-08T04:59:00Z',9,990,63564492),(10501,'2023-04-08T04:59:00Z',9,1062,67640144),(10501,'2023-04-08T04:59:00Z',9,1093,69616212),(10501,'2023-04-08T04:59:00Z',9,1102,70251652),(10501,'2023-04-08T04:59:00Z',9,1111,70428936),(10501,'2023-04-08T04:59:00Z',9,1144,72631732),(10501,'2023-04-08T04:59:00Z',9,1151,72733440),(10501,'2023-04-08T04:59:00Z',9,1172,73935512),(10501,'2023-04-08T04:59:00Z',9,1235,77416708),(10501,'2023-04-08T04:59:00Z',9,1267,79437144),(10501,'2023-04-08T04:59:00Z',9,1288,80898092),(10501,'202

Interspersed, slightly less frequent, were
Driver error was [2/1062]:
QMYSQL: Unable to execute query
Database error wa
2023-04-09 14:37:03.987264 E DB Error (delta position map insert):

after many more like the above:

Rebuild completed at Sun Apr 9 14:38:16 2023
2023-04-09 14:38:16.318015 E decoding error End of file (-541478725)
2023-04-09 14:38:16.319383 E DB Error (Duration insert):
Query was:
INSERT INTO recordedmarkup (chanid, starttime, mark, type, data)
VALUES ( 10501, '2023-04-08T04:59:00Z', 0, 33, 3772191);
Bindings were:
:CHANID=10501, :DATA=3772191, :STARTTIME=2023-04-08T04:59:00.000Z, :TYPE=33
Driver error was [2/1062]:
QMYSQL: Unable to execute query
Database error was:
Duplicate entry '10501-2023-04-08 04:59:00-33-0' for key 'PRIMARY'

2023-04-09 14:38:16.320471 E DB Error (Total Frames insert):

There are a lot of things that seem odd:
1. I thought the database did a check on restart--that seems to be in my logs.
2. Databases are supposed to be corruption resistant, though
mysql/maria may be less so (I know that used to be true).
2b. And I didn't even have an uncontrolled shutdown.
3. Why are the problems affecting brand new recordings?

recordedseek.MYD is 724Mb on disk; the .MYI file is 651Mb. The
partition has plenty of free space and is ext4 formatted.

Obviously, when I figure out how to recover I hope to do it just for
recordings affected by the problem.

Ross

Version info:
MythTV Version : v31.0
MythTV Branch :
Network Protocol : 91
Library API : 31.20200101-1
QT Version : 5.15.2
Options compiled in:
linux release use_hidesyms using_alsa using_jack using_oss
using_pulse using_pulseoutput using_backend using_bindings_perl
using_bindings_python using_bindings_php using_dvb using_firewire
using_frontend using_hdhomerun using_vbox using_ceton using_hdpvr
using_ivtv using_joystick_menu using_libcec using_libcrypto
using_gnutls using_libdns_sd using_libfftw3 using_libxml2 using_lirc
using_mheg using_opengl using_egl using_qtwebkit using_qtscript
using_qtdbus using_taglib using_v4l2 using_v4l2prime using_x11
using_libbluray_external using_xrandr using_systemd_notify
using_systemd_journal using_drm using_bindings_perl
using_bindings_python using_bindings_php using_freetype2
using_mythtranscode using_opengl using_egl using_drm using_vaapi
using_nvdec using_vdpau using_ffmpeg_threads using_mheg using_libass
using_libxml2 using_libmp3lame

On Sun, Apr 9, 2023 at 4:08?AM Stephen Worthington
<stephen_agent@jsw.gen.nz> wrote:
>
> On Sun, 9 Apr 2023 10:32:25 +0200, you wrote:
>
> >In order to exclude the possibility of database corruption, could you
> >check/repair the database?
> >
> >https://www.mythtv.org/wiki/Database_errors
>
> Yes, it is likely a corrupt recordedseek table. That is the table
> that is almost certain to be damaged if there is a crash or lockup or
> reboot done while a program is recording. Whenever your system is not
> shut down cleanly, you must always check all the database tables
> before starting any new recordings. The easiest way to do that is to
> just manually run the automated check and repair that should be being
> done automatically by anacron every day:
>
> /etc/cron.daily/optimize_mythdb
>
> Use sudo to run that file.
>
> If you do not have that file installed, you should fix that by copying
> the original version, which can be found here in the Ubuntu packages:
>
> /usr/share/doc/mythtv-backend/contrib/maintenance/optimize_mythdb.pl
>
> The optimize_mythdb script should be run when no recordings are in
> progress, so if you run it manually, you should probably shut down
> mythbackend first. Once it completes and says it has fixed the
> recordedseek table (and any other damaged tables), you can restart
> mythbackend, and then you will need to run mythcommflag --rebuild on
> any recordings made since the recordedseek table was damaged. If you
> do commercial skip processing, you will also need to run that again
> for those recordings, after running --rebuild. Find the mythcommflag
> settings you are using in:
>
> mythtv-setup > 1. General > Job Queue (Global) > Advert-detection
> command
>
> and run that command against each recording file. To specify one
> recording file to mythcommflag, use its "--chanid" and "--starttime"
> options.
>
> Note that successfully repairing the database tables requires that you
> have sufficient free space on the partition where the mythconverg
> database is stored to make copies of the largest table's files. The
> largest table is almost always recordedseek. Use this command to see
> what the largest tables are:
>
> sudo ls -alS /var/lib/mysql/mythconverg/ | head -n 20
>
> and this to see how much space the recordedseek tables use:
>
> sudo du -hc /var/lib/mysql/mythconverg/recordedseek*
>
> Debian may put the files in different places from Ubuntu, so check the
> locations.
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sun, 9 Apr 2023 15:14:40 -0700, you wrote:

>Jan and Stephen, thank you for your advice. I ran mysqlcheck and then
>(redundantly?) optimize_mythdb. The former reported OK for every
>table; I'm not sure if that means it was OK before or after the run.
>
>I restarted the backend (--version info below) and, at least for now,
>all recordings show as having positive size. Which is odd, since when
>I query the database directly there are just as many with
>recorded.filesize=0 as before.
>
>As root I ran mythcommflag --rebuild on one of the files. It produced
>a bunch of errors like
>2023-04-09 14:37:01.979810 E DB Error (delta position map insert):
>Query was:
>INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`)
>VALUES
>(10501,'2023-04-08T04:59:00Z',9,0,376),(10501,'2023-04-08T04:59:00Z',9,87,5975016),(10501,'2023-04-08T04:59:00Z',9,96,6595040),(10501,'2023-04-08T04:59:00Z',9,177,12453872),(10501,'2023-04-08T04:59:00Z',9,242,17192224),(10501,'2023-04-08T04:59:00Z',9,251,17963400),(10501,'2023-04-08T04:59:00Z',9,259,18314020),(10501,'2023-04-08T04:59:00Z',9,294,20370176),(10501,'2023-04-08T04:59:00Z',9,366,24934628),(10501,'2023-04-08T04:59:00Z',9,438,29536492),(10501,'2023-04-08T04:59:00Z',9,510,34433328),(10501,'2023-04-08T04:59:00Z',9,560,37335296),(10501,'2023-04-08T04:59:00Z',9,611,40210944),(10501,'2023-04-08T04:59:00Z',9,647,42266536),(10501,'2023-04-08T04:59:00Z',9,648,42378208),(10501,'2023-04-08T04:59:00Z',9,651,42579368),(10501,'2023-04-08T04:59:00Z',9,685,44738172),(10501,'2023-04-08T04:59:00Z',9,711,46408176),(10501,'2023-04-08T04:59:00Z',9,765,49665276),(10501,'2023-04-08T04:59:00Z',9,792,51293356),(10501,'2023-04-08T04:59:00Z',9,812,52881392),(10501,'2023-04-08T04:59:00Z',9,833,538667
0
>0),(10501,'2023-04-08T04:59:00Z',9,858,55807236),(10501,'2023-04-08T04:59:00Z',9,874,56442676),(10501,'2023-04-08T04:59:00Z',9,899,57998000),(10501,'2023-04-08T04:59:00Z',9,924,59380928),(10501,'2023-04-08T04:59:00Z',9,946,60735656),(10501,'2023-04-08T04:59:00Z',9,970,62145844),(10501,'2023-04-08T04:59:00Z',9,990,63564492),(10501,'2023-04-08T04:59:00Z',9,1062,67640144),(10501,'2023-04-08T04:59:00Z',9,1093,69616212),(10501,'2023-04-08T04:59:00Z',9,1102,70251652),(10501,'2023-04-08T04:59:00Z',9,1111,70428936),(10501,'2023-04-08T04:59:00Z',9,1144,72631732),(10501,'2023-04-08T04:59:00Z',9,1151,72733440),(10501,'2023-04-08T04:59:00Z',9,1172,73935512),(10501,'2023-04-08T04:59:00Z',9,1235,77416708),(10501,'2023-04-08T04:59:00Z',9,1267,79437144),(10501,'2023-04-08T04:59:00Z',9,1288,80898092),(10501,'202
>
>Interspersed, slightly less frequent, were
>Driver error was [2/1062]:
>QMYSQL: Unable to execute query
>Database error wa
>2023-04-09 14:37:03.987264 E DB Error (delta position map insert):
>
>after many more like the above:
>
>Rebuild completed at Sun Apr 9 14:38:16 2023
>2023-04-09 14:38:16.318015 E decoding error End of file (-541478725)
>2023-04-09 14:38:16.319383 E DB Error (Duration insert):
>Query was:
>INSERT INTO recordedmarkup (chanid, starttime, mark, type, data)
>VALUES ( 10501, '2023-04-08T04:59:00Z', 0, 33, 3772191);
>Bindings were:
>:CHANID=10501, :DATA=3772191, :STARTTIME=2023-04-08T04:59:00.000Z, :TYPE=33
>Driver error was [2/1062]:
>QMYSQL: Unable to execute query
>Database error was:
>Duplicate entry '10501-2023-04-08 04:59:00-33-0' for key 'PRIMARY'
>
>2023-04-09 14:38:16.320471 E DB Error (Total Frames insert):
>
>There are a lot of things that seem odd:
>1. I thought the database did a check on restart--that seems to be in my logs.

No, a full database check takes too long to be done when MySQL/MariaDB
starts. I believe that there is a very basic check done that the
database files are available, but nothing beyond that. The
/etc/cron.daily/optimize_mythdb script is run daily by anacron (if
present), and that normally keeps the database clean. Anacron will
also run it shortly after boot time if the PC is rebooted and it has
not been run in the last day. If you do not have daily runs of
optimize_mythdb happening, then any database errors will just
accumulate. These are errors that mysqlcheck from optimize_mythdb
will be able to fix, but without daily fixing, but eventually one
error lands on top of another and that then causes the problem to
become unfixable.

>2. Databases are supposed to be corruption resistant, though
>mysql/maria may be less so (I know that used to be true).

They are somewhat corruption resistant, but it varies. No database I
know of is resistant to what happens when it is not shut down and is
killed in the middle of doing something. Hence the need to run a full
check after each such event, to prevent the accumulation problem.

>2b. And I didn't even have an uncontrolled shutdown.

Your report says you probably did. Even though you were able to ssh
in and do a shutdown command, my guess is that the shutdown will have
taken a long time as something was locked up and failed to shut down.
That something may have had the database locked, so when systemd's
shutdown routines timed out that process and forcibly killed it, the
database was left corrupted.

>3. Why are the problems affecting brand new recordings?

Clearly the database is not working - the recordedseek table is still
corrupt.

>recordedseek.MYD is 724Mb on disk; the .MYI file is 651Mb. The
>partition has plenty of free space and is ext4 formatted.
>
>Obviously, when I figure out how to recover I hope to do it just for
>recordings affected by the problem.

At this point, I would advise finding your backups of your database
and copying all the files to somewhere else so they do not get
overwritten by new backups of the corrupted database. Backups get
rotated as each new one is done, so only 5 copies are kept. If your
database corruption has been there for a long time, potentially all of
the backups may be corrupt also. If the original problem is more
recent, the older backups should still be fine, but you do not want to
find out that only the oldest one that got deleted yesterday was the
last OK one.

The next thing to try is to manually do a full check of the
recordedseek table. Shut down mythbackend and do these commands:

sudo mysqlcheck --repair mythconverg recordedseek
sudo mysqlcheck --optimize mythconverg recordedseek
sudo mysqlcheck --analyze mythconverg recordedseek

If there are any errors at all, keep repeating those commands until
there are no more errors, or it does not fix anything. In the latter
case, the table is unfixable and will need to be deleted and
re-created, which it is possible to do, but takes a lot of work. Or
you can get back an older, possibly non-corrupt version from you
backups, which again is possible but is a bit difficult - you need a
good editor that can edit huge .sql text files in order to just edit
out the recordedseek table in the backup file. Most editors can not
handle files of that size.

If manually checking recordedseek does not show any errors, then you
need to try manually inserting data, like mythbackend is trying to do.
Copy one of the commands it is logging as failing and try running it
manually. Do only a single insert, not a multiple one as is in the
log messages. For example, from your post, you could try:

INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`)
VALUES (10501,'2023-04-08T04:59:00Z',9,0,376);

Keep track of any INSERT commands you do so you can do a corresponding
DELETE FROM command if it succeeds. So for the above, if it works,
you would want to do:

DELETE FROM recordedseek WHERE chanid=10501 and
starttime='2023-04-08T04:59:00Z' and type=9 and mark=0 and offset=376;

If that succeeds, try a full multiple insert that has failed and been
logged.

If that succeeds, then the recordedseek table is likely working
properly again and you can try starting mythbackend and running
mythcommflag --rebuild.

If you can not get recordedseek to work, then the next thing to try is
deleting just that table and restoring it from a backup. You do that
by first finding a non-corrupt backup file. Backups made from a
corrupt table will often stop part way through the backup, and this
normally results in the compressed backup file being significantly
smaller than previous compressed backup files. So look at your backup
files and see if that has happened, and choose the bigger file before
the smaller ones started to occur. Also look back in your mythbackend
log files and see if you can see when the error messages first
started. You may not be able to if it was too long ago, as the
mythbackend logs where it happened may have been rotated and deleted.
In which case you may want to try restoring from the oldest backup you
have.

Uncompress the chosen backup file, then edit the resulting .sql file
and cut out only the recordedseek table, including all the commands
that drop and restore it, just above the data. Save this to a file,
perhaps called recordedseek.sql.

Shut down mythbackend and attempt to make another database backup
using mythconverg_backup.pl, making sure to have first saved all the
old backups.

Then run this command from the directory where the recordedseek.sql
file is:

sudo mysql mythconverg

and from the MySQL/MariaDB prompt, run this command:

source recordedseek.sql

It is possible that you might get an error message about the "source"
command if your database has the wrong options set, in which case you
will need to either change that option, or try using the "mysql
mythconverg" command with an option to pipe it the recordedseek.sql
file. I can not remember the exact syntax for doing that at the
moment.

Once you have restored recordedseek, you will need to work out what
recordings you have that now do not have recordedseek entries and will
need mythcommflag --rebuild done for them. Doing SQL to compare the
chanid/starttime values in the recorded table and the recordedseek
table and listing those not in both should do that for you. Then you
will need to run mythcommflag for all those recordings. There is also
a new command I have never tried:

mythutil --checkrecordings

which may be able to tell you which recordings do not have seek
tables, and:

mythutil --checkrecordings --fixseetable

may then run mythcommflag --rebuild on them all automatically.

If your recordedseek table is unable to be recovered by restoring a
backup, you can still recover it as mythcommflag is able to recreate
all the data. So you would take your recordedseek.sql file and delete
all the commands that restore the data, then run that as above to
delete and re-create an empty recordedseek table. Then you can run:

mythutil --checkrecordings --fixseektable

and all the basic data (not the commercial flagging data) should be
re-created. If you have many recordings, it will of course take a
very long time.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sun, Apr 9, 2023 at 10:16?PM Ross Boylan
<rossboylan@stanfordalumni.org> wrote:

> 2. Databases are supposed to be corruption resistant, though
> mysql/maria may be less so (I know that used to be true).

The default (and supported) DB engine for
MythTV is MyISAM, which, explicitly, is NOT
crash or corruption resistant.

On a long term TODO list of some devs is
to verify/validate that MythTV works (for
MariaDB) with Aria (with is a corruption
resistant alternative to MySAM with high
MyISAM compatibility), or InnoDB (which
has some differences with MyISAM that
need to be considered).

Personally, I have moved all my DB tables
to InnoDB, but I have also played DBA in
past lives, so know how to carefully tune
and adjust the result (MyISAM, as used for
MythTV, is basically a set and forget solution).
My database is technically considered "corrupt"
by the project's definition (wrong DB engine),
so I can never, ever, ask for assistance on
this list, and I will never ever do so. If you
are not tall enough to ride the ride and do the
same, do not follow my approaches.

As Oracle (reportedly) pays Canonical to
promote MySQL over MariaDB in their
distro, and Ubuntu is reported to be the
most common OS for MythTV (which
may not reflect actual usage but just
reporting, but we do not know the full
truth), InnoDB may be the better choice
if and when some dev decides to
prioritize the validations even if Aria
could mean less work for a dev.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Stopped mythbackend and performed the repair steps

> sudo mysqlcheck --repair mythconverg recordedseek
> sudo mysqlcheck --optimize mythconverg recordedseek
> sudo mysqlcheck --analyze mythconverg recordedseek
>
No errors reported. Did same for recordedmarkup, also no errors.

Then I tried (had to delete the final Z in the timestamp)
INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`) VALUES
(10501,'2023-04-08T04:59:00',9,0,376);
result:
MariaDB [mythconverg]> ERROR 1062 (23000): Duplicate entry
'10501-2023-04-08 04:59:00-9-0' for key 'PRIMARY'
I verified that there was an existing entry already. As with the
duplicates in recordedmark, this doesn't actually seem to be a problem with
the database, but a problem with the program or, in this case, me, for
trying to insert a duplicate.

Further background info in response to previous points in this thread
====================================================

The logs from the initial shutdown show lots of systemd units stopping but
*not* maria. This supports the theory it did not shutdown clean. The
first occurrence of these errors was with a show that was recording before
the shutdown and that finished recording after the recovery.

The semi-crash involves the video driver and Xorg using 100% of a CPU. I
was not watching myth at the time it happened. That would seem to leave
the non-graphic parts of the system functional. But myth did lose a
connection to an HDHomeRun tuner at the same time the video went crazy. So
some other systems were affected.

Should this situation recur (unfortunately it happens regularly) I should
attempt to shutdown mythtv-backend and then mariadb separately, verify they
have completed, and only then bring the system down. Of course, maybe the
database was already corrupt and a regular shutdown database shutdown will
not be possible--almost certainly the shutdown procedure attempted to stop
it.

Since something is clearly screwy, it does seem reasonable to assume the
database was corrupted.

As packaged by Christian Marillat at deb-multimedia.org, mythtv is backed
up weekly with a cycle of 12. So I have a last good backup on Mar 31,
7am. That's ~4.5 days before the crash, and about 10 days old now.

Debian went with mariadb, not mysql, as a default, and that's what
mythconverg is in for me. All the recorded* tables (probably all tables)
are using MyISAM as the engine.

Ross

On Mon, 10 Apr 2023 14:08:45 -0700, you wrote:

>Stopped mythbackend and performed the repair steps
>
>> sudo mysqlcheck --repair mythconverg recordedseek
>> sudo mysqlcheck --optimize mythconverg recordedseek
>> sudo mysqlcheck --analyze mythconverg recordedseek
>>
>No errors reported. Did same for recordedmarkup, also no errors.
>
>Then I tried (had to delete the final Z in the timestamp)
>INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`) VALUES
>(10501,'2023-04-08T04:59:00',9,0,376);
>result:
>MariaDB [mythconverg]> ERROR 1062 (23000): Duplicate entry
>'10501-2023-04-08 04:59:00-9-0' for key 'PRIMARY'
>I verified that there was an existing entry already. As with the
>duplicates in recordedmark, this doesn't actually seem to be a problem with
>the database, but a problem with the program or, in this case, me, for
>trying to insert a duplicate.

Using mythcommflag --rebuild should not cause duplicates, as
mythcommflag deletes all recordedseek rows that match the recording
before it creates the new ones.

>Further background info in response to previous points in this thread
>====================================================
>
>The logs from the initial shutdown show lots of systemd units stopping but
>*not* maria. This supports the theory it did not shutdown clean. The
>first occurrence of these errors was with a show that was recording before
>the shutdown and that finished recording after the recovery.
>
>The semi-crash involves the video driver and Xorg using 100% of a CPU. I
>was not watching myth at the time it happened. That would seem to leave
>the non-graphic parts of the system functional. But myth did lose a
>connection to an HDHomeRun tuner at the same time the video went crazy. So
>some other systems were affected.
>
>Should this situation recur (unfortunately it happens regularly) I should
>attempt to shutdown mythtv-backend and then mariadb separately, verify they
>have completed, and only then bring the system down. Of course, maybe the
>database was already corrupt and a regular shutdown database shutdown will
>not be possible--almost certainly the shutdown procedure attempted to stop
>it.

You may not be able to get a good shutdown of MariaDB in this
situation in the future. So when rebooting afterwards, make sure to
always run optimize_mythdb.

>Since something is clearly screwy, it does seem reasonable to assume the
>database was corrupted.
>
>As packaged by Christian Marillat at deb-multimedia.org, mythtv is backed
>up weekly with a cycle of 12. So I have a last good backup on Mar 31,
>7am. That's ~4.5 days before the crash, and about 10 days old now.
>
>Debian went with mariadb, not mysql, as a default, and that's what
>mythconverg is in for me. All the recorded* tables (probably all tables)
>are using MyISAM as the engine.
>
>Ross

I would recommend daily database backups - I do both weekly and daily
backups, to two different locations.

Video driver problems are a not infrequent cause of system lockups and
crashes. If you are using the Nvidia drivers, then try to find a
version that does not cause the problem, such as an older version of
the same main version, or going to a later main version (eg going from
515 to 525). On Ubuntu, there is a repository that allows you to use
later versions than are in the main repository. It is there so that
you can use newer Nvidia cards that need the latest drivers, but it
also helps in situations like this where a particular set of their
drivers are causing problems. Maybe Debian has one also.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sunday 09 April 2023 02:50:16 AM (-05:00), Ross Boylan wrote:

...

> Apr 8 22:06:30 barley mythbackend[2592]: INSERT INTO recordedmarkup
> (chanid, starttime, mark, type, data) VALUES ( 10203,
> '2023-04-09T02:54:00Z', 0, 33, 7950000);
> Apr 8 22:06:30 barley mythbackend[2592]: Bindings were:
> Apr 8 22:06:30 barley mythbackend[2592]: :CHANID=10203,
> :DATA=7950000, :STARTTIME=2023-04-09T02:54:00.000Z, :TYPE=33

What version of mariadb are you running? mariadb --version

https://forum.mythtv.org/viewtopic.php?p=25746#p25746
--
Bill
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

mariadb --version
mariadb Ver 15.1 Distrib 10.5.19-MariaDB, for debian-linux-gnu (x86_64)
using EditLine wrapper

The thread you cite is interesting because it shows a recent update to
maria.
The last on my system seems (in dpkg.log) from 2023-03-11:
upgrade mariadb-server-core-10.5:amd64 1:10.5.18-0+deb11u1
1:10.5.19-0+deb11u1
which was quite awhile before I noticed a problem. This is a later version
than reported in the post you linked to. However, the system was up
continuously from 2023-03-05 until the problems on 2023-04-04, so maybe the
old binaries were in use until then.. Although I think the upgrade stops
the database, and I think I remember waiting til myth wasn't active before
applying the update to mariadb.

Ross

On Mon, Apr 10, 2023 at 7:01?PM Bill Meek <keemllib@gmail.com> wrote:

>
> On Sunday 09 April 2023 02:50:16 AM (-05:00), Ross Boylan wrote:
>
> ...
>
> > Apr 8 22:06:30 barley mythbackend[2592]: INSERT INTO recordedmarkup
> > (chanid, starttime, mark, type, data) VALUES ( 10203,
> > '2023-04-09T02:54:00Z', 0, 33, 7950000);
> > Apr 8 22:06:30 barley mythbackend[2592]: Bindings were:
> > Apr 8 22:06:30 barley mythbackend[2592]: :CHANID=10203,
> > :DATA=7950000, :STARTTIME=2023-04-09T02:54:00.000Z, :TYPE=33
>
> What version of mariadb are you running? mariadb --version
>
> https://forum.mythtv.org/viewtopic.php?p=25746#p25746
> --
> Bill
>

On Mon, Apr 10, 2023 at 6:24?PM Stephen Worthington <
stephen_agent@jsw.gen.nz> wrote:

> On Mon, 10 Apr 2023 14:08:45 -0700, you wrote:
>
> >Stopped mythbackend and performed the repair steps
> >
> >> sudo mysqlcheck --repair mythconverg recordedseek
> >> sudo mysqlcheck --optimize mythconverg recordedseek
> >> sudo mysqlcheck --analyze mythconverg recordedseek
> >>
> >No errors reported. Did same for recordedmarkup, also no errors.
> >
> >Then I tried (had to delete the final Z in the timestamp)
> >INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`) VALUES
> >(10501,'2023-04-08T04:59:00',9,0,376);
> >result:
> >MariaDB [mythconverg]> ERROR 1062 (23000): Duplicate entry
> >'10501-2023-04-08 04:59:00-9-0' for key 'PRIMARY'
> >I verified that there was an existing entry already. As with the
> >duplicates in recordedmark, this doesn't actually seem to be a problem
> with
> >the database, but a problem with the program or, in this case, me, for
> >trying to insert a duplicate.
>
> Using mythcommflag --rebuild should not cause duplicates, as
> mythcommflag deletes all recordedseek rows that match the recording
> before it creates the new ones.


Since mythcommflag shouldn't cause duplicates but it does apparently,
several possibilities occur:
1. The delete operations that clear things out are failing, in part or in
full.
2. The delete operations are not getting sequenced by the db ahead of the
subsequent writes.
3. mythcommflag is producing duplicates. As odd as that sounds, it's the
same thing that seems to be happening with the original problems of
duplicates in the recordedmarkup table, which is somehow getting attempted
writes of 2 different times for total duration.
4. My manual experiment got a duplicate warning because the bulk insert
from which the single insert was taken succeeded partially. So that
particular record was written. That leaves the source of the original
errors unclear.
5. The duplicate warning is itself spurious.

I'm having trouble imagining how a problem with the database would cause
the program to start producing duplicates, if it is not a failed deletion.
Then again, I'm having trouble imagining why it would produce 2 different
total durations at all.

It might be relevant that I run commercial flagging while recording the
show. The end of the recording and of the processing of the commercials
may happen fairly soon after one another, and I suppose each could generate
a total time for the recording. But wouldn't it be the same total time?

Thanks for your other points too. I understand the db may not be
shutdownable. Debian probably has more recent drivers available for more
cutting edge flavors; I might try them.

I'll consider more frequent backups once I have a database that doesn't
seem messed up.

Does the backup produce valid results even if the db has other activity
going on?

Ross

On 12/04/2023 10:41, Ross Boylan wrote:
>
>
> On Mon, Apr 10, 2023 at 6:24?PM Stephen Worthington
> <stephen_agent@jsw.gen.nz> wrote:
>
> On Mon, 10 Apr 2023 14:08:45 -0700, you wrote:
>
> >Stopped mythbackend and performed the repair steps
> >
> >> sudo mysqlcheck --repair mythconverg recordedseek
> >> sudo mysqlcheck --optimize mythconverg recordedseek
> >> sudo mysqlcheck --analyze mythconverg recordedseek
> >>
> >No errors reported.  Did same for recordedmarkup, also no errors.
> >
> >Then I tried (had to delete the final Z in the timestamp)
> >INSERT INTO recordedseek (chanid, starttime, type, mark,
> `offset`) VALUES
> >(10501,'2023-04-08T04:59:00',9,0,376);
> >result:
> >MariaDB [mythconverg]> ERROR 1062 (23000): Duplicate entry
> >'10501-2023-04-08 04:59:00-9-0' for key 'PRIMARY'
> >I verified that there was an existing entry already.  As with the
> >duplicates in recordedmark, this doesn't actually seem to be a
> problem with
> >the database, but a problem with the program or, in this case,
> me, for
> >trying to insert a duplicate.
>
> Using mythcommflag --rebuild should not cause duplicates, as
> mythcommflag deletes all recordedseek rows that match the recording
> before it creates the new ones.
>
>
> Since mythcommflag shouldn't cause duplicates but it does apparently,
> several possibilities occur:
> 1. The delete operations that clear things out are failing, in part or
> in full.
> 2. The delete operations are not getting sequenced by the db ahead of
> the subsequent writes.
> 3. mythcommflag is producing duplicates.  As odd as that sounds, it's
> the same thing that seems to be happening with the original problems
> of duplicates in the recordedmarkup table, which is somehow getting
> attempted writes of 2 different times for total duration.
> 4. My manual experiment  got a duplicate warning because the bulk
> insert from which the single insert was taken succeeded partially.  So
> that particular record was written.  That leaves the source of the
> original errors unclear.
> 5. The duplicate warning is itself spurious.
>
> I'm having trouble imagining how a problem with the database would
> cause the program to start producing duplicates, if it is not a failed
> deletion.  Then again, I'm having trouble imagining why it would
> produce 2 different total durations at all.
>
> It might be relevant that I run commercial flagging while recording
> the show.  The end of the recording and of the processing of the
> commercials may happen fairly soon after one another, and I suppose
> each could generate a total time for the recording.  But wouldn't it
> be the same total time?
>
I wonder if this is related to
https://forum.mythtv.org/viewtopic.php?f=36&t=5349 "No longer able to
delete recordings from Mythfrontend"

The Z on the end of the date could be the issue, which is unsupported in
mysql.

--
'ooroo

Stinga...(:)-)
---------------------------------------------------
Email:stinga+mythtv@wolf-rock.com o
You need only two tools. o /////
A hammer and duct tape. If it /@ `\ /) ~
doesn't move and it should use > (O) X< ~ Fish!!
the hammer. If it moves and `\___/' \) ~
shouldn't, use the tape. \\\
---------------------------------------------------

On Tuesday 11 April 2023 09:48:29 PM (-05:00), stinga via mythtv-users wrote:


On 12/04/2023 10:41, Ross Boylan wrote:





On Mon, Apr 10, 2023 at 6:24?PM Stephen Worthington <stephen_agent@jsw.gen.nz> wrote:

On Mon, 10 Apr 2023 14:08:45 -0700, you wrote:

>Stopped mythbackend and performed the repair steps
>
>> sudo mysqlcheck --repair mythconverg recordedseek
>> sudo mysqlcheck --optimize mythconverg recordedseek
>> sudo mysqlcheck --analyze mythconverg recordedseek
>>
>No errors reported. Did same for recordedmarkup, also no errors.
>
>Then I tried (had to delete the final Z in the timestamp)
>INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`) VALUES
>(10501,'2023-04-08T04:59:00',9,0,376);
>result:
>MariaDB [mythconverg]> ERROR 1062 (23000): Duplicate entry
>'10501-2023-04-08 04:59:00-9-0' for key 'PRIMARY'
>I verified that there was an existing entry already. As with the
>duplicates in recordedmark, this doesn't actually seem to be a problem with
>the database, but a problem with the program or, in this case, me, for
>trying to insert a duplicate.

Using mythcommflag --rebuild should not cause duplicates, as
mythcommflag deletes all recordedseek rows that match the recording
before it creates the new ones.


Since mythcommflag shouldn't cause duplicates but it does apparently, several possibilities occur:
1. The delete operations that clear things out are failing, in part or in full.

2. The delete operations are not getting sequenced by the db ahead of the subsequent writes.
3. mythcommflag is producing duplicates. As odd as that sounds, it's the same thing that seems to be happening with the original problems of duplicates in the recordedmarkup table, which is somehow getting attempted writes of 2 different times for total duration.
4. My manual experiment got a duplicate warning because the bulk insert from which the single insert was taken succeeded partially. So that particular record was written. That leaves the source of the original errors unclear.

5. The duplicate warning is itself spurious.



I'm having trouble imagining how a problem with the database would cause the program to start producing duplicates, if it is not a failed deletion. Then again, I'm having trouble imagining why it would produce 2 different total durations at all.


It might be relevant that I run commercial flagging while recording the show. The end of the recording and of the processing of the commercials may happen fairly soon after one another, and I suppose each could generate a total time for the recording. But wouldn't it be the same total time?



I wonder if this is related to https://forum.mythtv.org/viewtopic.php?f=36&t=5349 "No longer able to delete recordings from Mythfrontend"

The Z on the end of the date could be the issue, which is unsupported in mysql.


He's on mariadb Ver 15.1 Distrib 10.5.19-MariaDB, (I asked yesterday.)


Easy test. Delete a recording. If it comes back, then the issue matches.


I pushed a fix for master and v33, but stopped because the delete recordings issue isn't the only
user of that type of UPDATE. The forum comment and links from Roland point at a Qt issue. The
final patch is in my tree and I'm testing now.


--
Bill

I tried manually deleting one of the records with type 33 in
recordedmarkup. The deletion worked. Another theory apparently shot down.

The forum post was about deleting recordings from the client; my
speculation was about failed deletion of individual database records. So
somewhat different.

The Z in the time is kind of weird, since, as I found, it is not legal
input syntax. But I think that's just an output formatting issue, since
the errors I'm seeing are duplicate key errors, not invalid data format
errors.
Ross

On Tue, Apr 11, 2023 at 7:48?PM stinga via mythtv-users <
mythtv-users@mythtv.org> wrote:

> On 12/04/2023 10:41, Ross Boylan wrote:
>
>
>
> On Mon, Apr 10, 2023 at 6:24?PM Stephen Worthington <
> stephen_agent@jsw.gen.nz> wrote:
>
>> On Mon, 10 Apr 2023 14:08:45 -0700, you wrote:
>>
>> >Stopped mythbackend and performed the repair steps
>> >
>> >> sudo mysqlcheck --repair mythconverg recordedseek
>> >> sudo mysqlcheck --optimize mythconverg recordedseek
>> >> sudo mysqlcheck --analyze mythconverg recordedseek
>> >>
>> >No errors reported. Did same for recordedmarkup, also no errors.
>> >
>> >Then I tried (had to delete the final Z in the timestamp)
>> >INSERT INTO recordedseek (chanid, starttime, type, mark, `offset`) VALUES
>> >(10501,'2023-04-08T04:59:00',9,0,376);
>> >result:
>> >MariaDB [mythconverg]> ERROR 1062 (23000): Duplicate entry
>> >'10501-2023-04-08 04:59:00-9-0' for key 'PRIMARY'
>> >I verified that there was an existing entry already. As with the
>> >duplicates in recordedmark, this doesn't actually seem to be a problem
>> with
>> >the database, but a problem with the program or, in this case, me, for
>> >trying to insert a duplicate.
>>
>> Using mythcommflag --rebuild should not cause duplicates, as
>> mythcommflag deletes all recordedseek rows that match the recording
>> before it creates the new ones.
>
>
> Since mythcommflag shouldn't cause duplicates but it does apparently,
> several possibilities occur:
> 1. The delete operations that clear things out are failing, in part or in
> full.
> 2. The delete operations are not getting sequenced by the db ahead of the
> subsequent writes.
> 3. mythcommflag is producing duplicates. As odd as that sounds, it's the
> same thing that seems to be happening with the original problems of
> duplicates in the recordedmarkup table, which is somehow getting attempted
> writes of 2 different times for total duration.
> 4. My manual experiment got a duplicate warning because the bulk insert
> from which the single insert was taken succeeded partially. So that
> particular record was written. That leaves the source of the original
> errors unclear.
> 5. The duplicate warning is itself spurious.
>
> I'm having trouble imagining how a problem with the database would cause
> the program to start producing duplicates, if it is not a failed deletion.
> Then again, I'm having trouble imagining why it would produce 2 different
> total durations at all.
>
> It might be relevant that I run commercial flagging while recording the
> show. The end of the recording and of the processing of the commercials
> may happen fairly soon after one another, and I suppose each could generate
> a total time for the recording. But wouldn't it be the same total time?
>
> I wonder if this is related to
> https://forum.mythtv.org/viewtopic.php?f=36&t=5349 "No longer able to
> delete recordings from Mythfrontend"
>
> The Z on the end of the date could be the issue, which is unsupported in
> mysql.
>
> --
> 'ooroo
>
> Stinga...(:)-)
> ---------------------------------------------------
> Email: stinga+mythtv@wolf-rock.com o
> You need only two tools. o /////
> A hammer and duct tape. If it /@ `\ /) ~
> doesn't move and it should use > (O) X< ~ Fish!!
> the hammer. If it moves and `\___/' \) ~
> shouldn't, use the tape. \\\
> ---------------------------------------------------
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>

The Z in the time indicates that the time listed is in Zulu (Greenwich
Mean Time).

On 4/12/23 17:27, Ross Boylan wrote:
> I tried manually deleting one of the records with type 33 in
> recordedmarkup.  The deletion worked.  Another theory apparently shot
> down.
>
> The forum post was about deleting recordings from the client; my
> speculation was about failed deletion of individual database records. 
> So somewhat different.
>
> The Z in the time is kind of weird, since, as I found, it is not legal
> input syntax.  But I think that's just an output formatting issue,
> since the errors I'm seeing are duplicate key errors, not invalid data
> format errors.
> Ross
>
> On Tue, Apr 11, 2023 at 7:48?PM stinga via mythtv-users
> <mythtv-users@mythtv.org> wrote:
>
> On 12/04/2023 10:41, Ross Boylan wrote:
>>
>>
>> On Mon, Apr 10, 2023 at 6:24?PM Stephen Worthington
>> <stephen_agent@jsw.gen.nz> wrote:
>>
>> On Mon, 10 Apr 2023 14:08:45 -0700, you wrote:
>>
>> >Stopped mythbackend and performed the repair steps
>> >
>> >> sudo mysqlcheck --repair mythconverg recordedseek
>> >> sudo mysqlcheck --optimize mythconverg recordedseek
>> >> sudo mysqlcheck --analyze mythconverg recordedseek
>> >>
>> >No errors reported.  Did same for recordedmarkup, also no
>> errors.
>> >
>> >Then I tried (had to delete the final Z in the timestamp)
>> >INSERT INTO recordedseek (chanid, starttime, type, mark,
>> `offset`) VALUES
>> >(10501,'2023-04-08T04:59:00',9,0,376);
>> >result:
>> >MariaDB [mythconverg]> ERROR 1062 (23000): Duplicate entry
>> >'10501-2023-04-08 04:59:00-9-0' for key 'PRIMARY'
>> >I verified that there was an existing entry already.  As
>> with the
>> >duplicates in recordedmark, this doesn't actually seem to be
>> a problem with
>> >the database, but a problem with the program or, in this
>> case, me, for
>> >trying to insert a duplicate.
>>
>> Using mythcommflag --rebuild should not cause duplicates, as
>> mythcommflag deletes all recordedseek rows that match the
>> recording
>> before it creates the new ones.
>>
>>
>> Since mythcommflag shouldn't cause duplicates but it does
>> apparently, several possibilities occur:
>> 1. The delete operations that clear things out are failing, in
>> part or in full.
>> 2. The delete operations are not getting sequenced by the db
>> ahead of the subsequent writes.
>> 3. mythcommflag is producing duplicates.  As odd as that sounds,
>> it's the same thing that seems to be happening with the original
>> problems of duplicates in the recordedmarkup table, which is
>> somehow getting attempted writes of 2 different times for total
>> duration.
>> 4. My manual experiment  got a duplicate warning because the bulk
>> insert from which the single insert was taken succeeded
>> partially.  So that particular record was written.  That leaves
>> the source of the original errors unclear.
>> 5. The duplicate warning is itself spurious.
>>
>> I'm having trouble imagining how a problem with the database
>> would cause the program to start producing duplicates, if it is
>> not a failed deletion.  Then again, I'm having trouble imagining
>> why it would produce 2 different total durations at all.
>>
>> It might be relevant that I run commercial flagging while
>> recording the show.  The end of the recording and of the
>> processing of the commercials may happen fairly soon after one
>> another, and I suppose each could generate a total time for the
>> recording.  But wouldn't it be the same total time?
>>
> I wonder if this is related to
> https://forum.mythtv.org/viewtopic.php?f=36&t=5349
> <https://forum.mythtv.org/viewtopic.php?f=36&t=5349> "No longer
> able to delete recordings from Mythfrontend"
>
> The Z on the end of the date could be the issue, which is
> unsupported in mysql.
>
> --
> 'ooroo
>
> Stinga...(:)-)
> ---------------------------------------------------
> Email:stinga+mythtv@wolf-rock.com o
> You need only two tools. o /////
> A hammer and duct tape. If it /@ `\ /) ~
> doesn't move and it should use > (O) X< ~ Fish!!
> the hammer. If it moves and `\___/' \) ~
> shouldn't, use the tape. \\\
> ---------------------------------------------------
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums:https://forum.mythtv.org

On Wed, 12 Apr 2023 19:56:54 -0700, you wrote:

>The Z in the time indicates that the time listed is in Zulu (Greenwich
>Mean Time).

You mean UTC (Coordinated Universal Time), which is the standard time
systems that is used now. GMT is a local time zone in the UK, if it
even still exists.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 13/04/2023 04:58, Stephen Worthington wrote:
> On Wed, 12 Apr 2023 19:56:54 -0700, you wrote:
>
>> The Z in the time indicates that the time listed is in Zulu (Greenwich
>> Mean Time).
>
> You mean UTC (Coordinated Universal Time), which is the standard time
> systems that is used now. GMT is a local time zone in the UK, if it
> even still exists.

TTBOMK GMT is still the basis of legal time in the UK. In that context
it is now taken to be identical to UTC.

I have no problems in MythTV when using the concise form:
{{{
chanid 11179 starttime 2023-04-12 10:58:00 recordedid 39272
Reformatted starttime 20230412105800
Running: mythutil --getcutlist --chanid 11179 --starttime 20230412105800 -q
Cutlist: 0-2681,91553-99020
}}}

John P

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 2023-04-13 09:26, John Pilkington wrote:
>
> TTBOMK GMT is still the basis of legal time in the UK. In that context
> it is now taken to be identical to UTC.
>
I drove through the mean streets of Greenwich a couple of weeks ago.
Time still exists there, though it drags a bit due to the 20mph speed
limit.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Thu, Apr 13, 2023 at 6:38?AM TimP <mythtv@corky.co> wrote:
>
> On 2023-04-13 09:26, John Pilkington wrote:
> >
> > TTBOMK GMT is still the basis of legal time in the UK. In that context
> > it is now taken to be identical to UTC.
> >
> I drove through the mean streets of Greenwich a couple of weeks ago.
> Time still exists there, though it drags a bit due to the 20mph speed
> limit.

Nice to have a laugh this morning!
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

UTC and GMT are names given to the time zone which lies on the 0 degree
of longitude. Its counterpart on the 180 degree of longitude is the
International Date Line.

I guess my preference for GMT is an indicator of my age more than
anything else.

On 4/12/23 20:58, Stephen Worthington wrote:
> On Wed, 12 Apr 2023 19:56:54 -0700, you wrote:
>
>> The Z in the time indicates that the time listed is in Zulu (Greenwich
>> Mean Time).
> You mean UTC (Coordinated Universal Time), which is the standard time
> systems that is used now. GMT is a local time zone in the UK, if it
> even still exists.
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Thu, Apr 13, 2023 at 9:28?AM Mike Carron <jmcarron@gmx.com> wrote:
>
> UTC and GMT are names given to the time zone which lies on the 0 degree
> of longitude. Its counterpart on the 180 degree of longitude is the
> International Date Line.
>
> I guess my preference for GMT is an indicator of my age more than
> anything else.

Don't forget Zulu :)

That is where the Z comes from on "old" timestamps.. Age is just a number!

> On 4/12/23 20:58, Stephen Worthington wrote:
> > On Wed, 12 Apr 2023 19:56:54 -0700, you wrote:
> >
> >> The Z in the time indicates that the time listed is in Zulu (Greenwich
> >> Mean Time).
> > You mean UTC (Coordinated Universal Time), which is the standard time
> > systems that is used now. GMT is a local time zone in the UK, if it
> > even still exists.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Thu, 13 Apr 2023 07:26:55 -0700, you wrote:

>UTC and GMT are names given to the time zone which lies on the 0 degree
>of longitude. Its counterpart on the 180 degree of longitude is the
>International Date Line.

Not exactly. GMT was Greenwich Mean Time, which was calculated and
maintained using a mean value of astronomical observations, originally
done at the Greenwich Observatory. But along came atomic clocks, and
the need for a more accurate time system for scientific purposes, and
later for everyday purposes (such as in the GPS system). So a new
time system was developed that is based on atomic clocks: Coordinated
Universal Time (UTC). They are complete time systems, not the names
of time zones, and when used at the same time actually provide
slightly different time, due to how they work. Once the UTC time
system was adopted, it rapidly became used world-wide, and the
maintenance of GMT was stopped (no more calculations were done and
published) and the GMT time system was abandoned and discontinued. But
the press and everyday usage seems to have kept on using GMT when they
actually mean UTC. And, it seems that in Britain, the name GMT has
been kept as the name of the local timezone, without daylight saving.
But it is now only the name of a timezone defined by the UTC system as
UTC+0, in the same way that EST is a name for the UTC-5 timezone.

The name UTC is the name of the entire modern time system, but is also
used as an abbreviation for the UTC+0 timezone (with no daylight
saving).

So GMT is acceptable as an alternative name for UTC+0 in Britain. It
is not acceptable as the name of the time system, or to be used to say
a timezone is GMT-10, unless you really mean the time is an old one
that was calculated back then using the GMT system.

>I guess my preference for GMT is an indicator of my age more than
>anything else.

Yes, I am also that old. But using GMT like that is actually
imprecise - you could be meaning the old GMT time system, rather than
the UTC+0 timezone, and they refer to slightly different times. So it
grates on the sensibilities of those of us who use UTC properly and
prefer to remove such imprecisions and ambiguities.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

A couple of different topics, but no good news.

DELETING RECORDINGS
=====================
I can't. Ignore my earlier report this was working; it seemed to work
because the deleted recording disappears from the frontend display.
But, as in the report cited earlier, it's back after restarting the
frontend.
As in that report, there is no sign of trouble in the myth or maria
logs. The former does show things like
Apr 14 13:05:37 barley mythbackend[401915]: 2023-04-14 13:05:37.209373
I Reschedule requested for CHECK 0 740 0 DoHandleDelete1 | ......
but it does not show the execution of the handler I hung off the
system delete event.

The last execution of that handler seems to have been on 2023-04-04
03:23, well before the system lockup at 14:58 the same day. It's
possible I didn't watch or delete anything in between.

So maybe this is related to the other problems I'm having. It is
unclear to me, after reading the whole thread at
https://forum.mythtv.org/viewtopic.php?f=36&t=5349 and some of the
bugs referenced from there, what the problem is. Some say it's the Z
on the time; others say it's the interaction of that with indexing,
others that it's in Qt (but the bugs there reference a change in Maria
10.6, and I'm on 10.5).

And my logs do not show any updates to any of the suspect packages
shortly (2 days) before the problem emerged.

REBUILD
========
I took the last presumed good backup, carved out the SQL for
recordedmarkup (and recordedseek, though I haven't used it yet),
stopped the myth backend, and updated recordedmarkup.
The first step in the update was to delete the old table.

I've attached the script I ran to extract the code, in case it helps others.

I have not issued any mythcommflag commands. Despite that I can view
some of the shows for which the entries are now missing.

But I continue to get errors about duplicate insertions into recordedmarkup.

There are a couple of things about the generated SQL that seemed odd.
I expected there would be a separate section setting up indices; there
is not. However, the keys are defined as part of the table creation,
and perhaps that's enough?

The backup script produced a bunch of stuff in comments; should I be
doing something with that material? E.g.,
---------------------------------------------------------------
DROP TABLE IF EXISTS `recordedmarkup`;
/*!40101 SET @saved_cs_client = @@character_set_client */;
/*!40101 SET character_set_client = utf8 */;
CREATE TABLE `recordedmarkup` (
`chanid` int(10) unsigned NOT NULL DEFAULT 0,
`starttime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
`mark` mediumint(8) unsigned NOT NULL DEFAULT 0,
`type` tinyint(4) NOT NULL DEFAULT 0,
`data` int(11) unsigned DEFAULT NULL,
PRIMARY KEY (`chanid`,`starttime`,`type`,`mark`)
) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;
/*!40101 SET character_set_client = @saved_cs_client */;

--
-- Dumping data for table `recordedmarkup`
--

LOCK TABLES `recordedmarkup` WRITE;
/*!40000 ALTER TABLE `recordedmarkup` DISABLE KEYS */;
---many INSERT here
/*!40000 ALTER TABLE `recordedmarkup` ENABLE KEYS */;
UNLOCK TABLES;

-------------------------------

Ross

On Fri, 14 Apr 2023 17:42:43 -0700, you wrote:

>A couple of different topics, but no good news.
>
>DELETING RECORDINGS
>=====================
>I can't. Ignore my earlier report this was working; it seemed to work
>because the deleted recording disappears from the frontend display.
>But, as in the report cited earlier, it's back after restarting the
>frontend.

Deleting is a multi-step process. When you delete a recording, it
gets put into a queue to be moved to the Deleted recording group. Then
that queue gets processed and each recording in the queue actually
gets put into the Deleted recording group. If you delete a big list
of recordings, such as deleting an entire show with 100 recordings in
it, you will see the recordings being moved to the Deleted group at a
measured pace, with each automatic update of the mythfrontend screen
showing fewer recordings until they are all gone. This is done so as
not to overload the system or the database which can affect recordings
in progress at the time. Once this is done, all the deleted
recordings will be waiting in the Delete recording group, but they
will still be present in the database.

Some time later (or not) depending on your settings, the actual delete
process happens, which is called "expiring" in the logs. I have the
default settings, to it normally takes up to about 15 minutes before a
deleted recording gets expired. Until then, I can display the Deleted
recordings group and undelete things from there. You can change the
settings for how long deleted recordings are kept, including having
them kept forever and only being expired when you run out of disk
space.

When a recording is expired, either because it was in the Deleted
group or because you ran out of space and mythbackend chose some
recordings to expire to make some free space, the recording gets put
on an expiry queue. Then mythbackend will go through the expiry queue
and actually delete each recording, removing the database entries and
deleting the recording files. Again, it does not do all the deletions
at once, as that can overload the system or the database and cause
problems with recordings happening at the time, so it does the
deletions relatively slowly, one at a time.

So, if you are not seeing deletions happening, it may be because you
have the option set to keep recordings forever and only expire them
when free space is needed. If so, you should be able to do this:

mythfrontend > Media Library > Watch Recordings > M(enu) > Change
Group Filter

You should see the Deleted group - select that. And then you should
have a big list of old deleted recordings showing.

The All Programmes listing does not include recordings in the Deleted
group, but does show recordings that are in any other recording group.

Having recordings reappear when you have deleted them suggests that
they were never moved to the Deleted recording group. Mythfrontend
removes the displayed line for a recording that you do a Delete
command on, but if the deletion is queued and you exit the recordings
list and re-enter it before the deletion is processed out of the
queue, mythfrontend will display the recording again, and you will see
it disappear when the queued deletion actually happens.

>As in that report, there is no sign of trouble in the myth or maria
>logs. The former does show things like
>Apr 14 13:05:37 barley mythbackend[401915]: 2023-04-14 13:05:37.209373
>I Reschedule requested for CHECK 0 740 0 DoHandleDelete1 | ......
>but it does not show the execution of the handler I hung off the
>system delete event.

I am not sure what the delete event is triggered from - I thought it
was from when you did a delete of a recording after watching it, when
it was put into the Deleted recording group, but it is possible it may
be from when the recording is actually expired. I do get delete
events on my mother's MythTV box where I have a script run from the
"all events" event.

>The last execution of that handler seems to have been on 2023-04-04
>03:23, well before the system lockup at 14:58 the same day. It's
>possible I didn't watch or delete anything in between.
>
>So maybe this is related to the other problems I'm having. It is
>unclear to me, after reading the whole thread at
>https://forum.mythtv.org/viewtopic.php?f=36&t=5349 and some of the
>bugs referenced from there, what the problem is. Some say it's the Z
>on the time; others say it's the interaction of that with indexing,
>others that it's in Qt (but the bugs there reference a change in Maria
>10.6, and I'm on 10.5).
>
>And my logs do not show any updates to any of the suspect packages
>shortly (2 days) before the problem emerged.
>
>REBUILD
>========
>I took the last presumed good backup, carved out the SQL for
>recordedmarkup (and recordedseek, though I haven't used it yet),
>stopped the myth backend, and updated recordedmarkup.
>The first step in the update was to delete the old table.
>
>I've attached the script I ran to extract the code, in case it helps others.
>
>I have not issued any mythcommflag commands. Despite that I can view
>some of the shows for which the entries are now missing.

Yes, the recordedseek and recordedmarkup table entries are not
necessary for viewing recordings - mythfrontend still does a
creditable job without them. But you will find the skipping around in
the recording does not work well, so skipping over ad breaks will be
annoyingly difficult.

>But I continue to get errors about duplicate insertions into recordedmarkup.
>
>There are a couple of things about the generated SQL that seemed odd.
>I expected there would be a separate section setting up indices; there
>is not. However, the keys are defined as part of the table creation,
>and perhaps that's enough?

The KEY phrase in the CREATE TABLE is all that is needed to set up the
indexes. The index files are also deleted by DROP TABLE commands,
along with the main table file.

>The backup script produced a bunch of stuff in comments; should I be
>doing something with that material? E.g.,
>---------------------------------------------------------------
>DROP TABLE IF EXISTS `recordedmarkup`;
>/*!40101 SET @saved_cs_client = @@character_set_client */;
>/*!40101 SET character_set_client = utf8 */;
>CREATE TABLE `recordedmarkup` (
> `chanid` int(10) unsigned NOT NULL DEFAULT 0,
> `starttime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
> `mark` mediumint(8) unsigned NOT NULL DEFAULT 0,
> `type` tinyint(4) NOT NULL DEFAULT 0,
> `data` int(11) unsigned DEFAULT NULL,
> PRIMARY KEY (`chanid`,`starttime`,`type`,`mark`)
>) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;
>/*!40101 SET character_set_client = @saved_cs_client */;
>
>--
>-- Dumping data for table `recordedmarkup`

The commented lines are something I have never fully worked out the
reason for. They are not necessary to restoring the table into an
existing mythconverg database. But maybe they would be needed if you
created a new database without the character set setup that is done
when mythconverg is created? The creation of the mythconverg database
is done by the /usr/share/mythtv/sql/mc.sql file in Ubuntu. It has:

CREATE DATABASE IF NOT EXISTS mythconverg;
CREATE USER IF NOT EXISTS 'mythtv'@'localhost' IDENTIFIED WITH
mysql_native_password;
ALTER USER 'mythtv'@'localhost' IDENTIFIED BY 'mythtv';
GRANT ALL ON mythconverg.* TO mythtv@localhost;
FLUSH PRIVILEGES;
GRANT CREATE TEMPORARY TABLES ON mythconverg.* TO mythtv@localhost;
FLUSH PRIVILEGES;
ALTER DATABASE mythconverg DEFAULT CHARACTER SET utf8 COLLATE
utf8_general_ci;

So it sets up the character set defaults there.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Fri, Apr 14, 2023 at 9:50?PM Stephen Worthington
<stephen_agent@jsw.gen.nz> wrote:
>
> On Fri, 14 Apr 2023 17:42:43 -0700, you wrote:
>
> >A couple of different topics, but no good news.
> >
> >DELETING RECORDINGS
> >=====================
> >I can't. Ignore my earlier report this was working; it seemed to work
> >because the deleted recording disappears from the frontend display.
> >But, as in the report cited earlier, it's back after restarting the
> >frontend.
>
> Deleting is a multi-step process. When you delete a recording, it
> gets put into a queue to be moved to the Deleted recording group. Then
> that queue gets processed and each recording in the queue actually
> gets put into the Deleted recording group. ....
>
> Some time later (or not) depending on your settings, the actual delete
> process happens, which is called "expiring" in the logs. I have the
> default settings, to it normally takes up to about 15 minutes before a
> deleted recording gets expired. Until then, I can display the Deleted
> recordings group and undelete things from there. You can change the
> settings for how long deleted recordings are kept, including having
> them kept forever and only being expired when you run out of disk
> space.
>
> When a recording is expired, either because it was in the Deleted
> group or because you ran out of space and mythbackend chose some
> recordings to expire to make some free space, the recording gets put
> on an expiry queue. Then mythbackend will go through the expiry queue
> and actually delete each recording, removing the database entries and
> deleting the recording files. Again, it does not do all the deletions
> at once, as that can overload the system or the database and cause
> problems with recordings happening at the time, so it does the
> deletions relatively slowly, one at a time.
>
> So, if you are not seeing deletions happening, it may be because you
> have the option set to keep recordings forever and only expire them
> when free space is needed. If so, you should be able to do this:

Recordings were deleting fine before the "crash" on 4/4. The actual
deletion--the "expiry" step--was typically 5-15 minutes after I
requested the delete.

>
> mythfrontend > Media Library > Watch Recordings > M(enu) > Change
> Group Filter
>
> You should see the Deleted group - select that. And then you should
> have a big list of old deleted recordings showing.

I have looked for the Deleted group several times recently; it is not present.
Again, I used to be able to see it while things were awaiting expiry.

>
> The All Programmes listing does not include recordings in the Deleted
> group, but does show recordings that are in any other recording group.
>
> Having recordings reappear when you have deleted them suggests that
> they were never moved to the Deleted recording group. ...
>
> >As in that report, there is no sign of trouble in the myth or maria
> >logs. The former does show things like
> >Apr 14 13:05:37 barley mythbackend[401915]: 2023-04-14 13:05:37.209373
> >I Reschedule requested for CHECK 0 740 0 DoHandleDelete1 | ......
> >but it does not show the execution of the handler I hung off the
> >system delete event.
>
> I am not sure what the delete event is triggered from - I thought it
> was from when you did a delete of a recording after watching it, when
> it was put into the Deleted recording group, but it is possible it may
> be from when the recording is actually expired. I do get delete
> events on my mother's MythTV box where I have a script run from the
> "all events" event.

My observation before the recent mess was that the delete event was
only triggered when the expiry occurred.
In my settings table it has value = EventCmdRecDeleted.

>
> >The last execution of that handler seems to have been on 2023-04-04
> >03:23, well before the system lockup at 14:58 the same day. It's
> >possible I didn't watch or delete anything in between.
> >
> >So maybe this is related to the other problems I'm having. It is
> >unclear to me, after reading the whole thread at
> >https://forum.mythtv.org/viewtopic.php?f=36&t=5349 and some of the
> >bugs referenced from there, what the problem is. Some say it's the Z
> >on the time; others say it's the interaction of that with indexing,
> >others that it's in Qt (but the bugs there reference a change in Maria
> >10.6, and I'm on 10.5).
> >
> >And my logs do not show any updates to any of the suspect packages
> >shortly (2 days) before the problem emerged.

On reflection, an old theory I proposed seems more plausible: that
updates applied weeks earlier did not become effective until the
system restarted. I have been assuming that some damage done in the
sort of crash on 4/4 is behind the problems. But perhaps it was the
simple act of restarting that mattered, allowing some old changes to
affect the running services. This still seems a little shaky, since
mariadb was certainly stopped and started when upgraded. But part of
my problems seem identical to the "can't delete recordings" thread,
and the others might be from problems interpreting the datetimes, esp
the final Z.
> >
> >REBUILD
> >========
...
>
> >The backup script produced a bunch of stuff in comments; should I be
> >doing something with that material? E.g.,
> >---------------------------------------------------------------
> >DROP TABLE IF EXISTS `recordedmarkup`;
> >/*!40101 SET @saved_cs_client = @@character_set_client */;
> >/*!40101 SET character_set_client = utf8 */;
> >CREATE TABLE `recordedmarkup` (
> > `chanid` int(10) unsigned NOT NULL DEFAULT 0,
> > `starttime` datetime NOT NULL DEFAULT '0000-00-00 00:00:00',
> > `mark` mediumint(8) unsigned NOT NULL DEFAULT 0,
> > `type` tinyint(4) NOT NULL DEFAULT 0,
> > `data` int(11) unsigned DEFAULT NULL,
> > PRIMARY KEY (`chanid`,`starttime`,`type`,`mark`)
> >) ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_general_ci;
> >/*!40101 SET character_set_client = @saved_cs_client */;
> >
> >--
> >-- Dumping data for table `recordedmarkup`
>
> The commented lines are something I have never fully worked out the
> reason for. They are not necessary to restoring the table into an
> existing mythconverg database. But maybe they would be needed if you
> created a new database without the character set setup that is done
> when mythconverg is created? ....

The stuff in the comments seems to be a mix of things essential for
correctness, like getting the character set right, and optimizations
like turning off the KEYS during a bulk INSERT. Plus there are the
numeric codes like 40101 at the start of the line.
Ah: they aren't comments, really; they are conditionally executable
SQL. They execute if the SQL version is >= the numeric code
immediately after the !.
https://dev.mysql.com/doc/refman/8.0/en/comments.html

Ross
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org