Mailing List Archive

On Tue, Dec 27, 2022 at 8:14 PM <f-myth-users@media.mit.edu> wrote:

> Is there a good way to debug
> it further if this isn't a server issue? Thanks...

I don't know if it is a good way, but I would try:

* Add the flags "--verbose general,xmltv" to your
mythfilldatabase run and the output from the
grabber itself should be in the log (which may,
or may not, be helpful, of course). I always
run mythfilldatabase with those flag options
so that there is a record of any anomalies
(if you did not log/measure it, it did not happen,
although some grabbers are themselves so
verbose if you take their defaults that the logs
are ugly).

* Run the grabber manually, using the command
line from the run above (and under the appropriate
userid) that the logs show mythfilldatabase used
to see if it works (you can change the output file
name if you wish, as it is just a generated temp file),
or any additional information is provided in the
logging.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

[I take it other people -are- getting data from tv_grab_zz_sdjson?]

Might be a locked sqlite DB, but I'm not sure how it's locked.
See debugging stream below:

I -had- been running MFDB like this:

$ /usr/bin/mythfilldatabase --verbose general --loglevel info --quiet --syslog local7

and just now ran it by hand, as mythtv, using this:

$ /usr/bin/mythfilldatabase --verbose general.xmltv

with no other args. ("whoami" returns "mythtv",
so I'm definitely the right user while doing this by-hand run.)

I see a line I didn't see with the prior logging:

2022-12-27 16:39:05.222227 I Unable to create settings table in database /home/mythtv/.xmltv/SchedulesDirect.DB: attempt to write a readonly database

I don't know if that's related to anything or if that's happened forever
and I just never saw it with the prior logging, but I'm betting it's new
and related; see below. That file exists and is writable and was last
updated the last time MFDB ran successfully:

-rw-r--r-- 1 mythtv mythtv 296087552 Dec 25 18:16 /home/mythtv/.xmltv/SchedulesDirect.DB

I'm a little suspicious in that I had -just- started to look into
a different issue just after that run, involving getting rid of some
channels which Myth kept trying to record from but weren't tunable.
(Long standing issue I was finally getting to.) The only thing I
did was to do this (as myself, not myth or root):

$ sdjson-sqlite-select --list --database /home/mythtv/.xmltv/SchedulesDirect.DB --file current-callsigns

and then I didn't touch that DB again. I'd also for the first time
run find_orphans but hadn't allowed it to do anything (^C'ed out).

The /home filesystem is not mounted readonly; "touch foo" worked fine.

Config file /home/mythtv/.mythtv/SD.xmltv last modified 18 months ago
when I installed on this machine.

I get the same error trying to run the grabber manually:

$ tv_grab_zz_sdjson_sqlite --days 0 --config-file /home/mythtv/.mythtv/SD.xmltv
Opening the local database
Unable to create settings table in database /home/mythtv/.xmltv/SchedulesDirect.DB: attempt to write a readonly database

and it quits. When I did this 18 months ago when I was installing it,
the next line was instead:

Obtaining authentication token for Schedules Direct

and then several more lines of log and then a giant pile of XML.

Is there some way in which sqlite could have set the DB readonly,
even if the file itself appears to be writable? This page:

https://stackoverflow.com/questions/1518729/change-sqlite-database-mode-to-read-write

talks about it potentially being locked by some other process, but

# lsof | grep /home/mythtv/.xmltv/SchedulesDirect.DB

dosen't show any other processes have it open. And the directory it's
in appears writable by mythtv as well, which apparently is another way
the DB can claim to be locked.

/home/mythtv/.xmltv:
total used in directory 289192 available 11044444
drwxrwxr-x 2 mythtv mythtv 4096 Dec 25 20:11 .
drwxr-xr-x 8 mythtv mythtv 4096 Aug 24 2021 ..
-rw-r--r-- 1 mythtv mythtv 296087552 Dec 25 18:16 SchedulesDirect.DB

This page:

https://www.arysontechnologies.com/blog/fix-sqlite-error-database-locked/

claims that one can break the lock by creating a backup and then
swapping DB files. I can try it if nobody has any better ideas, but it
seems a little drastic and perhaps incorrect---especially since the file
claims a last-write date of the end of the last successful MFDB run,
-not- a few hours later when I first ran sdjson-sqlite-select (if that
even has anything to do with it)---does a sqlite lock update the file
itself? Would a reboot clear it? (Which I'm not anxious to do if
there's an easier way.)

Any other things I can do to debug this?

Thanks!
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Resolved!

I haven't gone back and retested, but it appears that running
sdjson-sqlite-select as myself (not as mythtv) may have been the
proximate cause. I found two files, SchedulesDirect.DB-shm and
SchedulesDirect.DB-wal, owned by myself and not as mythtv, in
/home/mythtv/xmltv---which is possible because I'm also in the
mythtv group, so the group-writability of the dir allowed this.

I didn't know those files were present at the time,
but while they were, attempting (as mythtv) to do

$ sqlite3 /home/mythtv/.xmltv/SchedulesDirect.DB
sqlite> .backup main /tmp/backup-SD-DB

spun one core at 100% while producing a 512-byte journal file and
a 0-byte Db file in /tmp and doing no disk IO that vmstat could see.
After 20-30 minutes of this, I noticed the other files. I aborted the
backup (had to kill it as root; ^C in the shell did nothing), chowned
those files to mythtv:mythtv, and then retrying the backup finished
effectively instantly. And at some point those two files vanished.

Putting the backed-up DB file back in place of the original then allowed
a manual run of the grabber to work fine. I just kicked off MFDB and
it finished while I was writing this.

Thanks for pointing me at increased logging and at running the grabber
by hand; those were the critical things which enabled finding all the
other sqlite peculiarities which eventually resolved this. I will be
adding ",xmltv" to the "general" logging in the script and will ensure
I never, ever run sdjson-sqlite-select as a non-mythtv user, even if
I'm not intending to make any changes to the sqlite DB. I can't say
I'm happy with the resiliency of sqlite.

It might also be useful for someone (unsure who) to update the various
error-handling in the grabber/mfdb so that a failure to be able to create
tables in sqlite was reported as that, and the rest of the run -aborted-,
instead of then complaining about an empty file farther downstream,
which makes this look like a server problem when the problem was that
the grabber didn't even try to connect.

P.S. One confusing part of this is that I have phpliteadmin configured,
but since apache can't write to the directory, it claims that any DB in
the dir is unwritable as well, hence muddying the waters about when my
fix actually fixed anything. Emptying the directory of -all- files
caused phplitadmin to emit a more-helpful error message (basically,
"whole dir is unwritable so you can't create any DB's"), which told me I
shouldn't use it to evaluate whether I'd unlocked the DB. Fortunately
that was only a few minutes wasted, and the grabber tells the tale...
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 27/12/2022 23:31, f-myth-users@media.mit.edu wrote:
>
(snip)
>
> I will be
> adding ",xmltv" to the "general" logging in the script and will ensure
> I never, ever run sdjson-sqlite-select as a non-mythtv user, even if
> I'm not intending to make any changes to the sqlite DB. I can't say
> I'm happy with the resiliency of sqlite.
>
Just a note to say, if you do add ",xmltv" to your logging the log files could
become extremely large, depending on what changes are being made to schedules.

Make sure you have enough logging space!

Also, you point up a general rule: never, ever, run anything mythtv related except as the mythtv
user. The number of problems caused by running as, say, yourself or root, are legion.

--

Mike Perkins


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

> Date: Wed, 28 Dec 2022 09:52:28 +0000
> From: Mike Perkins <mikep@randomtraveller.org.uk>

> Just a note to say, if you do add ",xmltv" to your logging the log files could
> become extremely large, depending on what changes are being made to schedules.

> Make sure you have enough logging space!

I'll keep an eye on it. So far, adding that only added 20 or so lines
to mythfilldatabase.log, but I'll watch it run for a day or so and
ensure that. If it gets huge, I'll turn it off again. (And if that's
true, then it's quite unfortunate that detailed-enough logging to catch
simple grabber errors like "DB locked" will generate so much output that
such logging can't be left on by default; seems there should be an
intermedaite solution.)

> Also, you point up a general rule: never, ever, run anything mythtv related except as the mythtv
> user. The number of problems caused by running as, say, yourself or root, are legion.

Yeah. It would not have occurred to me that -anything- which touches
sqlite, even a read-only query, might leave extra files sitting around
in the DB dir owned by the user and which would then get in the way of
other use. I'm guessing that sqlite3 aborted in some way (for no reason
I can see, and with no diagnostic) and didn't clean up those files, but
still, its behavior in this situation was extremely poor---not only not
warning about them in normal use and instead declaring the DB locked,
but also infinite-looping when I tried to back it up. It's also
unfortunate that MFDB doesn't handle an aborted grabber better and give
a more informative error message, since it papered over the underlying
error and just complained that the grabber's output file was empty---
the grabber returning an error code should have caused MFDB to abort
immediately, propagating the error, instead of acting like nothing was
wrong and then instead complaining about the empty file. Correctly
propagating the error would have had me looking at the grabber first
thing, instead of assuming it was a temporary server issue, especially
given that default logging didn't show the error.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Mike Hi

> Also, you point up a general rule: never, ever, run anything mythtv related except as the mythtv user. The number of problems caused by running as, say, yourself or root, are legion.

Since day 0 (IIRC 0.15 days) I refused to have a mythtv user or a cretinacious mythtv password. I'm not dissing your advice :-) but asking about the problems that I should expect.
I bristle when the ubuntu hierarchy declare 'We think ..." "that you ..." dmesg is a recent example all the way back to no root
ie Disk problem at boot. Type root passwd to continue ...

James

On 28/12/2022 22:06, James wrote:
> Mike Hi
>
>> Also, you point up a general rule: never, ever, run anything mythtv related except as the mythtv user. The number of problems caused by running as, say, yourself or root, are legion.
>
> Since day 0 (IIRC 0.15 days) I refused to have a mythtv user or a cretinacious mythtv password. I'm not dissing your advice :-) but asking about the problems that I should expect.
> I bristle when the ubuntu hierarchy declare 'We think ..." "that you ..." dmesg is a recent example all the way back to no root
> ie Disk problem at boot. Type root passwd to continue ...
>

Fairly straightforward. All the installation scripts and run instructions assume that you run mythtv
from a dedicated mythtv user - which might not be named mythtv, of course.

Remember that in most instances a myth system is one or more appliances, designed to be installed
and then used almost as a black box. You're supposed to be using it either from a front end or one
of the web interfaces, not the command line.

If you let the install scripts run, at least on Debian or Ubuntu, they will create a mythtv user as
a system user as part of the installation. I tend to bypass that, since an upgrade/host
move/reinstall may change the UID/GID of a system user (and cause infinite chaos in /var). I create
a mythtv user first as a normal user and give it a real password, which is only used on the rare
occasions I have to ssh into the box.

Also remember that the password which is in config.xml is the database password, not the user
password. It is perfectly possible to have a mythtv user without a password, provided you have some
means of accessing that user's files when required.

(I run plain Debian, but keep a bargepole for dealing with systemd. Other OSs are available. My
mythtv system is on dedicated hosts on a separate subnet. Only ssh and mythweb have access from
other subnets.)

--

Mike Perkins


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

James <jam@tigger.ws> wrote:

> Since day 0 (IIRC 0.15 days) I refused to have a mythtv user or a cretinacious mythtv password. I'm not dissing your advice :-) but asking about the problems that I should expect.

As well as Mike’s advice, it’s not so much that you must have a MythTV user - but “everything” should be done as one user. So, for example, if you have one user that the system runs the Myth packages under, and then you run tv_grab_xxx and mythfilldatabase as a different user, you are likely to experience some issues.
These may be simple ones which can be fixed by applying suitable permissions to certain files (e.g. to allow the different user to access the config files needed to run mythfilldatabase), they may be more “interesting” (can’t think of any examples right now). An easy mistake to make is to use a different user to put some files into the video or recordings storage group and forget to change ownership - this may be a good thing (Myth can’t delete them), but it also can be a bad thing (e.g. Myth can’t delete them - a problem if you’ve been migrating systems and the recordings files aren’t owned by Myth).
But note that in the case of running a grabber and mythfilldatabase - some files get deleted and replaced, hence the ownership will change to the user running the programs. So without taking measures to avoid it, you have a situation where Myth running the grabber could fail after you’ve messed up the file permissions by running as a different user.

So like many things there are alternative versions of the advice, the simple advice is “run everything as your Myth user”, while the more correct version is “run everything as the Myth user unless you understand what you are doing and what the consequences are and aren’t going to moan about any issues that arise”.


TL;DR version
It’s a simple rule that will allow most people to do most things without problems - and all the install/configure scripts assume the use of one user for all Myth related tasks. If you understand the issues (mostly around permissions on files), then you can depart from that rule - but then you have to take responsibility for dealing with the issues that can occur.

Simon

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

> On 29 Dec 2022, at 9:18 pm, Simon <linux@thehobsons.co.uk> wrote:
>
> James <jam@tigger.ws> wrote:
>
>> Since day 0 (IIRC 0.15 days) I refused to have a mythtv user or a cretinacious mythtv password. I'm not dissing your advice :-) but asking about the problems that I should expect.
>
> As well as Mike’s advice, it’s not so much that you must have a MythTV user - but “everything” should be done as one user. So, for example, if you have one user that the system runs the Myth packages under, and then you run tv_grab_xxx and mythfilldatabase as a different user, you are likely to experience some issues.
> These may be simple ones which can be fixed by applying suitable permissions to certain files (e.g. to allow the different user to access the config files needed to run mythfilldatabase), they may be more “interesting” (can’t think of any examples right now). An easy mistake to make is to use a different user to put some files into the video or recordings storage group and forget to change ownership - this may be a good thing (Myth can’t delete them), but it also can be a bad thing (e.g. Myth can’t delete them - a problem if you’ve been migrating systems and the recordings files aren’t owned by Myth).
> But note that in the case of running a grabber and mythfilldatabase - some files get deleted and replaced, hence the ownership will change to the user running the programs. So without taking measures to avoid it, you have a situation where Myth running the grabber could fail after you’ve messed up the file permissions by running as a different user.
>
> So like many things there are alternative versions of the advice, the simple advice is “run everything as your Myth user”, while the more correct version is “run everything as the Myth user unless you understand what you are doing and what the consequences are and aren’t going to moan about any issues that arise”.
>
>
> TL;DR version
> It’s a simple rule that will allow most people to do most things without problems - and all the install/configure scripts assume the use of one user for all Myth related tasks. If you understand the issues (mostly around permissions on files), then you can depart from that rule - but then you have to take responsibility for dealing with the issues that can occur.
>
Actually Simon I like your answer thanks
And as I said at start "I'm not dissing" any opinions. I tend to forget that I've been playing with unix longer than some of these folk have been born while seriously contemplating "why"
James

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org