Mailing List Archive

myth_archive_job.pl fails as user job but works when run as mythtv user on CLI
Running 33.1 fixes

I'm running out of space until I get a new drive for the mythbox, so
mounted a windows share and set up a user job using the script at:

https://www.mythtv.org/wiki/Myth_archive_job.pl

I had used this in the past on another system and it worked. All it does is
move the recording file if there's enough space. I chose a recording and
selected the user job. It failed.

The backend log:
Nov 1 14:40:46 bhmf mythbackend: mythbackend[210595]: E UserJob_920
jobqueue.cpp:2468 (DoUserJobThread) JobQueue: User Job '/usr/bin/
myth_archive_job.pl -d /media/disk2/mythtv -f 15709_20231031235900.ts"'
failed.

When I ran /usr/bin/myth_archive_job.pl -d /media/disk2/mythtv -f
15709_20231031235900.ts from the command line as the mythtv user it worked.

Any idea why it would fail as a user job? How can I get some more info on
why it failed?

Thanks.
Re: myth_archive_job.pl fails as user job but works when run as mythtv user on CLI [ In reply to ]
On Wed, 1 Nov 2023 16:43:59 -0400, you wrote:

>Running 33.1 fixes
>
>I'm running out of space until I get a new drive for the mythbox, so
>mounted a windows share and set up a user job using the script at:
>
>https://www.mythtv.org/wiki/Myth_archive_job.pl
>
>I had used this in the past on another system and it worked. All it does is
>move the recording file if there's enough space. I chose a recording and
>selected the user job. It failed.
>
>The backend log:
>Nov 1 14:40:46 bhmf mythbackend: mythbackend[210595]: E UserJob_920
>jobqueue.cpp:2468 (DoUserJobThread) JobQueue: User Job '/usr/bin/
>myth_archive_job.pl -d /media/disk2/mythtv -f 15709_20231031235900.ts"'
>failed.
>
>When I ran /usr/bin/myth_archive_job.pl -d /media/disk2/mythtv -f
>15709_20231031235900.ts from the command line as the mythtv user it worked.
>
>Any idea why it would fail as a user job? How can I get some more info on
>why it failed?
>
>Thanks.

User jobs get run by mythbackend, and so are run using the same user
as mythbackend. Jobs run from the command line are run as the user
used to log on that command line. So they are likely different users
with different ownership and permissions on directories. So what are
the ownership and permissions on your archive directories? Are they
in the mythtv group?

There is actually no need to run a script like Myth_archive_job.pl to
move recording files - you can use any normal tool like mv from the
command line or a GUI tool. As long as you do it when MythTV is not
in use (not recording or playing a recording and not within 1 minute
of a recording start time), it is safe to move recording files. You
should have your archive directories in a storage group so that MythTV
can still find the recordings in them - I just created a storage group
called "Archives" and put mine in that. Then as long as you do not
have any recording rules that specifically say to record to any of the
"Archives" directories, mythbackend will not record there but will
find the files for playback and deletion.

I have written myself a Python program (mythsqu) that has an option to
fill up my "Archives" partitions from my recording partitions. It
moves the recordings oldest first, and monitors MythTV's state so it
will pause whenever MythTV is busy. Then when that has run (or when I
have added a new or larger recording drive), I also run my
balance_storage.sh script that moves recordings between my recording
drives until they have roughly equal free space. This is very useful
when I replace an old 4 Tbyte drive with a new 20 Tbyte one, so that
the free space is spread around and mythbackend will record evenly
across my seven recording drives. The balance_storage.sh script is
older and only monitors MythTV's status for upcoming recordings - as
soon as one is within 5 minutes it stops, so you need to run it again
later when MythTV is idle again.

http://www.jsw.gen.nz/mythtv/mythsgu
http://www.jsw.gen.nz/mythtv/balance_storage.sh

I would suggest that the simplest thing for you to do would be to wait
for MythTV to be idle, then use a GUI file management program that can
sort by timestamp to select the oldest files in each of your recording
directories and move them to the Windows drive. Then run
balance_storage.sh. When you get your new drive, you can just add it
to your normal "Default" storage group and move all the files from the
Windows drive over to it, then run balance_storage.sh again.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: myth_archive_job.pl fails as user job but works when run as mythtv user on CLI [ In reply to ]
On Wed, Nov 1, 2023 at 11:02?PM Stephen Worthington <
stephen_agent@jsw.gen.nz> wrote:

> On Wed, 1 Nov 2023 16:43:59 -0400, you wrote:
>
> >Running 33.1 fixes
> >
> >I'm running out of space until I get a new drive for the mythbox, so
> >mounted a windows share and set up a user job using the script at:
> >
> >https://www.mythtv.org/wiki/Myth_archive_job.pl
> >
> >I had used this in the past on another system and it worked. All it does
> is
> >move the recording file if there's enough space. I chose a recording and
> >selected the user job. It failed.
> >
> >The backend log:
> >Nov 1 14:40:46 bhmf mythbackend: mythbackend[210595]: E UserJob_920
> >jobqueue.cpp:2468 (DoUserJobThread) JobQueue: User Job '/usr/bin/
> >myth_archive_job.pl -d /media/disk2/mythtv -f 15709_20231031235900.ts"'
> >failed.
> >
> >When I ran /usr/bin/myth_archive_job.pl -d /media/disk2/mythtv -f
> >15709_20231031235900.ts from the command line as the mythtv user it
> worked.
> >
> >Any idea why it would fail as a user job? How can I get some more info on
> >why it failed?
> >
> >Thanks.
>
> User jobs get run by mythbackend, and so are run using the same user
> as mythbackend. Jobs run from the command line are run as the user
> used to log on that command line. So they are likely different users
> with different ownership and permissions on directories. So what are
> the ownership and permissions on your archive directories? Are they
> in the mythtv group?
>
> There is actually no need to run a script like Myth_archive_job.pl to
> move recording files - you can use any normal tool like mv from the
> command line or a GUI tool. As long as you do it when MythTV is not
> in use (not recording or playing a recording and not within 1 minute
> of a recording start time), it is safe to move recording files. You
> should have your archive directories in a storage group so that MythTV
> can still find the recordings in them - I just created a storage group
> called "Archives" and put mine in that. Then as long as you do not
> have any recording rules that specifically say to record to any of the
> "Archives" directories, mythbackend will not record there but will
> find the files for playback and deletion.
>
> I have written myself a Python program (mythsqu) that has an option to
> fill up my "Archives" partitions from my recording partitions. It
> moves the recordings oldest first, and monitors MythTV's state so it
> will pause whenever MythTV is busy. Then when that has run (or when I
> have added a new or larger recording drive), I also run my
> balance_storage.sh script that moves recordings between my recording
> drives until they have roughly equal free space. This is very useful
> when I replace an old 4 Tbyte drive with a new 20 Tbyte one, so that
> the free space is spread around and mythbackend will record evenly
> across my seven recording drives. The balance_storage.sh script is
> older and only monitors MythTV's status for upcoming recordings - as
> soon as one is within 5 minutes it stops, so you need to run it again
> later when MythTV is idle again.
>
> http://www.jsw.gen.nz/mythtv/mythsgu
> http://www.jsw.gen.nz/mythtv/balance_storage.sh
>
> I would suggest that the simplest thing for you to do would be to wait
> for MythTV to be idle, then use a GUI file management program that can
> sort by timestamp to select the oldest files in each of your recording
> directories and move them to the Windows drive. Then run
> balance_storage.sh. When you get your new drive, you can just add it
> to your normal "Default" storage group and move all the files from the
> Windows drive over to it, then run balance_storage.sh again.
>

That's the weird part. The backend runs as mythtv, I ran the successful
command line test as the mythtv user. And the windows share is mounted as
mythtv owner and group. Per the wiki (
https://www.mythtv.org/wiki/Setup_Storage_Directories) the permissions are
2755.

I'll take your advice and check out your scripts. Thanks.

BTW, you mentioned that you had 7 -recording- drives...what case are you
using?
Re: myth_archive_job.pl fails as user job but works when run as mythtv user on CLI [ In reply to ]
On Thu, 2 Nov 2023 01:04:03 -0400, you wrote:

>On Wed, Nov 1, 2023 at 11:02?PM Stephen Worthington <
>stephen_agent@jsw.gen.nz> wrote:
>
>> On Wed, 1 Nov 2023 16:43:59 -0400, you wrote:
>>
>> >Running 33.1 fixes
>> >
>> >I'm running out of space until I get a new drive for the mythbox, so
>> >mounted a windows share and set up a user job using the script at:
>> >
>> >https://www.mythtv.org/wiki/Myth_archive_job.pl
>> >
>> >I had used this in the past on another system and it worked. All it does
>> is
>> >move the recording file if there's enough space. I chose a recording and
>> >selected the user job. It failed.
>> >
>> >The backend log:
>> >Nov 1 14:40:46 bhmf mythbackend: mythbackend[210595]: E UserJob_920
>> >jobqueue.cpp:2468 (DoUserJobThread) JobQueue: User Job '/usr/bin/
>> >myth_archive_job.pl -d /media/disk2/mythtv -f 15709_20231031235900.ts"'
>> >failed.
>> >
>> >When I ran /usr/bin/myth_archive_job.pl -d /media/disk2/mythtv -f
>> >15709_20231031235900.ts from the command line as the mythtv user it
>> worked.
>> >
>> >Any idea why it would fail as a user job? How can I get some more info on
>> >why it failed?
>> >
>> >Thanks.
>>
>> User jobs get run by mythbackend, and so are run using the same user
>> as mythbackend. Jobs run from the command line are run as the user
>> used to log on that command line. So they are likely different users
>> with different ownership and permissions on directories. So what are
>> the ownership and permissions on your archive directories? Are they
>> in the mythtv group?
>>
>> There is actually no need to run a script like Myth_archive_job.pl to
>> move recording files - you can use any normal tool like mv from the
>> command line or a GUI tool. As long as you do it when MythTV is not
>> in use (not recording or playing a recording and not within 1 minute
>> of a recording start time), it is safe to move recording files. You
>> should have your archive directories in a storage group so that MythTV
>> can still find the recordings in them - I just created a storage group
>> called "Archives" and put mine in that. Then as long as you do not
>> have any recording rules that specifically say to record to any of the
>> "Archives" directories, mythbackend will not record there but will
>> find the files for playback and deletion.
>>
>> I have written myself a Python program (mythsqu) that has an option to
>> fill up my "Archives" partitions from my recording partitions. It
>> moves the recordings oldest first, and monitors MythTV's state so it
>> will pause whenever MythTV is busy. Then when that has run (or when I
>> have added a new or larger recording drive), I also run my
>> balance_storage.sh script that moves recordings between my recording
>> drives until they have roughly equal free space. This is very useful
>> when I replace an old 4 Tbyte drive with a new 20 Tbyte one, so that
>> the free space is spread around and mythbackend will record evenly
>> across my seven recording drives. The balance_storage.sh script is
>> older and only monitors MythTV's status for upcoming recordings - as
>> soon as one is within 5 minutes it stops, so you need to run it again
>> later when MythTV is idle again.
>>
>> http://www.jsw.gen.nz/mythtv/mythsgu
>> http://www.jsw.gen.nz/mythtv/balance_storage.sh
>>
>> I would suggest that the simplest thing for you to do would be to wait
>> for MythTV to be idle, then use a GUI file management program that can
>> sort by timestamp to select the oldest files in each of your recording
>> directories and move them to the Windows drive. Then run
>> balance_storage.sh. When you get your new drive, you can just add it
>> to your normal "Default" storage group and move all the files from the
>> Windows drive over to it, then run balance_storage.sh again.
>>
>
>That's the weird part. The backend runs as mythtv, I ran the successful
>command line test as the mythtv user. And the windows share is mounted as
>mythtv owner and group. Per the wiki (
>https://www.mythtv.org/wiki/Setup_Storage_Directories) the permissions are
>2755.
>
>I'll take your advice and check out your scripts. Thanks.
>
>BTW, you mentioned that you had 7 -recording- drives...what case are you
>using?

And 4 movies/videos/photos/music etc drives, plus 6 archive drives. I
used to have 3 internal drives in my old case, with another 3 on
internal cables pulled out the back of the case through an unused
slot. The drives were sitting on an old cloth rag for noise
insulation, but were fine without any fan just sitting in the open
air. The final recording drive was on an eSATA external mount. The
videos drives are in a 4 drive eSATA/USB 3 mount with a handle - so I
can take it with me and attach it to my laptop when away from home.
The archive drives are in two USB 3 mounts, one of 4 drives and one of
2 drives. They are on a zigbee remote controlled mains socket so that
I can turn them on and off using Home Assistant remote commands. I
have some nice udev scripts that someone wrote that will automount all
the archive drives when they are powered on, and I wrote a script that
unmounts them and puts them to sleep then commands Home Assistant to
turn their power off. So they are only powered on for a short time
whenever I need them.

Earlier this year I found my old case was dying - the LEDs and power
button stopped working. So I went looking for something to replace
it. I found an older design that was still in production, the
SilverStone Grandia GD08B HTPC case. It has two slots for optical
drives (I needed one), and I think up to 12 internal 3.5" drive slots,
although one of them is awkwardly placed on the side of the case and
may not fit depending on your motherboard. The other 11? are all in a
pull out drive carrier (along with the optical drives) and my 7 fitted
in very nicely. The space between the back of the drives in the
carrier and the cards fitted into the PCIe slots is tight, but it does
all fit. I have 4 PCIe cards: Nvidia GT1030, TBS6209 (8 x DVT-T2
tuners), TBS 6909 (8 x DVB-S2 tuners), Mellanox CX354A-QCBT (40 Gbit/s
Infiniband), so that does not help. The pull out drive carrier is an
excellent idea as it makes it easy to fit the drives. The B version
of the GD08 has a couple of updates to modernise the design, one of
which is USB 3 ports on the front instead of USB 2. I had to wait two
months for a new production run of the GD08B cases before they came
back in stock, but I am now very happy with the result. It is just as
quiet as my old case, even though it has 3 fans compared to 2
previously.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: myth_archive_job.pl fails as user job but works when run as mythtv user on CLI [ In reply to ]
On 02/11/2023 05:04, Ian Evans wrote:
>
> That's the weird part. The backend runs as mythtv, I ran the successful
> command line test as the mythtv user. And the windows share is mounted as
> mythtv owner and group. Per the wiki (
> https://www.mythtv.org/wiki/Setup_Storage_Directories) the permissions are
> 2755.
>
I would note that 2755 does not give /group/ mythtv write permissions. Although it is unlikely to
affect you if you are logged in as mythtv, it may come back to bite you in the future if you
overlook that fact.

--

Mike Perkins


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org
Re: myth_archive_job.pl fails as user job but works when run as mythtv user on CLI [ In reply to ]
On Wed, Nov 1, 2023 at 9:48?PM Ian Evans <dheianevans@gmail.com> wrote:

> Running 33.1 fixes
>
> I'm running out of space until I get a new drive for the mythbox, so
> mounted a windows share and set up a user job using the script at:
>
> https://www.mythtv.org/wiki/Myth_archive_job.pl
>
> I had used this in the past on another system and it worked. All it does
> is move the recording file if there's enough space. I chose a recording and
> selected the user job. It failed.
>
> The backend log:
> Nov 1 14:40:46 bhmf mythbackend: mythbackend[210595]: E UserJob_920
> jobqueue.cpp:2468 (DoUserJobThread) JobQueue: User Job '/usr/bin/
> myth_archive_job.pl -d /media/disk2/mythtv -f 15709_20231031235900.ts"'
> failed.
>
> When I ran /usr/bin/myth_archive_job.pl -d /media/disk2/mythtv -f
> 15709_20231031235900.ts from the command line as the mythtv user it worked.
>
> Any idea why it would fail as a user job? How can I get some more info on
> why it failed?
>
> Thanks.
>
>
Is it a typo or does the user-job command actually end with one unmatched
double quote (") ?
Check in mysql with
SELECT
<http://10.0.0.190/phpmyadmin/url.php?url=https://dev.mysql.com/doc/refman/5.5/en/select.html>
* FROM settings WHERE value LIKE
<http://10.0.0.190/phpmyadmin/url.php?url=https://dev.mysql.com/doc/refman/5.5/en/string-comparison-functions.html#operator_like>
'UserJob_'
Re: myth_archive_job.pl fails as user job but works when run as mythtv user on CLI [ In reply to ]
On Thu, Nov 2, 2023, 3:11?p.m. Roland Ernst <rcrernst@gmail.com> wrote:

>
>
> On Wed, Nov 1, 2023 at 9:48?PM Ian Evans <dheianevans@gmail.com> wrote:
>
>> Running 33.1 fixes
>>
>> I'm running out of space until I get a new drive for the mythbox, so
>> mounted a windows share and set up a user job using the script at:
>>
>> https://www.mythtv.org/wiki/Myth_archive_job.pl
>>
>> I had used this in the past on another system and it worked. All it does
>> is move the recording file if there's enough space. I chose a recording and
>> selected the user job. It failed.
>>
>> The backend log:
>> Nov 1 14:40:46 bhmf mythbackend: mythbackend[210595]: E UserJob_920
>> jobqueue.cpp:2468 (DoUserJobThread) JobQueue: User Job '/usr/bin/
>> myth_archive_job.pl -d /media/disk2/mythtv -f 15709_20231031235900.ts"'
>> failed.
>>
>> When I ran /usr/bin/myth_archive_job.pl -d /media/disk2/mythtv -f
>> 15709_20231031235900.ts from the command line as the mythtv user it worked.
>>
>> Any idea why it would fail as a user job? How can I get some more info on
>> why it failed?
>>
>> Thanks.
>>
>>
> Is it a typo or does the user-job command actually end with one unmatched
> double quote (") ?
> Check in mysql with
> SELECT
> <http://10.0.0.190/phpmyadmin/url.php?url=https://dev.mysql.com/doc/refman/5.5/en/select.html>
> * FROM settings WHERE value LIKE
> <http://10.0.0.190/phpmyadmin/url.php?url=https://dev.mysql.com/doc/refman/5.5/en/string-comparison-functions.html#operator_like>
> 'UserJob_'
>
>
Dang! Good eye! I'll check it out later.

>
>
Re: myth_archive_job.pl fails as user job but works when run as mythtv user on CLI [ In reply to ]
On Thu, Nov 2, 2023 at 3:45?PM Ian Evans <dheianevans@gmail.com> wrote:

> On Thu, Nov 2, 2023, 3:11?p.m. Roland Ernst <rcrernst@gmail.com> wrote:
>
>>
>>
>> On Wed, Nov 1, 2023 at 9:48?PM Ian Evans <dheianevans@gmail.com> wrote:
>>
>>> Running 33.1 fixes
>>>
>>> I'm running out of space until I get a new drive for the mythbox, so
>>> mounted a windows share and set up a user job using the script at:
>>>
>>> https://www.mythtv.org/wiki/Myth_archive_job.pl
>>>
>>> I had used this in the past on another system and it worked. All it does
>>> is move the recording file if there's enough space. I chose a recording and
>>> selected the user job. It failed.
>>>
>>> The backend log:
>>> Nov 1 14:40:46 bhmf mythbackend: mythbackend[210595]: E UserJob_920
>>> jobqueue.cpp:2468 (DoUserJobThread) JobQueue: User Job '/usr/bin/
>>> myth_archive_job.pl -d /media/disk2/mythtv -f 15709_20231031235900.ts"'
>>> failed.
>>>
>>> When I ran /usr/bin/myth_archive_job.pl -d /media/disk2/mythtv -f
>>> 15709_20231031235900.ts from the command line as the mythtv user it worked.
>>>
>>> Any idea why it would fail as a user job? How can I get some more info
>>> on why it failed?
>>>
>>> Thanks.
>>>
>>>
>> Is it a typo or does the user-job command actually end with one unmatched
>> double quote (") ?
>> Check in mysql with
>> SELECT
>> <http://10.0.0.190/phpmyadmin/url.php?url=https://dev.mysql.com/doc/refman/5.5/en/select.html>
>> * FROM settings WHERE value LIKE
>> <http://10.0.0.190/phpmyadmin/url.php?url=https://dev.mysql.com/doc/refman/5.5/en/string-comparison-functions.html#operator_like>
>> 'UserJob_'
>>
>>
> Dang! Good eye! I'll check it out later.
>
>>
>>
Yep, it was the double quote that was stopping it, Thanks again for the
sharp eye.