Mailing List Archive

How to get alerts on errors for cron jobs on toolforge
Hello all,

I have a bot at toolforge, running weekly once in cron job using "jsub"

Followed the below link for the setup.
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot

Getting the JOBNAME.out and JOBNAME.err files at the home folder.

Is there any other way like email notification to get alerts, if some
error happens?

I think the files JOBNAME.out and JOBNAME.err will be keep on growing
on the bason host login.toolforge.org

How to avoid this?

Thanks.

--
Regards,
T.Shrinivasan


My Life with GNU/Linux : http://goinggnu.wordpress.com
Free E-Magazine on Free Open Source Software in Tamil : http://kaniyam.com

Get Free Tamil Ebooks for Android, iOS, Kindle, Computer :
http://FreeTamilEbooks.com
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: How to get alerts on errors for cron jobs on toolforge [ In reply to ]
I think the only way is to have your cronjob do the emailing of the
output, and then delete (or move) the output file so it's not repeated
on the next run. Something like:

#!/bin/bash

## Call actual job script here.
./do-tool-things.sh

## Email output to maintainers.
echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain;
charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out |
/usr/sbin/exim -odf -i tools.toolname@tools.wmflabs.org

## Remove output file.
rm /data/project/toolname/cron-tools.jobname.out

Although of course the details might differ for different tools.


On 22/6/21 4:52 pm, Shrinivasan T wrote:
> Hello all,
>
> I have a bot at toolforge, running weekly once in cron job using "jsub"
>
> Followed the below link for the setup.
> https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot
>
> Getting the JOBNAME.out and JOBNAME.err files at the home folder.
>
> Is there any other way like email notification to get alerts, if some
> error happens?
>
> I think the files JOBNAME.out and JOBNAME.err will be keep on growing
> on the bason host login.toolforge.org
>
> How to avoid this?
>
> Thanks.
>
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
Re: How to get alerts on errors for cron jobs on toolforge [ In reply to ]
I do remember getting emails (last year?) from running cronjobs on the grid
until I disabled them, iirc it was when they failed, but not necessarily a
description of the error. I literally cannot find any documentation
regarding this anymore so I almost have to think if I didn't imagine this
entire ordeal.

Best,
QEDK

On Wed, Jun 23, 2021, 09:01 Sam Wilson <sam@samwilson.id.au> wrote:

> I think the only way is to have your cronjob do the emailing of the
> output, and then delete (or move) the output file so it's not repeated
> on the next run. Something like:
>
> #!/bin/bash
>
> ## Call actual job script here.
> ./do-tool-things.sh
>
> ## Email output to maintainers.
> echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain;
> charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out |
> /usr/sbin/exim -odf -i tools.toolname@tools.wmflabs.org
>
> ## Remove output file.
> rm /data/project/toolname/cron-tools.jobname.out
>
> Although of course the details might differ for different tools.
>
>
> On 22/6/21 4:52 pm, Shrinivasan T wrote:
> > Hello all,
> >
> > I have a bot at toolforge, running weekly once in cron job using "jsub"
> >
> > Followed the below link for the setup.
> > https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot
> >
> > Getting the JOBNAME.out and JOBNAME.err files at the home folder.
> >
> > Is there any other way like email notification to get alerts, if some
> > error happens?
> >
> > I think the files JOBNAME.out and JOBNAME.err will be keep on growing
> > on the bason host login.toolforge.org
> >
> > How to avoid this?
> >
> > Thanks.
> >
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
Re: How to get alerts on errors for cron jobs on toolforge [ In reply to ]
Cron should send mail with the output of the command in the crontab.
Because the grid crontab runs jsub, and not your command directly, the
only mail you will get from cron is when your job has been submitted.

You can use the -m option on jsub to send mail in a few different
cases (see https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#jsub_options).
jsub -m a ... should send mail when the job is aborted, but won't
include any more information.

You can also send mail directly from your code,
https://wikitech.wikimedia.org/wiki/Help:Toolforge/Email has some more
information. Many logging frameworks, including `logging` in Python,
support sending log messages over SMTP.

ACN

On Wed, Jun 23, 2021 at 5:03 AM QEDK <qedk.en@gmail.com> wrote:
>
> I do remember getting emails (last year?) from running cronjobs on the grid until I disabled them, iirc it was when they failed, but not necessarily a description of the error. I literally cannot find any documentation regarding this anymore so I almost have to think if I didn't imagine this entire ordeal.
>
> Best,
> QEDK
>
> On Wed, Jun 23, 2021, 09:01 Sam Wilson <sam@samwilson.id.au> wrote:
>>
>> I think the only way is to have your cronjob do the emailing of the
>> output, and then delete (or move) the output file so it's not repeated
>> on the next run. Something like:
>>
>> #!/bin/bash
>>
>> ## Call actual job script here.
>> ./do-tool-things.sh
>>
>> ## Email output to maintainers.
>> echo -e 'Subject: [tools.toolname] Job output\nContent-Type: text/plain;
>> charset=UTF-8' | cat - /data/project/toolname/cron-tools.jobname.out |
>> /usr/sbin/exim -odf -i tools.toolname@tools.wmflabs.org
>>
>> ## Remove output file.
>> rm /data/project/toolname/cron-tools.jobname.out
>>
>> Although of course the details might differ for different tools.
>>
>>
>> On 22/6/21 4:52 pm, Shrinivasan T wrote:
>> > Hello all,
>> >
>> > I have a bot at toolforge, running weekly once in cron job using "jsub"
>> >
>> > Followed the below link for the setup.
>> > https://wikitech.wikimedia.org/wiki/Help:Toolforge/Pywikibot
>> >
>> > Getting the JOBNAME.out and JOBNAME.err files at the home folder.
>> >
>> > Is there any other way like email notification to get alerts, if some
>> > error happens?
>> >
>> > I think the files JOBNAME.out and JOBNAME.err will be keep on growing
>> > on the bason host login.toolforge.org
>> >
>> > How to avoid this?
>> >
>> > Thanks.
>> >
>> _______________________________________________
>> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
>> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
>> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/