Mailing List Archive

mysql inserts and message queues
I have enabled mysql and disk message queues as per below:

$ModLoad immark.so # provides --MARK-- message capability
$ModLoad imuxsock.so # provides support for local system logging (e.g.
via logger command)
$ModLoad imklog.so # kernel logging (formerly provided by rklogd)
$ModLoad ommysql.so
$ModLoad imudp.so # provides UDP syslog reception
$UDPServerRun 514 # start a UDP syslog server at standard port 514

$WorkDirectory /var/rsyslog

$ActionQueueType LinkedList
$ActionQueueFileName dbq
$ActionResumeRetryCount -1

*.* :ommysql:127.0.0.1,Syslog,syslogwriter,writeme
# *.* /var/log/messages


The question I have is... does the disk queue drain w/o me having to
intervene? I currently have 323 queue files (I was adding some keys and
playing with tuning mysql... kind of slowed it down) and the number of
queue files is not going down.

thx
mysql inserts and message queues [ In reply to ]
I don't see any obvious error and, yes, the queue files should be
deleted. Actually, you should only see any files at all if the inserts
take too long. What you have may be an artifact of a previous failure.

To get down to what it is, we need to check how things progress. Is your
database populated? In any case, a debug log (rsyslogd with -dn
additional options run interactively) will tell us what is there. Also,
there should be a .qi file. Let us know its contents.

Thanks,
Rainer

> -----Original Message-----
> From: rsyslog-bounces at lists.adiscon.com [mailto:rsyslog-
> bounces at lists.adiscon.com] On Behalf Of Rory Toma
> Sent: Wednesday, July 30, 2008 3:15 AM
> To: rsyslog-users
> Subject: [rsyslog] mysql inserts and message queues
>
> I have enabled mysql and disk message queues as per below:
>
> $ModLoad immark.so # provides --MARK-- message capability
> $ModLoad imuxsock.so # provides support for local system logging (e.g.
> via logger command)
> $ModLoad imklog.so # kernel logging (formerly provided by rklogd)
> $ModLoad ommysql.so
> $ModLoad imudp.so # provides UDP syslog reception
> $UDPServerRun 514 # start a UDP syslog server at standard port 514
>
> $WorkDirectory /var/rsyslog
>
> $ActionQueueType LinkedList
> $ActionQueueFileName dbq
> $ActionResumeRetryCount -1
>
> *.* :ommysql:127.0.0.1,Syslog,syslogwriter,writeme
> # *.* /var/log/messages
>
>
> The question I have is... does the disk queue drain w/o me having to
> intervene? I currently have 323 queue files (I was adding some keys
and
> playing with tuning mysql... kind of slowed it down) and the number of
> queue files is not going down.
>
> thx
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
mysql inserts and message queues [ In reply to ]
Rainer Gerhards wrote:
> I don't see any obvious error and, yes, the queue files should be
> deleted. Actually, you should only see any files at all if the inserts
> take too long. What you have may be an artifact of a previous failure.
>
> To get down to what it is, we need to check how things progress. Is your
> database populated? In any case, a debug log (rsyslogd with -dn
> additional options run interactively) will tell us what is there. Also,
> there should be a .qi file. Let us know its contents.
>
> Thanks,
> Rainer
>
>
the db is getting populated and there is no .qi file. If I run in with
"-dn" there is a ton of output, so I need to know what to look for. I've
been running for about a day, and I'm at 20+ million records.

I do get an error -2040 when accessing on-disk files. Should I just nuke
these?
mysql inserts and message queues [ In reply to ]
As a side note, I stopped mysql, saw the number of queue files increase,
restarted mysql and they went back down to the original 300 or so.

So, maybe I just have some horked queue files.


Rory Toma wrote:
> Rainer Gerhards wrote:
>
>> I don't see any obvious error and, yes, the queue files should be
>> deleted. Actually, you should only see any files at all if the inserts
>> take too long. What you have may be an artifact of a previous failure.
>>
>> To get down to what it is, we need to check how things progress. Is your
>> database populated? In any case, a debug log (rsyslogd with -dn
>> additional options run interactively) will tell us what is there. Also,
>> there should be a .qi file. Let us know its contents.
>>
>> Thanks,
>> Rainer
>>
>>
>>
> the db is getting populated and there is no .qi file. If I run in with
> "-dn" there is a ton of output, so I need to know what to look for. I've
> been running for about a day, and I'm at 20+ million records.
>
> I do get an error -2040 when accessing on-disk files. Should I just nuke
> these?
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
>
mysql inserts and message queues [ In reply to ]
On Tue, 2008-07-29 at 22:58 -0700, Rory Toma wrote:
> Rainer Gerhards wrote:
> > I don't see any obvious error and, yes, the queue files should be
> > deleted. Actually, you should only see any files at all if the inserts
> > take too long. What you have may be an artifact of a previous failure.
> >
> > To get down to what it is, we need to check how things progress. Is your
> > database populated? In any case, a debug log (rsyslogd with -dn
> > additional options run interactively) will tell us what is there. Also,
> > there should be a .qi file. Let us know its contents.
> >
> > Thanks,
> > Rainer
> >
> >
> the db is getting populated and there is no .qi file.

No .qi file is a good indication (though not sufficient) that the queue
files are artifacts of a previous failure.

> If I run in with
> "-dn" there is a ton of output, so I need to know what to look for. I've
> been running for about a day, and I'm at 20+ million records.

I usually know only when I see. Based on this case, I'd be interested in
the first 2000 lines and another 1000 lines while it is processing
records. That should at least provide enough of a clue to ask for
something more specific.

>
> I do get an error -2040 when accessing on-disk files. Should I just nuke
> these?

This can be OK. I think you see the -2040 (file not found) when it is
looking for the .qi files.

Rainer
>
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
mysql inserts and message queues [ In reply to ]
On Tue, 2008-07-29 at 23:07 -0700, Rory Toma wrote:
> As a side note, I stopped mysql, saw the number of queue files increase,
> restarted mysql and they went back down to the original 300 or so.
>
> So, maybe I just have some horked queue files.

that's now even more probable. But let's verify with the debug log.

Rainer
>
>
> Rory Toma wrote:
> > Rainer Gerhards wrote:
> >
> >> I don't see any obvious error and, yes, the queue files should be
> >> deleted. Actually, you should only see any files at all if the inserts
> >> take too long. What you have may be an artifact of a previous failure.
> >>
> >> To get down to what it is, we need to check how things progress. Is your
> >> database populated? In any case, a debug log (rsyslogd with -dn
> >> additional options run interactively) will tell us what is there. Also,
> >> there should be a .qi file. Let us know its contents.
> >>
> >> Thanks,
> >> Rainer
> >>
> >>
> >>
> > the db is getting populated and there is no .qi file. If I run in with
> > "-dn" there is a ton of output, so I need to know what to look for. I've
> > been running for about a day, and I'm at 20+ million records.
> >
> > I do get an error -2040 when accessing on-disk files. Should I just nuke
> > these?
> >
> >
> > _______________________________________________
> > rsyslog mailing list
> > http://lists.adiscon.net/mailman/listinfo/rsyslog
> >
>
> _______________________________________________
> rsyslog mailing list
> http://lists.adiscon.net/mailman/listinfo/rsyslog
mysql inserts and message queues [ In reply to ]
Rainer Gerhards wrote:
> On Tue, 2008-07-29 at 23:07 -0700, Rory Toma wrote:
>
>> As a side note, I stopped mysql, saw the number of queue files increase,
>> restarted mysql and they went back down to the original 300 or so.
>>
>> So, maybe I just have some horked queue files.
>>
>
> that's now even more probable. But let's verify with the debug log.
>
>
OK, here's the log file, about 20-30 seconds of runtime...

http://www.colinburns.com/rsyslog/rsyslog.txt.gz

thx