Mailing List Archive

multilog not reliable on CentOs-6 x86_64
Hi there

I've noticed that the standard "log/run" logging subprocess doesn't seem
that reliable since moving to CentOs-6 64bit from previous CentOs 32bit
environments. It is always fine after a reboot, but if we ever need to
"svc -t" qmail/dnscache/tinydns, it is fairly likely it will break
logging - which basically immediately breaks the application as it
rapidly blocks once the logging stops flushing out

Running a strace against the parent (eg qmail-start) shows it's sitting
on "starting..." - demonstrating that it has sent that to stdout, but
the multilog process hasn't picked it up. Disabling the service and
running it direct from a terminal shows it all works fine - the logging
goes to the screen and everyone is happy. I have had to hack it to stop
using the "log/run" system - eg instead of

#qmail/run
exec qmail-start "$aliasempty"
#qmail/log/run
exec setuidgid qmaillog splogger qmail

I have

#qmail/run
exec qmail-start "$aliasempty" 2>&1|setuidgid qmaillog splogger qmail

and that works fine. So I've just replaced one way of doing pipes with
another - they should both work reliably and yet they don't?

Has anyone else run into this? Any ideas how to fix this?

Thanks

PS: I've also seen something similar with some perl scripts I've written
that via "$|=1" used to write to stdout in a similar way to "tail -f".
The script on 32bit CentOS worked fine, but on 64bit CentOS6 blocks and
the data comes through in chunks instead of line at a time. This seems
to imply there has been a fundamental change in which the 32bit and
64bit systems are flushing? May be a red herring, but maybe not

--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: multilog not reliable on CentOs-6 x86_64 [ In reply to ]
On 09/03/14 09:42, Paul Jarc wrote:
> Did your original qmail/run script not have "2>&1", or did you just
> not include that line? If it wasn't in the original script, that
> suggests the problem has to do with writing to stderr, rather than the
> logging pipe.

On my older 32bit systems where everything works, the /service/qmail/run
does not contain "2>&1". I just put that into my hack simply because I
always do that :-)

So I guess the issue is that on the 64bit system, that now matters? That
doesn't make any sense. However, it's not 32bit vs 64bit, it's probably
really CentOS4 and CentOS5 vs CentOS6 - I could very well see the same
issue with CentOS6 32bit (I have no such systems so cannot answer for sure)

> Are you using svscanboot? Is readproctitle still running? paul

I have a /etc/init/svscan file ending in "exec /usr/bin/svscanboot", but
ps shows


root 2472 0.0 0.0 3920 368 ? S Mar08 0:06
readproctitle service errors: ...mporary failure?supervise: fatal:
unable to acquire qmail/supervise/lock: temporary failure?supervise:
fatal: unable to acquire smtpd/supervise/lock: temporary
failure?supervise: fatal: unable to acquire log/supervise/lock:
temporary failure?supervise: fatal: unable to acquire
qread/supervise/lock: temporary failure?supervise: fatal: unable to
acquire dnscache/supervise/lock: temporary failure?

I don't see any readproctitle processes on the old 32bit systems - so
they must be running and exiting cleanly whereas they are hanging around
on the CentOS6 systems?

--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1
Re: multilog not reliable on CentOs-6 x86_64 [ In reply to ]
Jason Haar <Jason_Haar@trimble.com> wrote:
>
> I've noticed that the standard "log/run" logging subprocess doesn't seem
> that reliable since moving to CentOs-6 64bit from previous CentOs 32bit
> environments.

Interesting.

> Has anyone else run into this?

I've been running daemontools (lots of different services, not just qmail) on
64-bit Linux boxen of various distributions and many different kernel versions
and have never encountered the problem you describe. But it does sound like a
deep problem -- libc or kernel.

But one I haven't tried recently in CentOS. I wonder if it's something odd in
the RHEL/CentOS kernels? RHEL patches/backports the heck out of an ancient
kernel, doesn't it?

Do you see the same behaviour if you run a mainline kernel without patches?

> PS: I've also seen something similar with some perl scripts I've written
> that via "$|=1" used to write to stdout in a similar way to "tail -f".
> The script on 32bit CentOS worked fine, but on 64bit CentOS6 blocks and
> the data comes through in chunks instead of line at a time. This seems
> to imply there has been a fundamental change in which the 32bit and
> 64bit systems are flushing?

Perhaps, but if so, it sounds like it would specific to your environment. It
certainly doesn't do that in any 64-bit environment I've seen.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: multilog not reliable on CentOs-6 x86_64 [ In reply to ]
On 09/03/14 18:45, Paul Jarc wrote:
> readproctitle is designed to run forever to read any errors output
> from svscan, supervise, and multilog. But if your run script doesn't
> do 2>&1, then that service's error messages will also go to
> readproctitle. And if readproctitle stops reading its input, then
> other processes will still be able to write to the pipe until it fills
> up, at which point anything writing to it will block.

Yup. That did the trick! Thank you very much. We use the same src.rpm on
both the old and new CentOS systems - so I have no idea why it worked on
the old systems.

I'll change it on all systems :-)

Thanks again

--
Cheers

Jason Haar
Information Security Manager, Trimble Navigation Ltd.
Phone: +1 408 481 8171
PGP Fingerprint: 7A2E 0407 C9A6 CAF6 2B9F 8422 C063 5EBB FE1D 66D1