Mailing List Archive

qmail-smtpd segfault
Anyone checked this:

http://www.guninski.com/qmailcrash.html

?

--
Jose Celestino Email: japc@co.sapo.pt

gpg key fingerprint : D3F3 B47B F20C 3B1E 488C B949 1B8B 8141 07B1 363B
Re: qmail-smtpd segfault [ In reply to ]
Thank you for the email,

I have increased the softlimit from 30000000 to 40000000 a few days ago,
it didn't seem to help. Should I go higher yet?


On 03/31/2014 10:20 AM, Erwin Hoffmann wrote:
> Hi,
>
> thanks for the information.
>
> It seems to be a heavy patched qmail system, you are using.
>
> Before going on: Check your "softlimit" settings. Maybe they are to
> small on the new system. The memory footprint of qmail-smtpd is
> usually small, but you load vchkpw, simscan, and the virus scanner in
> addition.
>
> regards.
> --eh.
>
> --On 31. März 2014 09:58:36 -0500 Leon F Wiederholt
> <jaglists@jagtec.net> wrote:
>
>> hopefully here is the information you requested..
>>
>> __________________________________________________________________________
>>
>>
>> # uname -a
>> Linux mail.jagtec.net 2.6.32-431.5.1.el6.i686 #1 SMP Tue Feb 11 21:56:33
>> UTC 2014 i686 i686 i386 GNU/Linux
>> __________________________________________________________________________
>>
>>
>> gcc -v
>> Using built-in specs.
>> Target: i686-redhat-linux
>> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
>> --infodir=/usr/share/info
>> --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap
>> --enable-shared --enable-threads=posix --enable-checking=release
>> --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
>> --enable-gnu-unique-object
>> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
>> --enable-java-awt=gtk --disable-dssi
>> --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
>> --enable-libgcj-multifile --enable-java-maintainer-mode
>> --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
>> --disable-libjava-multilib
>> --with-ppl --with-cloog --with-tune=generic --with-arch=i686
>> --build=i686-redhat-linux
>> Thread model: posix
>> gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
>>
>> _________________________________________________________________________
>>
>> __
>> smtpd/run
>>
>> # !/bin/sh
>> QMAILQUEUE="/var/qmail/bin/simscan"; export QMAILQUEUE
>> QMAILDUID=`id -u vpopmail`
>> NOFILESGID=`id -g vpopmail`
>> MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
>> SOFTLIMIT=`cat /var/qmail/control/softlimit`
>>
>> # This enables greetdelay for qmail-smtpd. Put 0 if you decide to delay
>> # rblsmtpd instead.
>> export SMTPD_GREETDELAY=5
>> export DROP_PRE_GREET=1
>>
>> # This enables greetdelay for rblsmtpd
>> # export GREETDELAY=30
>> # This disables rblsmtpd reject
>> # export RBLSMTPD=""
>>
>> # You MUST export this, otherwise you'd get a 30 sec timeout
>> export SMTPAUTH=""
>> export FORCETLS=0
>>
>> # This enables chkuser
>> export CHKUSER_START=ALWAYS
>>
>> # DKIM - SURBL configuration
>> # DKIMQUEUE and SURBLQUEUE are front-ends of qmail-queue
>> # export SURBL=1 # Comment out to enable SURBL filtering
>> # export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
>> # export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim
>> after
>> # sublfilter export DKIMQUEUE=/var/qmail/bin/simscan # simscan is
>> # executed after qmail-dkim DKIM verification. Use carefully
>> # export DKIMVERIFY="FGHKLMNOQRTVWjp"
>> # This is to avoid verification of outgoing messages
>> # export RELAYCLIENT_NODKIMVERIFY=1
>>
>> # This enables simscan debug
>> # export SIMSCAN_DEBUG=2
>>
>> exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
>> /usr/local/bin/tcpserver -v -H -R -l 0 \
>> -x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
>> -u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
>> /var/qmail/bin/qmail-smtpd \
>> /home/vpopmail/bin/vchkpw /bin/true 2>&1
>> ~
>>
>>
>>
>> On 03/29/2014 10:07 AM, Erwin Hoffmann wrote:
>>> Hi,
>>>
>>> show us
>>>
>>> a) the 'run' script
>>>
>>> b) 'uname -a' and
>>>
>>> c) 'gcc -v' the version of the C-compiler you used.
>>>
>>> In addition, re-compile qmail while keeping the logs/warnings and post
>>> those.
>>>
>>> regards.
>>> --eh.
>>>
>>>
>>> --On 29. März 2014 08:52:01 -0500 Leon F Wiederholt
>>> <jaglists@jagtec.net> wrote:
>>>
>>>> I am getting this kernel error: qmail-smtpd[14321]: segfault at
>>>> 9f37000
>>>> ip 08059428 sp bf919fec error 4 in qmail-smtpd[8048000+2a000]
>>>>
>>>> This machine is a dual processor IBM Netfinity running CentOS 6.5
>>>> 32bit,
>>>> 4 gb ram, IBM serveRAID 5. Qmail is patched to the netqmail 1.06
>>>> level.
>>>> vpopmail, mysql, dovecot, simscan.... I am not running clamd yet
>>>> as it
>>>> is causing a 451 error...
>>>>
>>>> This is a new installation to replace a server that was in a fire that
>>>> destroyed my business. It has about 100 email accounts set up now
>>>> that
>>>> are functioning. Will have a total of about 350 accounts when
>>>> complete.
>>>>
>>>> THANKS in advance!
>>>
>>>
>>>
>>> --
>>> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id:
>>> 7E4034BE
>>
>
>
>
> --
> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id:
> 7E4034BE
Re: qmail-smtpd segfault [ In reply to ]
Hi,

--On 31. März 2014 10:25:37 -0500 Leon F Wiederholt <jaglists@jagtec.net>
wrote:

> Thank you for the email,
>
> I have increased the softlimit from 30000000 to 40000000 a few days ago,
> it didn't seem to help. Should I go higher yet?

Probably yes: From the dump, I can read the 30 MByte size (qmail-smtpd
abends allocating > 30 MByte). 40 MByte seems not be be enough in your case

regards.
--eh.

>
>
> On 03/31/2014 10:20 AM, Erwin Hoffmann wrote:
>> Hi,
>>
>> thanks for the information.
>>
>> It seems to be a heavy patched qmail system, you are using.
>>
>> Before going on: Check your "softlimit" settings. Maybe they are to
>> small on the new system. The memory footprint of qmail-smtpd is
>> usually small, but you load vchkpw, simscan, and the virus scanner in
>> addition.
>>
>> regards.
>> --eh.
>>
>> --On 31. März 2014 09:58:36 -0500 Leon F Wiederholt
>> <jaglists@jagtec.net> wrote:
>>
>>> hopefully here is the information you requested..
>>>
>>> _______________________________________________________________________
>>> ___
>>>
>>>
>>> # uname -a
>>> Linux mail.jagtec.net 2.6.32-431.5.1.el6.i686 #1 SMP Tue Feb 11 21:56:33
>>> UTC 2014 i686 i686 i386 GNU/Linux
>>> _______________________________________________________________________
>>> ___
>>>
>>>
>>> gcc -v
>>> Using built-in specs.
>>> Target: i686-redhat-linux
>>> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
>>> --infodir=/usr/share/info
>>> --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap
>>> --enable-shared --enable-threads=posix --enable-checking=release
>>> --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions
>>> --enable-gnu-unique-object
>>> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
>>> --enable-java-awt=gtk --disable-dssi
>>> --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
>>> --enable-libgcj-multifile --enable-java-maintainer-mode
>>> --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
>>> --disable-libjava-multilib
>>> --with-ppl --with-cloog --with-tune=generic --with-arch=i686
>>> --build=i686-redhat-linux
>>> Thread model: posix
>>> gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
>>>
>>> _______________________________________________________________________
>>> __
>>>
>>> __
>>> smtpd/run
>>>
>>> # !/bin/sh
>>> QMAILQUEUE="/var/qmail/bin/simscan"; export QMAILQUEUE
>>> QMAILDUID=`id -u vpopmail`
>>> NOFILESGID=`id -g vpopmail`
>>> MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
>>> SOFTLIMIT=`cat /var/qmail/control/softlimit`
>>>
>>> # This enables greetdelay for qmail-smtpd. Put 0 if you decide to delay
>>> # rblsmtpd instead.
>>> export SMTPD_GREETDELAY=5
>>> export DROP_PRE_GREET=1
>>>
>>> # This enables greetdelay for rblsmtpd
>>> # export GREETDELAY=30
>>> # This disables rblsmtpd reject
>>> # export RBLSMTPD=""
>>>
>>> # You MUST export this, otherwise you'd get a 30 sec timeout
>>> export SMTPAUTH=""
>>> export FORCETLS=0
>>>
>>> # This enables chkuser
>>> export CHKUSER_START=ALWAYS
>>>
>>> # DKIM - SURBL configuration
>>> # DKIMQUEUE and SURBLQUEUE are front-ends of qmail-queue
>>> # export SURBL=1 # Comment out to enable SURBL filtering
>>> # export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
>>> # export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim
>>> after
>>> # sublfilter export DKIMQUEUE=/var/qmail/bin/simscan # simscan is
>>> # executed after qmail-dkim DKIM verification. Use carefully
>>> # export DKIMVERIFY="FGHKLMNOQRTVWjp"
>>> # This is to avoid verification of outgoing messages
>>> # export RELAYCLIENT_NODKIMVERIFY=1
>>>
>>> # This enables simscan debug
>>> # export SIMSCAN_DEBUG=2
>>>
>>> exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
>>> /usr/local/bin/tcpserver -v -H -R -l 0 \
>>> -x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
>>> -u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
>>> /var/qmail/bin/qmail-smtpd \
>>> /home/vpopmail/bin/vchkpw /bin/true 2>&1
>>> ~
>>>
>>>
>>>
>>> On 03/29/2014 10:07 AM, Erwin Hoffmann wrote:
>>>> Hi,
>>>>
>>>> show us
>>>>
>>>> a) the 'run' script
>>>>
>>>> b) 'uname -a' and
>>>>
>>>> c) 'gcc -v' the version of the C-compiler you used.
>>>>
>>>> In addition, re-compile qmail while keeping the logs/warnings and post
>>>> those.
>>>>
>>>> regards.
>>>> --eh.
>>>>
>>>>
>>>> --On 29. März 2014 08:52:01 -0500 Leon F Wiederholt
>>>> <jaglists@jagtec.net> wrote:
>>>>
>>>>> I am getting this kernel error: qmail-smtpd[14321]: segfault at
>>>>> 9f37000
>>>>> ip 08059428 sp bf919fec error 4 in qmail-smtpd[8048000+2a000]
>>>>>
>>>>> This machine is a dual processor IBM Netfinity running CentOS 6.5
>>>>> 32bit,
>>>>> 4 gb ram, IBM serveRAID 5. Qmail is patched to the netqmail 1.06
>>>>> level.
>>>>> vpopmail, mysql, dovecot, simscan.... I am not running clamd yet
>>>>> as it
>>>>> is causing a 451 error...
>>>>>
>>>>> This is a new installation to replace a server that was in a fire that
>>>>> destroyed my business. It has about 100 email accounts set up now
>>>>> that
>>>>> are functioning. Will have a total of about 350 accounts when
>>>>> complete.
>>>>>
>>>>> THANKS in advance!
>>>>
>>>>
>>>>
>>>> --
>>>> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id:
>>>> 7E4034BE
>>>
>>
>>
>>
>> --
>> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id:
>> 7E4034BE
>



--
Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id: 7E4034BE
Re: qmail-smtpd segfault [ In reply to ]
OK, thanks we will increase the softlimit and post back the results
after some testing..

THANKS, and best regards. leon

On 03/31/2014 10:41 AM, Erwin Hoffmann wrote:
> Hi,
>
> --On 31. März 2014 10:25:37 -0500 Leon F Wiederholt
> <jaglists@jagtec.net> wrote:
>
>> Thank you for the email,
>>
>> I have increased the softlimit from 30000000 to 40000000 a few days ago,
>> it didn't seem to help. Should I go higher yet?
>
> Probably yes: From the dump, I can read the 30 MByte size (qmail-smtpd
> abends allocating > 30 MByte). 40 MByte seems not be be enough in your
> case
>
> regards.
> --eh.
>
>>
>>
>> On 03/31/2014 10:20 AM, Erwin Hoffmann wrote:
>>> Hi,
>>>
>>> thanks for the information.
>>>
>>> It seems to be a heavy patched qmail system, you are using.
>>>
>>> Before going on: Check your "softlimit" settings. Maybe they are to
>>> small on the new system. The memory footprint of qmail-smtpd is
>>> usually small, but you load vchkpw, simscan, and the virus scanner in
>>> addition.
>>>
>>> regards.
>>> --eh.
>>>
>>> --On 31. März 2014 09:58:36 -0500 Leon F Wiederholt
>>> <jaglists@jagtec.net> wrote:
>>>
>>>> hopefully here is the information you requested..
>>>>
>>>> _______________________________________________________________________
>>>>
>>>> ___
>>>>
>>>>
>>>> # uname -a
>>>> Linux mail.jagtec.net 2.6.32-431.5.1.el6.i686 #1 SMP Tue Feb 11
>>>> 21:56:33
>>>> UTC 2014 i686 i686 i386 GNU/Linux
>>>> _______________________________________________________________________
>>>>
>>>> ___
>>>>
>>>>
>>>> gcc -v
>>>> Using built-in specs.
>>>> Target: i686-redhat-linux
>>>> Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
>>>> --infodir=/usr/share/info
>>>> --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap
>>>> --enable-shared --enable-threads=posix --enable-checking=release
>>>> --with-system-zlib --enable-__cxa_atexit
>>>> --disable-libunwind-exceptions
>>>> --enable-gnu-unique-object
>>>> --enable-languages=c,c++,objc,obj-c++,java,fortran,ada
>>>> --enable-java-awt=gtk --disable-dssi
>>>> --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-1.5.0.0/jre
>>>> --enable-libgcj-multifile --enable-java-maintainer-mode
>>>> --with-ecj-jar=/usr/share/java/eclipse-ecj.jar
>>>> --disable-libjava-multilib
>>>> --with-ppl --with-cloog --with-tune=generic --with-arch=i686
>>>> --build=i686-redhat-linux
>>>> Thread model: posix
>>>> gcc version 4.4.7 20120313 (Red Hat 4.4.7-4) (GCC)
>>>>
>>>> _______________________________________________________________________
>>>>
>>>> __
>>>>
>>>> __
>>>> smtpd/run
>>>>
>>>> # !/bin/sh
>>>> QMAILQUEUE="/var/qmail/bin/simscan"; export QMAILQUEUE
>>>> QMAILDUID=`id -u vpopmail`
>>>> NOFILESGID=`id -g vpopmail`
>>>> MAXSMTPD=`cat /var/qmail/control/concurrencyincoming`
>>>> SOFTLIMIT=`cat /var/qmail/control/softlimit`
>>>>
>>>> # This enables greetdelay for qmail-smtpd. Put 0 if you decide to
>>>> delay
>>>> # rblsmtpd instead.
>>>> export SMTPD_GREETDELAY=5
>>>> export DROP_PRE_GREET=1
>>>>
>>>> # This enables greetdelay for rblsmtpd
>>>> # export GREETDELAY=30
>>>> # This disables rblsmtpd reject
>>>> # export RBLSMTPD=""
>>>>
>>>> # You MUST export this, otherwise you'd get a 30 sec timeout
>>>> export SMTPAUTH=""
>>>> export FORCETLS=0
>>>>
>>>> # This enables chkuser
>>>> export CHKUSER_START=ALWAYS
>>>>
>>>> # DKIM - SURBL configuration
>>>> # DKIMQUEUE and SURBLQUEUE are front-ends of qmail-queue
>>>> # export SURBL=1 # Comment out to enable SURBL filtering
>>>> # export QMAILQUEUE=/var/qmail/bin/surblqueue # executes surblfilter
>>>> # export SURBLQUEUE=/var/qmail/bin/qmail-dkim # executes qmail-dkim
>>>> after
>>>> # sublfilter export DKIMQUEUE=/var/qmail/bin/simscan # simscan is
>>>> # executed after qmail-dkim DKIM verification. Use carefully
>>>> # export DKIMVERIFY="FGHKLMNOQRTVWjp"
>>>> # This is to avoid verification of outgoing messages
>>>> # export RELAYCLIENT_NODKIMVERIFY=1
>>>>
>>>> # This enables simscan debug
>>>> # export SIMSCAN_DEBUG=2
>>>>
>>>> exec /usr/local/bin/softlimit -m "$SOFTLIMIT" \
>>>> /usr/local/bin/tcpserver -v -H -R -l 0 \
>>>> -x /home/vpopmail/etc/tcp.smtp.cdb -c "$MAXSMTPD" \
>>>> -u "$QMAILDUID" -g "$NOFILESGID" 0 25 \
>>>> /var/qmail/bin/qmail-smtpd \
>>>> /home/vpopmail/bin/vchkpw /bin/true 2>&1
>>>> ~
>>>>
>>>>
>>>>
>>>> On 03/29/2014 10:07 AM, Erwin Hoffmann wrote:
>>>>> Hi,
>>>>>
>>>>> show us
>>>>>
>>>>> a) the 'run' script
>>>>>
>>>>> b) 'uname -a' and
>>>>>
>>>>> c) 'gcc -v' the version of the C-compiler you used.
>>>>>
>>>>> In addition, re-compile qmail while keeping the logs/warnings and
>>>>> post
>>>>> those.
>>>>>
>>>>> regards.
>>>>> --eh.
>>>>>
>>>>>
>>>>> --On 29. März 2014 08:52:01 -0500 Leon F Wiederholt
>>>>> <jaglists@jagtec.net> wrote:
>>>>>
>>>>>> I am getting this kernel error: qmail-smtpd[14321]: segfault at
>>>>>> 9f37000
>>>>>> ip 08059428 sp bf919fec error 4 in qmail-smtpd[8048000+2a000]
>>>>>>
>>>>>> This machine is a dual processor IBM Netfinity running CentOS 6.5
>>>>>> 32bit,
>>>>>> 4 gb ram, IBM serveRAID 5. Qmail is patched to the netqmail 1.06
>>>>>> level.
>>>>>> vpopmail, mysql, dovecot, simscan.... I am not running clamd yet
>>>>>> as it
>>>>>> is causing a 451 error...
>>>>>>
>>>>>> This is a new installation to replace a server that was in a fire
>>>>>> that
>>>>>> destroyed my business. It has about 100 email accounts set up now
>>>>>> that
>>>>>> are functioning. Will have a total of about 350 accounts when
>>>>>> complete.
>>>>>>
>>>>>> THANKS in advance!
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id:
>>>>> 7E4034BE
>>>>
>>>
>>>
>>>
>>> --
>>> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id:
>>> 7E4034BE
>>
>
>
>
> --
> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id:
> 7E4034BE
Re: qmail-smtpd segfault [ In reply to ]
On Mon, Mar 31, 2014 at 8:46 AM, Leon F Wiederholt <jaglists@jagtec.net> wrote:
> OK, thanks we will increase the softlimit and post back the results after
> some testing..

I'm sorry, perhaps this is too obvious, but - why not set SOFTLIMIT to
be '=' and measure the actual size before restricting it? This will
also tell you if your hard limit is too restrictive.

- M
Re: qmail-smtpd segfault [ In reply to ]
Thanks, not too obvious for me, will give it a try..
leon

On 03/31/2014 11:14 AM, Michael Sierchio wrote:
> On Mon, Mar 31, 2014 at 8:46 AM, Leon F Wiederholt <jaglists@jagtec.net> wrote:
>> OK, thanks we will increase the softlimit and post back the results after
>> some testing..
> I'm sorry, perhaps this is too obvious, but - why not set SOFTLIMIT to
> be '=' and measure the actual size before restricting it? This will
> also tell you if your hard limit is too restrictive.
>
> - M
Re: qmail-smtpd segfault [ In reply to ]
I'm with Michael, increasing SOFTLIMIT to 50000... will solve your issue.


Saludos.

Iván L. Morales - Dpto. comercial.
Logosur de telecomunicación S.L.
ivanlopez@logosur.com <mailto:ivanlopez@logosur.com> - www.LogoSur.com
<http://www.logosur.com>
··············································································

Pol.Ind. Zeus S/N
41703 Dos hermanas
Sevilla
Teléfono 955669108
Móvil 657019769
FAX 954728055


El 31/03/2014 19:09, Leon F Wiederholt escribió:
> Thanks, not too obvious for me, will give it a try..
> leon
>
> On 03/31/2014 11:14 AM, Michael Sierchio wrote:
>> On Mon, Mar 31, 2014 at 8:46 AM, Leon F Wiederholt
>> <jaglists@jagtec.net> wrote:
>>> OK, thanks we will increase the softlimit and post back the results
>>> after
>>> some testing..
>> I'm sorry, perhaps this is too obvious, but - why not set SOFTLIMIT to
>> be '=' and measure the actual size before restricting it? This will
>> also tell you if your hard limit is too restrictive.
>>
>> - M
>
Re: qmail-smtpd segfault [ In reply to ]
We will go with 50.... and let her run for a while and post back results...

THANKS very much, leon

On 03/31/2014 12:12 PM, Ivan L. Morales wrote:
> I'm with Michael, increasing SOFTLIMIT to 50000... will solve your issue.
>
>
> Saludos.
>
> Iván L. Morales - Dpto. comercial.
> Logosur de telecomunicación S.L.
> ivanlopez@logosur.com <mailto:ivanlopez@logosur.com> - www.LogoSur.com
> <http://www.logosur.com>
> ··············································································
>
> Pol.Ind. Zeus S/N
> 41703 Dos hermanas
> Sevilla
> Teléfono 955669108
> Móvil 657019769
> FAX 954728055
>
>
> El 31/03/2014 19:09, Leon F Wiederholt escribió:
>> Thanks, not too obvious for me, will give it a try..
>> leon
>>
>> On 03/31/2014 11:14 AM, Michael Sierchio wrote:
>>> On Mon, Mar 31, 2014 at 8:46 AM, Leon F Wiederholt
>>> <jaglists@jagtec.net> wrote:
>>>> OK, thanks we will increase the softlimit and post back the results
>>>> after
>>>> some testing..
>>> I'm sorry, perhaps this is too obvious, but - why not set SOFTLIMIT to
>>> be '=' and measure the actual size before restricting it? This will
>>> also tell you if your hard limit is too restrictive.
>>>
>>> - M
>>
>
>
Re: qmail-smtpd segfault [ In reply to ]
Greetings

Thanks again for the help on my qmail server....

The mail server is still having about 6 segfaults a day, the mail server
seems to be functioning fine other than the logged segfault occurrences.

We have tried various softlimit settings up to 50mb, and raising the
memory soft limit up to the hard limit (setting "softlimit -m =" in the
run file) didn't help.

When running top, I see 1 or 2 instances of qmail-smtpd appear
periodically for only a few seconds using about 0.3% cpu 0.1% mem each.
The server has 4gb of ram total.

Here is a part of ps axfu that might be helpful. Is it normal to have 7
or more instance of qmail-smtpd here? We have about 200 email accounts
entered now, and will have approximately 350 to 400 email accounts
total... the instances of qmail-smtpd live few seconds each..


svscan /service
root 10968 0.0 0.0 1876 328 ? S Mar28 0:00 | \_
supervise vpopmaild
root 13390 0.0 0.0 1900 324 ? S 09:16 0:00 | |
\_ /usr/local/bin/tcpserver -v -H -R -l 0 -u 0 -g 0 0 89 /ho
root 10969 0.0 0.0 1876 332 ? S Mar28 0:00 | \_
supervise log
qmaill 10977 0.0 0.0 1884 328 ? S Mar28 0:00 | |
\_ /usr/local/bin/multilog t /var/log/qmail/vpopmaild
root 10970 0.0 0.0 1876 328 ? S Mar28 0:00 | \_
supervise qmail-smtpd
vpopmail 13387 0.0 0.0 1900 348 ? S 09:16 0:00 | |
\_ /usr/local/bin/tcpserver -v -H -R -l 0 -x /home/vpopmail/
vpopmail 16246 0.0 0.0 8800 3056 ? S 10:05 0:00 |
| \_ /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw
vpopmail 16378 0.1 0.0 8608 2548 ? S 10:07 0:00 |
| \_ /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw
vpopmail 16438 0.0 0.0 8608 1644 ? S 10:08 0:00 |
| \_ /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw
vpopmail 16448 0.1 0.0 8668 2224 ? S 10:08 0:00 |
| \_ /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw
clamav 16470 0.0 0.0 2248 676 ? S 10:08 0:00 |
| | \_ /var/qmail/bin/simscan
clamav 16473 0.0 0.0 7128 1332 ? S 10:08 0:00 |
| | \_ spamc -u sctc@jagtec.net
vpopmail 16468 0.0 0.0 8608 1656 ? S 10:08 0:00 |
| \_ /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw
vpopmail 16477 0.0 0.0 8608 1528 ? S 10:08 0:00 |
| \_ /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw
vpopmail 16478 0.0 0.0 8608 1524 ? S 10:08 0:00 |
| \_ /var/qmail/bin/qmail-smtpd /home/vpopmail/bin/vchkpw
root 10971 0.0 0.0 1876 332 ? S Mar28 0:00 | \_
supervise log
qmaill 10980 0.0 0.0 2016 396 ? S Mar28 1:53 | |
\_ /usr/local/bin/multilog t /var/log/qmail/smtpd
root 10972 0.0 0.0 1876 332 ? S Mar28 0:00 | \_
supervise qmail-submission


my recent log...
Mar 31 14:53:53 mail kernel: eth0: link up, 100Mbps, full-duplex
Apr 1 00:39:25 mail kernel: qmail-smtpd[26131]: segfault at 993c000 ip
08059449 sp bf86d8ec error 6 in qmail-smtpd[8048000+2a000]
Apr 1 06:29:08 mail kernel: qmail-smtpd[7709]: segfault at 90b3000 ip
08059428 sp bf9d5fec error 4 in qmail-smtpd[8048000+2a000]
Apr 1 06:35:51 mail kernel: qmail-smtpd[7893]: segfault at 8473000 ip
08059428 sp bfefe8fc error 4 in qmail-smtpd[8048000+2a000]
Apr 1 13:45:56 mail kernel: qmail-smtpd[29877]: segfault at 9628000 ip
08059428 sp bfa1acfc error 4 in qmail-smtpd[8048000+2a000]
Apr 1 15:34:53 mail kernel: qmail-smtpd[3537]: segfault at 8e73000 ip
08059428 sp bfc866fc error 4 in qmail-smtpd[8048000+2a000]
Apr 1 16:00:45 mail kernel: qmail-smtpd[4784]: segfault at 82ed000 ip
08059449 sp bfde8e7c error 6 in qmail-smtpd[8048000+2a000]
Apr 1 17:16:59 mail kernel: qmail-smtpd[8501]: segfault at 8990000 ip
08059428 sp bfd9142c error 4 in qmail-smtpd[8048000+2a000]
Apr 1 17:43:15 mail kernel: qmail-smtpd[9978]: segfault at 968a000 ip
08059428 sp bfd831fc error 4 in qmail-smtpd[8048000+2a000]
Apr 2 05:53:26 mail kernel: qmail-smtpd[3742]: segfault at 866e000 ip
08059449 sp bfb0537c error 6 in qmail-smtpd[8048000+2a000]
Apr 2 10:12:19 mail kernel: qmail-smtpd[15984]: segfault at 9451000 ip
08059428 sp bfb936dc error 4 in qmail-smtpd[8048000+2a000]

Thanks so much, leon


On 03/31/2014 01:18 PM, Leon F Wiederholt wrote:
> We will go with 50.... and let her run for a while and post back
> results...
>
> THANKS very much, leon
>
> On 03/31/2014 12:12 PM, Ivan L. Morales wrote:
>> I'm with Michael, increasing SOFTLIMIT to 50000... will solve your
>> issue.
>>
>>
>> Saludos.
>>
>> Iván L. Morales - Dpto. comercial.
>> Logosur de telecomunicación S.L.
>> ivanlopez@logosur.com <mailto:ivanlopez@logosur.com> -
>> www.LogoSur.com <http://www.logosur.com>
>> ··············································································
>>
>> Pol.Ind. Zeus S/N
>> 41703 Dos hermanas
>> Sevilla
>> Teléfono 955669108
>> Móvil 657019769
>> FAX 954728055
>>
>>
>> El 31/03/2014 19:09, Leon F Wiederholt escribió:
>>> Thanks, not too obvious for me, will give it a try..
>>> leon
>>>
>>> On 03/31/2014 11:14 AM, Michael Sierchio wrote:
>>>> On Mon, Mar 31, 2014 at 8:46 AM, Leon F Wiederholt
>>>> <jaglists@jagtec.net> wrote:
>>>>> OK, thanks we will increase the softlimit and post back the
>>>>> results after
>>>>> some testing..
>>>> I'm sorry, perhaps this is too obvious, but - why not set SOFTLIMIT to
>>>> be '=' and measure the actual size before restricting it? This will
>>>> also tell you if your hard limit is too restrictive.
>>>>
>>>> - M
>>>
>>
>>
>
Re: qmail-smtpd segfault [ In reply to ]
Leon F Wiederholt <jaglists@jagtec.net> wrote:
>
> The mail server is still having about 6 segfaults a day, the mail
> server seems to be functioning fine other than the logged segfault
> occurrences.
>
> We have tried various softlimit settings up to 50mb, and raising the
> memory soft limit up to the hard limit (setting "softlimit -m =" in
> the run file) didn't help.
[...]
> my recent log...
> Mar 31 14:53:53 mail kernel: eth0: link up, 100Mbps, full-duplex
> Apr 1 00:39:25 mail kernel: qmail-smtpd[26131]: segfault at 993c000
> ip 08059449 sp bf86d8ec error 6 in qmail-smtpd[8048000+2a000]
> Apr 1 06:29:08 mail kernel: qmail-smtpd[7709]: segfault at 90b3000
> ip 08059428 sp bf9d5fec error 4 in qmail-smtpd[8048000+2a000]
> Apr 1 06:35:51 mail kernel: qmail-smtpd[7893]: segfault at 8473000
> ip 08059428 sp bfefe8fc error 4 in qmail-smtpd[8048000+2a000]
> Apr 1 13:45:56 mail kernel: qmail-smtpd[29877]: segfault at 9628000
> ip 08059428 sp bfa1acfc error 4 in qmail-smtpd[8048000+2a000]
[...]

It sounds like you have a genuine problem here -- possibly hardware, possibly
software. To eliminate software (bad compiler or libc? bad patches on top of
qmail proper?), I think you'll need to configure your system to generate core
dumps when this happens, and examine the dumps with a debugger. If they're
all similar or identical stack traces, it's probably a software problem. If
they're all different and random, that would point to hardware issues, I
think.

You may want to consider running hardware diagnostics at that point to find
problems with heat, flaky memory, flaky power, etc.

> When running top, I see 1 or 2 instances of qmail-smtpd appear
> periodically for only a few seconds using about 0.3% cpu 0.1% mem
> each. The server has 4gb of ram total.
>
> Here is a part of ps axfu that might be helpful. Is it normal to
> have 7 or more instance of qmail-smtpd here?

This is perfectly normal for qmail, but you really should already know that
since you're administering the system...

Have you read the qmail documentation, Life with qmail, the qmail big picture,
the included and online FAQs, and other common documents about this software?
If not, you should.

Charles
--
--------------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at: http://pyropus.ca/software/
Read http://pyropus.ca/personal/writings/12-steps-to-qmail-list-bliss.html
--------------------------------------------------------------------------
Re: qmail-smtpd segfault [ In reply to ]
Charles

Thanks for the email.

Please find my reply below...


On 04/04/2014 10:19 PM, Charles Cazabon wrote:
> Leon F Wiederholt <jaglists@jagtec.net> wrote:
>>
>> The mail server is still having about 6 segfaults a day, the mail
>> server seems to be functioning fine other than the logged segfault
>> occurrences.
>>
>> We have tried various softlimit settings up to 50mb, and raising the
>> memory soft limit up to the hard limit (setting "softlimit -m =" in
>> the run file) didn't help.
> [...]
>> my recent log...
>> Mar 31 14:53:53 mail kernel: eth0: link up, 100Mbps, full-duplex
>> Apr 1 00:39:25 mail kernel: qmail-smtpd[26131]: segfault at 993c000
>> ip 08059449 sp bf86d8ec error 6 in qmail-smtpd[8048000+2a000]
>> Apr 1 06:29:08 mail kernel: qmail-smtpd[7709]: segfault at 90b3000
>> ip 08059428 sp bf9d5fec error 4 in qmail-smtpd[8048000+2a000]
>> Apr 1 06:35:51 mail kernel: qmail-smtpd[7893]: segfault at 8473000
>> ip 08059428 sp bfefe8fc error 4 in qmail-smtpd[8048000+2a000]
>> Apr 1 13:45:56 mail kernel: qmail-smtpd[29877]: segfault at 9628000
>> ip 08059428 sp bfa1acfc error 4 in qmail-smtpd[8048000+2a000]
> [...]
>
> It sounds like you have a genuine problem here -- possibly hardware, possibly
> software. To eliminate software (bad compiler or libc? bad patches on top of
> qmail proper?), I think you'll need to configure your system to generate core
> dumps when this happens, and examine the dumps with a debugger. If they're
> all similar or identical stack traces, it's probably a software problem. If
> they're all different and random, that would point to hardware issues, I
> think.

I will have to read up on doing a core dump, that will be new work I
don't have experience with.
>
> You may want to consider running hardware diagnostics at that point to find
> problems with heat, flaky memory, flaky power, etc.

I ran extensive tests before Linux was installed last month. No errors
appeared until qmail-smtpd segfault. I will test hardware again. IBM
has decent server lightpath and diagnostics.

>
>> When running top, I see 1 or 2 instances of qmail-smtpd appear
>> periodically for only a few seconds using about 0.3% cpu 0.1% mem
>> each. The server has 4gb of ram total.
>>
>> Here is a part of ps axfu that might be helpful. Is it normal to
>> have 7 or more instance of qmail-smtpd here?
>
> This is perfectly normal for qmail, but you really should already know that
> since you're administering the system...

I pretty thought I knew that, but the old saying is.. "you don't know
what you don't know" ;) I don't claim to be a good sysadmin, as I am
just a small business owner (ISP) My old servers were lost in a fire
that took a lot away from me last month.. :( The old qmail server ran
10 years without needing much from me at all.

>
> Have you read the qmail documentation, Life with qmail, the qmail big picture,
> the included and online FAQs, and other common documents about this software?
> If not, you should.

I purchased the Qmail Handbood by Dave Sill years ago. And have notes
written on many of the pages. I have looked all over for qmail-smtpd
segfault information and have found very little.


THANKS again very much... leon

>
> Charles
>
Re: qmail-smtpd segfault [ In reply to ]
He Leon,


I understand your frustration. Once I had a similar problem with a broken
gcc compiler (apart from bugs I occassionaly include in my own SW) ...

However, only a systematic approach will eventually succeed:

a) Since you have adjusted the systems' softlimits, this can be ruled out
(one last try: simply remove softlimit settings).

b) From your observation, qmail-smtpd only abends OCCASIONALLY. This might
be because it enters (internally) a bad piece of code (due to some patches)
- or because an external prog is called with the same problem. In oder to
check for the last one, simply disable all plugins for a while and observe
the system. Check in addition the state of your patches (from the provider;
I don't know if you depend on RPMs or so).

c) In case this does not help, I simply recommand to run basic qmail-smtpd
(without patches); or - alternativly use other ones - perhaps my
Spamcontrol patch, if required.

d) There is of course the chance that either the gcc is broken, or you have
some bad memory or disk. For the first part, please us some search engine
to check for complaints, for the rest, you probably have to rely on some
maintanence.

The last resort would be to ask for paid consultancy.

Good luck.
--eh.


--On 5. April 2014 11:30:05 -0500 Leon F Wiederholt <jaglists@jagtec.net>
wrote:

> Charles
>
> Thanks for the email.
>
> Please find my reply below...
>
>
> On 04/04/2014 10:19 PM, Charles Cazabon wrote:
>> Leon F Wiederholt <jaglists@jagtec.net> wrote:
>>>
>>> The mail server is still having about 6 segfaults a day, the mail
>>> server seems to be functioning fine other than the logged segfault
>>> occurrences.
>>>
>>> We have tried various softlimit settings up to 50mb, and raising the
>>> memory soft limit up to the hard limit (setting "softlimit -m =" in
>>> the run file) didn't help.
>> [...]
>>> my recent log...
>>> Mar 31 14:53:53 mail kernel: eth0: link up, 100Mbps, full-duplex
>>> Apr 1 00:39:25 mail kernel: qmail-smtpd[26131]: segfault at 993c000
>>> ip 08059449 sp bf86d8ec error 6 in qmail-smtpd[8048000+2a000]
>>> Apr 1 06:29:08 mail kernel: qmail-smtpd[7709]: segfault at 90b3000
>>> ip 08059428 sp bf9d5fec error 4 in qmail-smtpd[8048000+2a000]
>>> Apr 1 06:35:51 mail kernel: qmail-smtpd[7893]: segfault at 8473000
>>> ip 08059428 sp bfefe8fc error 4 in qmail-smtpd[8048000+2a000]
>>> Apr 1 13:45:56 mail kernel: qmail-smtpd[29877]: segfault at 9628000
>>> ip 08059428 sp bfa1acfc error 4 in qmail-smtpd[8048000+2a000]
>> [...]
>>
>> It sounds like you have a genuine problem here -- possibly hardware,
>> possibly software. To eliminate software (bad compiler or libc? bad
>> patches on top of qmail proper?), I think you'll need to configure your
>> system to generate core dumps when this happens, and examine the dumps
>> with a debugger. If they're all similar or identical stack traces, it's
>> probably a software problem. If they're all different and random, that
>> would point to hardware issues, I think.
>
> I will have to read up on doing a core dump, that will be new work I
> don't have experience with.
>>
>> You may want to consider running hardware diagnostics at that point to
>> find problems with heat, flaky memory, flaky power, etc.
>
> I ran extensive tests before Linux was installed last month. No errors
> appeared until qmail-smtpd segfault. I will test hardware again. IBM has
> decent server lightpath and diagnostics.
>
>>
>>> When running top, I see 1 or 2 instances of qmail-smtpd appear
>>> periodically for only a few seconds using about 0.3% cpu 0.1% mem
>>> each. The server has 4gb of ram total.
>>>
>>> Here is a part of ps axfu that might be helpful. Is it normal to
>>> have 7 or more instance of qmail-smtpd here?
>>
>> This is perfectly normal for qmail, but you really should already know
>> that since you're administering the system...
>
> I pretty thought I knew that, but the old saying is.. "you don't know
> what you don't know" ;) I don't claim to be a good sysadmin, as I am
> just a small business owner (ISP) My old servers were lost in a fire that
> took a lot away from me last month.. :( The old qmail server ran 10
> years without needing much from me at all.
>
>>
>> Have you read the qmail documentation, Life with qmail, the qmail big
>> picture, the included and online FAQs, and other common documents about
>> this software? If not, you should.
>
> I purchased the Qmail Handbood by Dave Sill years ago. And have notes
> written on many of the pages. I have looked all over for qmail-smtpd
> segfault information and have found very little.
>
>
> THANKS again very much... leon
>
>>
>> Charles
>>



--
Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id: 7E4034BE
Re: qmail-smtpd segfault [ In reply to ]
Thus said Leon F Wiederholt on Sat, 05 Apr 2014 11:30:05 -0500:

> I will have to read up on doing a core dump, that will be new work I
> don't have experience with.

More than likely it is due to one of the various patches you may have
applied to qmail.

Andy
--
TAI64 timestamp: 4000000053404c28
Re: qmail-smtpd segfault [ In reply to ]
Dear qmail list..

Thanks for all the good help and advice on my segfault problem. It will
take a while to do the testing that needs to be done, but I will post
back with what I find. Hopefully I will have good news, and maybe
someday can help someone else on this list..

The segfaults are happening less often now, about 3 in 24 hours. A week
or so ago there would be 8 to 10 occurrences logged. I cannot find that
any negative effects from this other than the logged event, but I know
it needs to be fixed.

Best Regards, leon

On 04/05/2014 12:23 PM, Erwin Hoffmann wrote:
> He Leon,
>
>
> I understand your frustration. Once I had a similar problem with a
> broken gcc compiler (apart from bugs I occassionaly include in my own
> SW) ...
>
> However, only a systematic approach will eventually succeed:
>
> a) Since you have adjusted the systems' softlimits, this can be ruled
> out (one last try: simply remove softlimit settings).
>
> b) From your observation, qmail-smtpd only abends OCCASIONALLY. This
> might be because it enters (internally) a bad piece of code (due to some
> patches) - or because an external prog is called with the same problem.
> In oder to check for the last one, simply disable all plugins for a
> while and observe the system. Check in addition the state of your
> patches (from the provider; I don't know if you depend on RPMs or so).
>
> c) In case this does not help, I simply recommand to run basic
> qmail-smtpd (without patches); or - alternativly use other ones -
> perhaps my Spamcontrol patch, if required.
>
> d) There is of course the chance that either the gcc is broken, or you
> have some bad memory or disk. For the first part, please us some search
> engine to check for complaints, for the rest, you probably have to rely
> on some maintanence.
>
> The last resort would be to ask for paid consultancy.
>
> Good luck.
> --eh.
>
>
> --On 5. April 2014 11:30:05 -0500 Leon F Wiederholt
> <jaglists@jagtec.net> wrote:
>
>> Charles
>>
>> Thanks for the email.
>>
>> Please find my reply below...
>>
>>
>> On 04/04/2014 10:19 PM, Charles Cazabon wrote:
>>> Leon F Wiederholt <jaglists@jagtec.net> wrote:
>>>>
>>>> The mail server is still having about 6 segfaults a day, the mail
>>>> server seems to be functioning fine other than the logged segfault
>>>> occurrences.
>>>>
>>>> We have tried various softlimit settings up to 50mb, and raising the
>>>> memory soft limit up to the hard limit (setting "softlimit -m =" in
>>>> the run file) didn't help.
>>> [...]
>>>> my recent log...
>>>> Mar 31 14:53:53 mail kernel: eth0: link up, 100Mbps, full-duplex
>>>> Apr 1 00:39:25 mail kernel: qmail-smtpd[26131]: segfault at 993c000
>>>> ip 08059449 sp bf86d8ec error 6 in qmail-smtpd[8048000+2a000]
>>>> Apr 1 06:29:08 mail kernel: qmail-smtpd[7709]: segfault at 90b3000
>>>> ip 08059428 sp bf9d5fec error 4 in qmail-smtpd[8048000+2a000]
>>>> Apr 1 06:35:51 mail kernel: qmail-smtpd[7893]: segfault at 8473000
>>>> ip 08059428 sp bfefe8fc error 4 in qmail-smtpd[8048000+2a000]
>>>> Apr 1 13:45:56 mail kernel: qmail-smtpd[29877]: segfault at 9628000
>>>> ip 08059428 sp bfa1acfc error 4 in qmail-smtpd[8048000+2a000]
>>> [...]
>>>
>>> It sounds like you have a genuine problem here -- possibly hardware,
>>> possibly software. To eliminate software (bad compiler or libc? bad
>>> patches on top of qmail proper?), I think you'll need to configure your
>>> system to generate core dumps when this happens, and examine the dumps
>>> with a debugger. If they're all similar or identical stack traces, it's
>>> probably a software problem. If they're all different and random, that
>>> would point to hardware issues, I think.
>>
>> I will have to read up on doing a core dump, that will be new work I
>> don't have experience with.
>>>
>>> You may want to consider running hardware diagnostics at that point to
>>> find problems with heat, flaky memory, flaky power, etc.
>>
>> I ran extensive tests before Linux was installed last month. No errors
>> appeared until qmail-smtpd segfault. I will test hardware again. IBM has
>> decent server lightpath and diagnostics.
>>
>>>
>>>> When running top, I see 1 or 2 instances of qmail-smtpd appear
>>>> periodically for only a few seconds using about 0.3% cpu 0.1% mem
>>>> each. The server has 4gb of ram total.
>>>>
>>>> Here is a part of ps axfu that might be helpful. Is it normal to
>>>> have 7 or more instance of qmail-smtpd here?
>>>
>>> This is perfectly normal for qmail, but you really should already know
>>> that since you're administering the system...
>>
>> I pretty thought I knew that, but the old saying is.. "you don't know
>> what you don't know" ;) I don't claim to be a good sysadmin, as I am
>> just a small business owner (ISP) My old servers were lost in a fire that
>> took a lot away from me last month.. :( The old qmail server ran 10
>> years without needing much from me at all.
>>
>>>
>>> Have you read the qmail documentation, Life with qmail, the qmail big
>>> picture, the included and online FAQs, and other common documents about
>>> this software? If not, you should.
>>
>> I purchased the Qmail Handbood by Dave Sill years ago. And have notes
>> written on many of the pages. I have looked all over for qmail-smtpd
>> segfault information and have found very little.
>>
>>
>> THANKS again very much... leon
>>
>>>
>>> Charles
>>>
>
>
>
> --
> Dr. Erwin Hoffmann | FEHCom | http://www.fehcom.de/ | PGP-Key-Id: 7E4034BE
Re: qmail-smtpd segfault [ In reply to ]
Thus said Leon F Wiederholt on Mon, 07 Apr 2014 15:20:46 -0500:

> The segfaults are happening less often now, about 3 in 24 hours. A
> week or so ago there would be 8 to 10 occurrences logged. I cannot
> find that any negative effects from this other than the logged event,
> but I know it needs to be fixed.

That could be an indication that these were messages queued up on remote
hosts while your mail server was down. They started flushing their
queues once you recovered your mail server, but given that each delivery
attempt they make crashes, they will eventually time out of the remote
queue and fail to be delivered (with a potential bounce to the sender).

Andy
--
TAI64 timestamp: 400000005343589b