Mailing List Archive: Openrc service crash

Openrc service crash

confabulate at kintzios

Apr 7, 2023, 5:32 AM

Post #1 of 10 (252 views)

I noticed 'www-servers/gatling' now crashes at boot. When I check with
netstat there is no listening port for connections. rc-service reports
gatling has crashed, there is a PID for it, but it is not listed under ps:

# rc-service gatling status
* status: crashed
# cat /run/gatling.pid
4021
# ps axfu | grep gatling
root 4165 0.0 0.0 6572 2224 pts/1 S+ 13:13 0:00
\_ grep --colour=auto gatling

If I try to start it I get a service already started complain. All I see in
the logs is:

* ip link set dev enp2s0 up
* Configuring enp2s0 for MAC address 00:AA:BB:CC:DD:FF ...
[ ok ]
* Starting ifplugd on enp2s0 ...
* start-stop-daemon: fopen `/run/ifplugd.enp2s0.pid': No such file or
directory
* Detaching to start `/usr/sbin/ifplugd' ...
[ ok ]
* Backgrounding ...
* WARNING: net.enp2s0 has started, but is inactive
* WARNING: gatling will start when net.enp2s0 has started

I think this problem started when I changed this PC's LAN network and it now
has IPv6 as well as IPv4, but this could be a coincidence. The init.d script
contains net as a startup dependency:

depend() {
need net
}

Could it be negotiating IP addresses now takes too long and gatling fails to
initialize and crashes for this reason? It is configured to only listen to an
IPv4 address, so I assume the addition of IPv6 should not really affect it.

Restarting it manually works and the service stays up, until the next reboot.

Any ideas what might be causing this?

Re: Openrc service crash [ In reply to ]

confabulate at kintzios

Apr 8, 2023, 1:00 AM

Post #2 of 10 (252 views)

On Friday, 7 April 2023 13:32:52 BST Michael wrote:
> I noticed 'www-servers/gatling' now crashes at boot. When I check with
> netstat there is no listening port for connections. rc-service reports
> gatling has crashed, there is a PID for it, but it is not listed under ps:
>
> # rc-service gatling status
> * status: crashed
> # cat /run/gatling.pid
> 4021
> # ps axfu | grep gatling
> root 4165 0.0 0.0 6572 2224 pts/1 S+ 13:13 0:00
> \_ grep --colour=auto gatling
>
>
> If I try to start it I get a service already started complain. All I see in
> the logs is:
>
> * ip link set dev enp2s0 up
> * Configuring enp2s0 for MAC address 00:AA:BB:CC:DD:FF ...
> [ ok ]
> * Starting ifplugd on enp2s0 ...
> * start-stop-daemon: fopen `/run/ifplugd.enp2s0.pid': No such file or
> directory
> * Detaching to start `/usr/sbin/ifplugd' ...
> [ ok ]
> * Backgrounding ...
> * WARNING: net.enp2s0 has started, but is inactive
> * WARNING: gatling will start when net.enp2s0 has started
>
>
> I think this problem started when I changed this PC's LAN network and it now
> has IPv6 as well as IPv4, but this could be a coincidence. The init.d
> script contains net as a startup dependency:
>
> depend() {
> need net
> }
>
> Could it be negotiating IP addresses now takes too long and gatling fails to
> initialize and crashes for this reason? It is configured to only listen to
> an IPv4 address, so I assume the addition of IPv6 should not really affect
> it.
>
> Restarting it manually works and the service stays up, until the next
> reboot.
>
> Any ideas what might be causing this?

It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
before the gatling init.d exec command and it now starts normally.

Re: Openrc service crash [ In reply to ]

arve.barsnes at gmail

Apr 8, 2023, 1:28 AM

Post #3 of 10 (252 views)

On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
> before the gatling init.d exec command and it now starts normally.

If it was indeed the network that was missing, you could try changing
the service from "need" to "after" starting the net service.

Add to /etc/rc.conf:
rc_gatling_after="net"

Regards,
Arve

Re: Openrc service crash [ In reply to ]

confabulate at kintzios

Apr 8, 2023, 1:32 AM

Post #4 of 10 (252 views)

On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
> On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> > It was probably the addition of IPv6 - as a workaround I added 'sleep 5s'
> > before the gatling init.d exec command and it now starts normally.
>
> If it was indeed the network that was missing, you could try changing
> the service from "need" to "after" starting the net service.
>
> Add to /etc/rc.conf:
> rc_gatling_after="net"
>
> Regards,
> Arve

Thanks Arve, I will try it. Should this replace the "need net" string here,
or added somewhere else?

depend() {
need net
}

Re: Openrc service crash [ In reply to ]

confabulate at kintzios

Apr 8, 2023, 1:33 AM

Post #5 of 10 (252 views)

On Saturday, 8 April 2023 09:32:31 BST Michael wrote:
> On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
> > On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> > > It was probably the addition of IPv6 - as a workaround I added 'sleep
> > > 5s'
> > > before the gatling init.d exec command and it now starts normally.
> >
> > If it was indeed the network that was missing, you could try changing
> > the service from "need" to "after" starting the net service.
> >
> > Add to /etc/rc.conf:
> > rc_gatling_after="net"
> >
> > Regards,
> > Arve
>
> Thanks Arve, I will try it. Should this replace the "need net" string here,
> or added somewhere else?
>
> depend() {
> need net
> }

Oops! Please ignore my question ... I didn't have enough coffee yet. :-)

Re: Openrc service crash [ In reply to ]

confabulate at kintzios

Apr 8, 2023, 1:41 AM

Post #6 of 10 (252 views)

On Saturday, 8 April 2023 09:33:23 BST Michael wrote:
> On Saturday, 8 April 2023 09:32:31 BST Michael wrote:
> > On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
> > > On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
> > > > It was probably the addition of IPv6 - as a workaround I added 'sleep
> > > > 5s'
> > > > before the gatling init.d exec command and it now starts normally.
> > >
> > > If it was indeed the network that was missing, you could try changing
> > > the service from "need" to "after" starting the net service.
> > >
> > > Add to /etc/rc.conf:
> > > rc_gatling_after="net"
> > >
> > > Regards,
> > > Arve
> >
> > Thanks Arve, I will try it. Should this replace the "need net" string
> > here, or added somewhere else?
> >
> > depend() {
> >
> > need net
> >
> > }
>
> Oops! Please ignore my question ... I didn't have enough coffee yet. :-)

OK, I added rc_gatling_after="net" in rc.conf, but it didn't work. gatling
crashed. However, adding 'sleep 5s' in its init.d startup script works. What
might be causing this discrepancy?

Re: Openrc service crash [ In reply to ]

arve.barsnes at gmail

Apr 8, 2023, 2:10 AM

Post #7 of 10 (252 views)

On Sat, 8 Apr 2023 at 10:41, Michael <confabulate@kintzios.com> wrote:
> OK, I added rc_gatling_after="net" in rc.conf, but it didn't work. gatling
> crashed. However, adding 'sleep 5s' in its init.d startup script works. What
> might be causing this discrepancy?

I see now in your first message that the output says that gatling will
wait and start after net.enp2s0 has started, I guess there must be a
crash while waiting. Is there no logs from the daemon? There might be
a way to add a delay in rc.conf as well, so you don't directly edit
the init script.

Regards,
Arve

Re: Openrc service crash [ In reply to ]

holger at applied-asynchrony

Apr 8, 2023, 3:08 AM

Post #8 of 10 (252 views)

On 2023-04-08 10:41, Michael wrote:
> On Saturday, 8 April 2023 09:33:23 BST Michael wrote:
>> On Saturday, 8 April 2023 09:32:31 BST Michael wrote:
>>> On Saturday, 8 April 2023 09:28:16 BST Arve Barsnes wrote:
>>>> On Sat, 8 Apr 2023 at 10:00, Michael <confabulate@kintzios.com> wrote:
>>>>> It was probably the addition of IPv6 - as a workaround I added 'sleep
>>>>> 5s'
>>>>> before the gatling init.d exec command and it now starts normally.
>>>>
>>>> If it was indeed the network that was missing, you could try changing
>>>> the service from "need" to "after" starting the net service.
>>>>
>>>> Add to /etc/rc.conf:
>>>> rc_gatling_after="net"
>>>>
>>>> Regards,
>>>> Arve
>>>
>>> Thanks Arve, I will try it. Should this replace the "need net" string
>>> here, or added somewhere else?
>>>
>>> depend() {
>>>
>>> need net
>>>
>>> }
>>
>> Oops! Please ignore my question ... I didn't have enough coffee yet. :-)
>
> OK, I added rc_gatling_after="net" in rc.conf, but it didn't work. gatling
> crashed. However, adding 'sleep 5s' in its init.d startup script works. What
> might be causing this discrepancy?

Without having more information, I wouldn't be surprised if it's the notorious
IPv6 DAD delay:

https://bugs.gentoo.org/831743

cheers
Holger

Re: Openrc service crash [ In reply to ]

confabulate at kintzios

Apr 8, 2023, 3:24 AM

Post #9 of 10 (252 views)

On Saturday, 8 April 2023 10:10:49 BST Arve Barsnes wrote:

> I see now in your first message that the output says that gatling will
> wait and start after net.enp2s0 has started, I guess there must be a
> crash while waiting. Is there no logs from the daemon? There might be
> a way to add a delay in rc.conf as well, so you don't directly edit
> the init script.
>
> Regards,
> Arve

The gatling webserver is rather simple with no logging or debugging options
that I have found. The rc.conf log does not capture gatling's internal code
crash, so I'm none the wiser. gatling works nicely once up & running with a
minimal footprint, which is why I chose it when boa fell off the tree. I
suppose commensurate with this is its lack of more sophisticated features,
like backgrounding at launch without crashing. ;-p

The *BSD's rc.conf has a netwait option, but I don't think Gentoo's rc uses
this. Hacking the init script with a simple 'sleep' addition is not the best
approach, but I'm happy it works for now. :-)

Re: Re: Openrc service crash [ In reply to ]

confabulate at kintzios

Apr 8, 2023, 3:32 AM

Post #10 of 10 (252 views)

On Saturday, 8 April 2023 11:08:06 BST Holger Hoffstätte wrote:
> On 2023-04-08 10:41, Michael wrote:

> > OK, I added rc_gatling_after="net" in rc.conf, but it didn't work.
> > gatling
> > crashed. However, adding 'sleep 5s' in its init.d startup script works.
> > What might be causing this discrepancy?
>
> Without having more information, I wouldn't be surprised if it's the
> notorious IPv6 DAD delay:
>
> https://bugs.gentoo.org/831743
>
> cheers
> Holger

Yes, quite likely. Although this application is configured to listen only on
IPv4, the problem manifested after I moved this system on a new network port
which also has IPv6.

I've come across posts complaining about services crashing at startup, but
causal attribution was somewhat conflated between 1Gb switch gear and IPv6.
Since this PC was on the same 1Gbps switch before and only the addition of
IPv6 coincided with this problem, I'd think the bug you mention is a more
probable cause.