Mailing List Archive

Unexplained Interchange crashes
Hello All



I am hoping for some advice on a problem I have.



This Interchange server has been up working 24/7 for several years, and
interchange has been rock solid and never crashes.



The physical server was moved to a new building, and since then the
Interchange service randomly stops for no obvious reason.



As this service is started from /etc/init.d I cannot get it to simply
respawn a new process.



Has anyone any tips on diagnosing why the process just dies, or a better way
of restarting IC if it randomly stops?



Thanks in advance



Phil.
Re: Unexplained Interchange crashes [ In reply to ]
On 23/08/22 22:35, Phil Smith wrote:
> Hello All
>
> I am hoping for some advice on a problem I have.
>
> This Interchange server has been up working 24/7 for several years, and
> interchange has been rock solid and never crashes.
>
> The physical server was moved to a new building, and since then the
> Interchange service randomly stops for no obvious reason.
>
> As this service is started from /etc/init.d I cannot get it to simply
> respawn a new process.
>
> Has anyone any tips on diagnosing why the process just dies, or a better
> way of restarting IC if it randomly stops?

If it's crashing just since you moved the server to a new building and
you literally changed nothing else then the issue has to be
environmental, or hardware-related. Something as simple as a power
glitch could cause issues that you did not anticipate.

That said, what you can do for now, and it's probably a good idea to do
anyways, is set up a "watchdog" task that runs from cron. Have it
attempt to fetch an IC page and if it cannot then it should restart the
service (and send you a notice that it did).

If you're running a Linux distribution with systemd then consider using
a systemd .service file to start IC instead of an init.d script. Here's
one I use, with an additional line I recently added to tell systemd to
automatically restart IC on failure. Note you can get this file by
unpacking the interchange package from GhettoForge. Note also that I've
changed the locations of many of the interchange files in this to be
more appropriate to a packaged interchange, so you may have to change
some of this stuff (or omit some of the *Directory directives alltogether).

See systemd documentation to see how all of this works:

# /usr/lib/systemd/system/interchange.service
[Unit]
Description=Interchange web application platform
After=syslog.target network.target

[Service]
Type=forking
PIDFile=/var/run/interchange/interchange.pid
EnvironmentFile=-/etc/sysconfig/network
User=interch
Group=interch
ExecStart=/usr/sbin/interchange -q
ExecStop=/usr/sbin/interchange -q --stop
RuntimeDirectory=interchange
CacheDirectory=interchange
LogsDirectory=interchange
StateDirectory=interchange
ConfigurationDirectory=interchange


[Install]
WantedBy=multi-user.target

# /etc/systemd/system/interchange.service.d/override.conf
[Service]
Restart=on-failure



Peter
_______________________________________________
interchange-users mailing list
interchange-users@interchangecommerce.org
https://www.interchangecommerce.org/mailman/listinfo/interchange-users
Re: Unexplained Interchange crashes [ In reply to ]
> On 23/08/22 22:35, Phil Smith wrote:
> > Hello All
> >
> > I am hoping for some advice on a problem I have.
> >
> > This Interchange server has been up working 24/7 for several years,
> > and interchange has been rock solid and never crashes.
> >
> >[ ....]
>
> If it's crashing just since you moved the server to a new building and you
> literally changed nothing else then the issue has to be environmental, or
> hardware-related. Something as simple as a power glitch could cause
issues
> that you did not anticipate.
>
> That said, what you can do for now, and it's probably a good idea to do
> anyways, is set up a "watchdog" task that runs from cron. Have it attempt
to
> fetch an IC page and if it cannot then it should restart the service (and
send
> you a notice that it did).
>
> If you're running a Linux distribution with systemd then consider using a
> systemd .service file to start IC instead of an init.d script. Here's one
I use,
> with an additional line I recently added to tell systemd to automatically
> restart IC on failure. Note you can get this file by unpacking the
interchange
> package from GhettoForge. Note also that I've changed the locations of
> many of the interchange files in this to be more appropriate to a packaged
> interchange, so you may have to change some of this stuff (or omit some of
> the *Directory directives alltogether).
>
> See systemd documentation to see how all of this works:
>
> # /usr/lib/systemd/system/interchange.service
> [Unit]
> Description=Interchange web application platform After=syslog.target
> network.target
>
> [Service]
> Type=forking
> PIDFile=/var/run/interchange/interchange.pid
> EnvironmentFile=-/etc/sysconfig/network
> User=interch
> Group=interch
> ExecStart=/usr/sbin/interchange -q
> ExecStop=/usr/sbin/interchange -q --stop RuntimeDirectory=interchange
> CacheDirectory=interchange LogsDirectory=interchange
> StateDirectory=interchange ConfigurationDirectory=interchange
>
>
> [Install]
> WantedBy=multi-user.target
>
> # /etc/systemd/system/interchange.service.d/override.conf
> [Service]
> Restart=on-failure
>
>
>
> Peter

Thank you Peter, the problem with Environmental issues is - this new Comms
room is 1000% better than old building - UPS backed and air conditioned. No
other service seems to be affected - just IC.

I have already set up a 'watchdog' process, it pulls the PID from
/etc/interchange/etc/interchange.pid and periodically checks that is still
alive from `ps -p $pid`. If its not running it currently emails me to go
restart it.

I will have a look through your interchange.service above and see if I can
tweak it to fit my system (thanks).

I just wish something was logging why it has started crashing.

Thanks

Phil.

_______________________________________________
interchange-users mailing list
interchange-users@interchangecommerce.org
https://www.interchangecommerce.org/mailman/listinfo/interchange-users