Mailing List Archive: polling consoles

polling consoles

Oct 22, 2002, 7:46 AM

Post #1 of 11 (1579 views)

I was wondering if there is any way for conserver to test the connection
between itself and the consoles it is aware of.

We run into a problem that if a terminal server goes down unexpectedly,
conserver will not realize that the TCP connections have been broken and
will keep them marked as up. The terminal server, once back up, no
longer knows about the connections so none of the console messages are
put into the logs, until conserver is either restarted or an explicit
connection is made to each machine.

A similar problem occurs if one of the nodes in our cluster crashes.
Once it is fixed and brought back up, conserver does not kill the bad
connection and reestablish a connection with the node until an explicit
connection is made to the node.

I know this popped up on the list about a year ago and at that time
there was nothing that could be done within conserver. I was wondering
if there is now a way to monitor connection status and if not get
suggestions as to how to best implement it.

Corey

Re: polling consoles [ In reply to ]

cfowler at outpostsentinel

Oct 22, 2002, 7:53 AM

Post #2 of 11 (1568 views)

Permalink

I have this same problem with the console client. We have been
embedding conserver and rebooting our embedded devices often. So when
someone reboots a device console does not know of that connection death.

On Tue, 2002-10-22 at 10:46, Corey Hirschman wrote:
> I was wondering if there is any way for conserver to test the connection
> between itself and the consoles it is aware of.
>
> We run into a problem that if a terminal server goes down unexpectedly,
> conserver will not realize that the TCP connections have been broken and
> will keep them marked as up. The terminal server, once back up, no
> longer knows about the connections so none of the console messages are
> put into the logs, until conserver is either restarted or an explicit
> connection is made to each machine.
>
> A similar problem occurs if one of the nodes in our cluster crashes.
> Once it is fixed and brought back up, conserver does not kill the bad
> connection and reestablish a connection with the node until an explicit
> connection is made to the node.
>
> I know this popped up on the list about a year ago and at that time
> there was nothing that could be done within conserver. I was wondering
> if there is now a way to monitor connection status and if not get
> suggestions as to how to best implement it.
>
> Corey
>
>
>
>
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
>
>

Re: polling consoles [ In reply to ]

cfowler at outpostsentinel

Oct 22, 2002, 8:11 AM

Post #3 of 11 (1569 views)

Permalink

My problem is that I only get to Error("lost connection"); After
console.c attempts a write. I also get garbage displayed on tty after
the lost connection message is displayed. I'm tracking that down now!

On Tue, 2002-10-22 at 10:53, cfowler wrote:
> I have this same problem with the console client. We have been
> embedding conserver and rebooting our embedded devices often. So when
> someone reboots a device console does not know of that connection death.
>
>
> On Tue, 2002-10-22 at 10:46, Corey Hirschman wrote:
> > I was wondering if there is any way for conserver to test the connection
> > between itself and the consoles it is aware of.
> >
> > We run into a problem that if a terminal server goes down unexpectedly,
> > conserver will not realize that the TCP connections have been broken and
> > will keep them marked as up. The terminal server, once back up, no
> > longer knows about the connections so none of the console messages are
> > put into the logs, until conserver is either restarted or an explicit
> > connection is made to each machine.
> >
> > A similar problem occurs if one of the nodes in our cluster crashes.
> > Once it is fixed and brought back up, conserver does not kill the bad
> > connection and reestablish a connection with the node until an explicit
> > connection is made to the node.
> >
> > I know this popped up on the list about a year ago and at that time
> > there was nothing that could be done within conserver. I was wondering
> > if there is now a way to monitor connection status and if not get
> > suggestions as to how to best implement it.
> >
> > Corey
> >
> >
> >
> >
> > _______________________________________________
> > users mailing list
> > users@conserver.com
> > https://www.conserver.com/mailman/listinfo/users
> >
> >
>
>
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
>
>

Re: polling consoles [ In reply to ]

corey at rentec

Oct 22, 2002, 8:41 AM

Post #4 of 11 (1564 views)

Permalink

You seem to have a greater problem than I do. :)

Once the terminal server is back up, I can do console <node name>,
conserver then realizes the connection is really not up and does a reset
to bring it up. After that I am good to go. The problem is that I have
to do that to reestablish the connection, or restart conserver. It
would be better (at least I think so) if conserver did this
automatically at a time specified in minutes on the command line when
conserver is launched or perhaps in the conserver.cf file. This way,
any console messages will continue to be logged automatically after only
a short interruption.

On Tue, 2002-10-22 at 11:34, cfowler wrote:
> Here is a chnage I made to console.c form 7.2.3 It needs a tweak but it
> clears up the garbage on connection loss.
>
>
> Index: console.c
> ==================================================================
> RCS file: /u01/ENS/CVS/outpost/src/conserver/console/console.c,v
> retrieving revision 1.2
> diff -u -r1.2 console.c
> --- console.c 16 Oct 2002 18:00:55 -0000 1.2
> +++ console.c 22 Oct 2002 15:32:15 -0000
> @@ -1113,11 +1113,9 @@
>
> /* anything from socket? */
> if (FD_ISSET(fileFDNum(pcf), &rmask)) {
> - if ((nc = fileRead(pcf, acMesg, sizeof(acMesg))) == 0) {
> - if (justProcessedUrg) {
> - printf("\n");
> - Error("lost connection");
> - }
> + if ((nc = fileRead(pcf, acMesg, sizeof(acMesg))) <= 0) {
> + printf("\n");
> + Error("lost connection");
> break;
> }
> if (fStrip) {
> @@ -1141,6 +1139,9 @@
> c2cooked();
> if (fVerbose)
> printf("Console %s closed.\n", pcMach);
> +
> + printf("\n");
> +
> return 0;
> }
>
>
>
> On Tue, 2002-10-22 at 10:46, Corey Hirschman wrote:
> > I was wondering if there is any way for conserver to test the connection
> > between itself and the consoles it is aware of.
> >
> > We run into a problem that if a terminal server goes down unexpectedly,
> > conserver will not realize that the TCP connections have been broken and
> > will keep them marked as up. The terminal server, once back up, no
> > longer knows about the connections so none of the console messages are
> > put into the logs, until conserver is either restarted or an explicit
> > connection is made to each machine.
> >
> > A similar problem occurs if one of the nodes in our cluster crashes.
> > Once it is fixed and brought back up, conserver does not kill the bad
> > connection and reestablish a connection with the node until an explicit
> > connection is made to the node.
> >
> > I know this popped up on the list about a year ago and at that time
> > there was nothing that could be done within conserver. I was wondering
> > if there is now a way to monitor connection status and if not get
> > suggestions as to how to best implement it.
> >
> > Corey
> >
> >
> >
> >
> > _______________________________________________
> > users mailing list
> > users@conserver.com
> > https://www.conserver.com/mailman/listinfo/users
> >
> >
>
>
>

Re: polling consoles [ In reply to ]

cfowler at outpostsentinel

Oct 22, 2002, 9:41 AM

Post #5 of 11 (1572 views)

Permalink

Yea I see. I've modified my conserver daemon to send notifications of
these conditions. I need this condition to tell administrators that the
managed device is down. Of course I use SNMP traps to do this. I
pretty much all the time know what state any console is in.

]

On Tue, 2002-10-22 at 11:41, Corey Hirschman wrote:
> You seem to have a greater problem than I do. :)
>
> Once the terminal server is back up, I can do console <node name>,
> conserver then realizes the connection is really not up and does a reset
> to bring it up. After that I am good to go. The problem is that I have
> to do that to reestablish the connection, or restart conserver. It
> would be better (at least I think so) if conserver did this
> automatically at a time specified in minutes on the command line when
> conserver is launched or perhaps in the conserver.cf file. This way,
> any console messages will continue to be logged automatically after only
> a short interruption.
>
>
> On Tue, 2002-10-22 at 11:34, cfowler wrote:
> > Here is a chnage I made to console.c form 7.2.3 It needs a tweak but it
> > clears up the garbage on connection loss.
> >
> >
> > Index: console.c
> > ==================================================================
> > RCS file: /u01/ENS/CVS/outpost/src/conserver/console/console.c,v
> > retrieving revision 1.2
> > diff -u -r1.2 console.c
> > --- console.c 16 Oct 2002 18:00:55 -0000 1.2
> > +++ console.c 22 Oct 2002 15:32:15 -0000
> > @@ -1113,11 +1113,9 @@
> >
> > /* anything from socket? */
> > if (FD_ISSET(fileFDNum(pcf), &rmask)) {
> > - if ((nc = fileRead(pcf, acMesg, sizeof(acMesg))) == 0) {
> > - if (justProcessedUrg) {
> > - printf("\n");
> > - Error("lost connection");
> > - }
> > + if ((nc = fileRead(pcf, acMesg, sizeof(acMesg))) <= 0) {
> > + printf("\n");
> > + Error("lost connection");
> > break;
> > }
> > if (fStrip) {
> > @@ -1141,6 +1139,9 @@
> > c2cooked();
> > if (fVerbose)
> > printf("Console %s closed.\n", pcMach);
> > +
> > + printf("\n");
> > +
> > return 0;
> > }
> >
> >
> >
> > On Tue, 2002-10-22 at 10:46, Corey Hirschman wrote:
> > > I was wondering if there is any way for conserver to test the connection
> > > between itself and the consoles it is aware of.
> > >
> > > We run into a problem that if a terminal server goes down unexpectedly,
> > > conserver will not realize that the TCP connections have been broken and
> > > will keep them marked as up. The terminal server, once back up, no
> > > longer knows about the connections so none of the console messages are
> > > put into the logs, until conserver is either restarted or an explicit
> > > connection is made to each machine.
> > >
> > > A similar problem occurs if one of the nodes in our cluster crashes.
> > > Once it is fixed and brought back up, conserver does not kill the bad
> > > connection and reestablish a connection with the node until an explicit
> > > connection is made to the node.
> > >
> > > I know this popped up on the list about a year ago and at that time
> > > there was nothing that could be done within conserver. I was wondering
> > > if there is now a way to monitor connection status and if not get
> > > suggestions as to how to best implement it.
> > >
> > > Corey
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > users mailing list
> > > users@conserver.com
> > > https://www.conserver.com/mailman/listinfo/users
> > >
> > >
> >
> >
> >
>
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
>
>

Re: polling consoles [ In reply to ]

bryan at conserver

Oct 22, 2002, 2:38 PM

Post #6 of 11 (1567 views)

Permalink

Quoting cfowler <cfowler@outpostsentinel.com>:
> I have this same problem with the console client. We have been
> embedding conserver and rebooting our embedded devices often. So when
> someone reboots a device console does not know of that connection
> death.

assuming you're still using the code based on what is released on
conserver.com, you will get notifications to all the clients if you
actually kill the conserver daemon (gracefully) instead of just halting
the operating system or sending it a SIGKILL.

just my 2 cents, for now.

Bryan

Re: polling consoles [ In reply to ]

bryan at conserver

Oct 22, 2002, 2:43 PM

Post #7 of 11 (1567 views)

Permalink

Quoting Corey Hirschman <corey@rentec.com>:
> I was wondering if there is any way for conserver to test the
> connection
> between itself and the consoles it is aware of.

you know, that's a good question. i'm not sure if there is, but it
would be an interesting feature to add. perhaps, since most terminal
vendors use the "reverse-telnet" stuff we can send a noop-type telnet
escape sequence if there hasn't been activity. but, that doesn't
really help in other cases, such as an ssh connection done via the
"|/usr/local/bin/ssh..." type syntax...that too will just sorta hang
and not notice an abrupt drop in a terminal server.

so, for a general case, i don't have any good ideas...but perhaps we
can do something for the telnet-based terminal servers.

anyone have any other bright ideas? some socket-level thing we can do?
although that doesn't help the ssh-type stuff...ugh.

we can't send regular traffic, that's for sure.

Bryan

Re: polling consoles [ In reply to ]

cfowler at outpostsentinel

Oct 22, 2002, 2:45 PM

Post #8 of 11 (1561 views)

Permalink

Oops. Sorry I confused some. Yes it is based on conserver.com. I get
snmp traps when any console drops its DTR.

On Tue, 2002-10-22 at 17:38, bryan@conserver.com wrote:
> Quoting cfowler <cfowler@outpostsentinel.com>:
> > I have this same problem with the console client. We have been
> > embedding conserver and rebooting our embedded devices often. So when
> > someone reboots a device console does not know of that connection
> > death.
>
> assuming you're still using the code based on what is released on
> conserver.com, you will get notifications to all the clients if you
> actually kill the conserver daemon (gracefully) instead of just halting
> the operating system or sending it a SIGKILL.
>
> just my 2 cents, for now.
>
> Bryan
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
>
>

Re: polling consoles [ In reply to ]

cfowler at outpostsentinel

Oct 22, 2002, 2:47 PM

Post #9 of 11 (1565 views)

Permalink

What about SOKEEPALIVE?

On Tue, 2002-10-22 at 17:43, bryan@conserver.com wrote:
> Quoting Corey Hirschman <corey@rentec.com>:
> > I was wondering if there is any way for conserver to test the
> > connection
> > between itself and the consoles it is aware of.
>
> you know, that's a good question. i'm not sure if there is, but it
> would be an interesting feature to add. perhaps, since most terminal
> vendors use the "reverse-telnet" stuff we can send a noop-type telnet
> escape sequence if there hasn't been activity. but, that doesn't
> really help in other cases, such as an ssh connection done via the
> "|/usr/local/bin/ssh..." type syntax...that too will just sorta hang
> and not notice an abrupt drop in a terminal server.
>
> so, for a general case, i don't have any good ideas...but perhaps we
> can do something for the telnet-based terminal servers.
>
> anyone have any other bright ideas? some socket-level thing we can do?
> although that doesn't help the ssh-type stuff...ugh.
>
> we can't send regular traffic, that's for sure.
>
> Bryan
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
>
>

Re: polling consoles [ In reply to ]

corey at rentec

Oct 24, 2002, 8:41 AM

Post #10 of 11 (1563 views)

Permalink

As of version 7.2.4 conserver is actually using SOKEEPALIVE. This
explains that weird 2 hour behavior I was seeing. If I killed the power
on the console server and then powered it back up, I would get no
logging for 2 hours. Then a keepalive packet would be sent to all the
connections that were marked as up. Since they were really bad
connections at that point, they would be marked as down. Conserver
would then do it's auto up and reestablish the connections. So this
pretty much takes care of polling the machines. You do lose 2 hours of
logging (unless you feel like playing with TCP), but that is better than
it not coming up at all.

Corey

Re: polling consoles [ In reply to ]

bryan at conserver

Oct 24, 2002, 8:52 PM

Post #11 of 11 (1563 views)

Permalink

actually, it was there before that...can't check on when it got into the
code, but 7.2.2 had it as well (heck, could have been there since 7.0.0).
the only sad part is that it will take a couple hours (for at least some
operating systems) for the keepalive packet to go out...would be
nice to have something a bit quicker. but, at least it's something,
as you say. ;-)

Bryan

Quoting Corey Hirschman <corey@rentec.com>:
> As of version 7.2.4 conserver is actually using SOKEEPALIVE. This
> explains that weird 2 hour behavior I was seeing. If I killed the
> power
> on the console server and then powered it back up, I would get no
> logging for 2 hours. Then a keepalive packet would be sent to all the
> connections that were marked as up. Since they were really bad
> connections at that point, they would be marked as down. Conserver
> would then do it's auto up and reestablish the connections. So this
> pretty much takes care of polling the machines. You do lose 2 hours of
> logging (unless you feel like playing with TCP), but that is better
> than
> it not coming up at all.
>
> Corey
>
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
>