Mailing List Archive: console -A and console down

console -A and console down

Nov 24, 2003, 8:13 PM

Post #1 of 4 (789 views)

I have a console that suddenly decided to go into the down state. The
log the for console looks like this:

[Mon Nov 24 11:30:00 2003] Mon Nov 24 11:30:00 EST 2003
[Mon Nov 24 11:30:00 2003]^M
[Mon Nov 24 11:45:00 2003] Mon Nov 24 11:45:00 EST 2003
[Mon Nov 24 11:45:00 2003]^M
Mon Nov 24 12:00:01 EST 2003
[Mon Nov 24 12:00:01 2003]^M
[-- Console down -- Mon Nov 24 12:11:00 2003]
[-- Console up -- Mon Nov 24 12:11:22 2003]

[Mon Nov 24 12:11:23 2003]Password OK
[-- Console down -- Mon Nov 24 12:12:18 2003]
[-- Console up -- Mon Nov 24 12:12:30 2003]

[Mon Nov 24 12:12:30 2003]Password OK
[-- Console down -- Mon Nov 24 12:14:04 2003]
[-- Console up -- Mon Nov 24 12:14:13 2003]

[Mon Nov 24 12:14:14 2003]Password OK
[Mon Nov 24 12:15:00 2003] Mon Nov 24 12:15:00 EST 2003
[Mon Nov 24 12:15:00 2003]^M
[-- Console down -- Mon Nov 24 12:25:44 2003]
[-- Console up -- Mon Nov 24 12:25:44 2003]

[Mon Nov 24 12:25:45 2003]Password OK
[-- Console down -- Mon Nov 24 14:25:44 2003]
[-- Console up -- Mon Nov 24 14:25:45 2003]
[-- Console down -- Mon Nov 24 14:25:45 2003]
[-- Console up -- Mon Nov 24 14:37:16 2003]
[-- Console down -- Mon Nov 24 14:37:17 2003]

Each machine prints the time to the console every 15 minutes to confirm
that the console is working. We are using Cisco 3620 terminal servers.
Any idea why the console would suddenly go down like that?

When I try to connect to the console:

$ console -A foo
[Enter `^Ec?' for help]
[line to console is down]
[replay]
[no log file on this console]

But there is a log file for this console. /var/log/consoles/foo exists
and has data in it.

The conserver.cf entry looks like this:

console foo {
host bar;
port 2020;
logfile /var/log/consoles/&;
master localhost;
type host;
timestamp 1la;
}

Re: console -A and console down [ In reply to ]

bryan at conserver

Nov 27, 2003, 8:23 PM

Post #2 of 4 (763 views)

Permalink

On Mon, Nov 24, 2003 at 10:13:42PM -0500, Matt Selsky wrote:
> Each machine prints the time to the console every 15 minutes to
> confirm that the console is working. We are using Cisco 3620 terminal
> servers. Any idea why the console would suddenly go down like that?

are there any messages in the conserver logfile (either stdout/stderr
or -L/'logfile' option)? that would be the only place there might be a
clue, unless the cisco is logging something on it's console. i'd check
both.

> When I try to connect to the console:
>
> $ console -A foo
> [Enter `^Ec?' for help]
> [line to console is down]
> [replay]
> [no log file on this console]
>
> But there is a log file for this console. /var/log/consoles/foo exists
> and has data in it.

yeah, the logfile isn't accessible unless the console is up. now that
you've pointed it out, that really seems wrong. personally, i think it
would be useful to replay the logfile if the console is down...might
help explain why it went down, or let you fix problems. i'll see if i
can "fix" this. i don't *think* there's a problem (inside the code)
with having the logfile open even though the console is down. there
should be only a few things to adjust (i think!) to use that philosophy.

Bryan

Re: console -A and console down [ In reply to ]

selsky at columbia

Dec 1, 2003, 2:46 PM

Post #3 of 4 (766 views)

Permalink

> are there any messages in the conserver logfile (either stdout/stderr
> or -L/'logfile' option)? that would be the only place there might be a
> clue, unless the cisco is logging something on it's console. i'd check
> both.
>

Our cisco apparently isn't sysloging anywhere since it's on the private
vlan. Our conserver logfile contains:

[Mon Nov 24 12:25:44 2003] conserver (27009): [foo] automatic
reinitialization
[Mon Nov 24 14:25:44 2003] conserver (27009): [foo] automatic
reinitialization
[Mon Nov 24 14:25:45 2003] conserver (27009): ERROR: [foo] connect(22):
Connection refused: forcing down

> yeah, the logfile isn't accessible unless the console is up. now that
> you've pointed it out, that really seems wrong. personally, i think it
> would be useful to replay the logfile if the console is down...might
> help explain why it went down, or let you fix problems. i'll see if i
> can "fix" this. i don't *think* there's a problem (inside the code)
> with having the logfile open even though the console is down. there
> should be only a few things to adjust (i think!) to use that
> philosophy.

Thanks.

Re: console -A and console down [ In reply to ]

bryan at conserver

Dec 1, 2003, 5:53 PM

Post #4 of 4 (761 views)

Permalink

On Mon, Dec 01, 2003 at 04:46:22PM -0500, Matt Selsky wrote:
> Our cisco apparently isn't sysloging anywhere since it's on the private
> vlan. Our conserver logfile contains:
>
> [Mon Nov 24 12:25:44 2003] conserver (27009): [foo] automatic
> reinitialization
> [Mon Nov 24 14:25:44 2003] conserver (27009): [foo] automatic
> reinitialization
> [Mon Nov 24 14:25:45 2003] conserver (27009): ERROR: [foo] connect(22):
> Connection refused: forcing down
>

hmmm...looks like i'm missing a failure message in there, but from the
code, i think it means it got a read() failure...probably (confident,
aren't i?). then after attacking the term server a couple times on the
port, the cisco probably refused the connection and so conserver stopped
pounding it.

i'm not sure what to tell you. we could put some more messages in
conserver, which might help, but to really tell what's going on, i'd
look at the cisco console or it's syslog data...if you can get to
either. but, if you'd like a patch for more log info, let me know (i'll
be expanding it in the next release).

i suppose the other option is to take your config file, comment out all
consoles except this one, and run another copy of conserver, in debug
mode (probably -DD). just make sure the first copy of conserver either
has this console commented out (and send it a HUP) or just let it stay
down (^Ecd). then we can see a whole lot more of what's happening
inside conserver when it fails. once it does, you can shut down the
second conserver and just bring the original console back up. when
running the second conserver, just include something like '-p 7777' so
it attaches to a different port (and you can use 'console -p 7777
<consolename>') to attach to it. we may not see more than a read()
failure, however. if so, we'll have to look at the cisco logs to get
better info.

Bryan