Mailing List Archive

Using conserver with vmware
Hi,

LOOOOONG time serial console user, been running conserver for many years.
Great package.

For most sites, conserver primarily represents OOB access to server
consoles for beating on them in singleuser mode, etc. With things like
IP KVM viewers, hypervisors, etc., most sites seem to be ditching serial
consoles.

We've been on a virtualization path for a while now, here, as well. A
lot of our legacy gear has been powered off. But I find KVM access to be
annoying and lacking in several ways, specifically:

1) Bandwidth intensive

2) Inability to replay "wtf just happened that rebooted my server."

So I'm playing with a trial of VMware's products, and one of the things
that seems to be available is the ability for the host system to connect
a virtual serial port ... somewhere. Listening is also possible.

If you set the host to listen, you can have conserver connect to a serial
console of a VM on the host (vmware doesn't support serial consoles, but
the guest OS's do). There's just one little problem: you have to wire in
knowledge of what VM is on which host, which makes migrations a nightmare.

You can also set the host to establish a connection ... somewhere.

Like to the conserver box. So my obvious first attempt was to use netcat,
using "nc -l <port>" like so:

default vmlisten { type exec; portbase 7000; portinc 1;
exec /usr/bin/nc -o -l P;
execsubst P=Pd; }

And this works... at first. But if there's a console that's NOT up, say
because the VM is down, our system is designed to restart conserver every
night when it rotates logs, and "nc -l <port>" processes start to stack up,
because nc doesn't stop waiting for an incoming connection just because its
stdin has vanished. Foo.

I suppose another option is to write a smarter version of netcat that does
not act stupid. It's trivial to do, but watching nc processes stack up in
an unexpected manner just suggested to me that maybe this was the wrong way
to go about it.

So I was left wondering if I was reinventing the wheel, here. Has anyone
hacked conserver to itself listen for connections, rather than initiating
them? I find it hard to believe that no one else has found serial consoles
to virtual machines to be useful functionality.

... JG
_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Using conserver with vmware [ In reply to ]
On Wed, 2011-12-07 at 07:15 -0600, Joe Greco wrote:
> Anyways, the point here is that we're trying to get rid of unnecessary
> hardware. We're already ahead of the curve in terms of getting rid of
> lots of serial ports, terminal servers, and associated wiring. I do
> encourageyou to look at this issue and see if you've not made your
> environment more complicated than necessary by adding a multiport
> serial board.

Joe,

I don't do migrations. My environment is not as complex.

I use VMware to provide me guests for running my software and for
development needs. The only migration I have done has been to convert
a guest into real hardware.

I have expanded my guest usage by using Rackspace and I expect to grow
into using them a lot more. They do not support any form of headless
monitoring.

"Both of THOSE problems would be solved if there was a
well-integrated feature in conserver to listen for connections."

Years ago I considered myself a "UNIX guru". I thought I knew
everything there was about UNIX. I thought I was a UNIX administrator.
I was naive. When I purchased "Advanced Programming in the UNIX
Environment", read it, and then starting coding. That is when I learned
UNIX. I guess it was when I was able to really start telling UNIX what
to do instead of it telling me what to do.

If you were to go down the path of making work what you have instead of
modification of VMware and conserver then I would write a program on the
host that runs conserver that listens for connections. I've not thought
about your problem in detail so what I suggest may not solve it.
Configure conserver to connect to pseudo ttys on the host. Those ttys
are created by the process that listens and marries the pseudos to the
remote device. I've done this many times when I've written serial tcp
code and RFC2217 type stuff.

Administrators don't like to program because programming is boring and
the task of CS grads. No son, programming allows you to be the boss of
your systems not a slave. You don't like what your server is doing.
Tell it how to do it better. I'm not a CS grad. I'm an administrator
that has followed the path of a Jedi.

I have another problem with VMware that I can't fix since I don't have
the VM code. VMware does not respect tty settings in the guest and
apply those on the host. Not only do I have a serial port in my server
I also have an 8 port modem board. VMware treats these ports as raw
with no flow control. When old ascii devices dial in and dump alarms I
miss data when the software is too busy doing stuff to listen that ms
for incoming data. My solution is to remove that part of the software
out of the guest and run it on the host directly. Another solution is
to go real hardware with that piece. It may be possible to write code
on the host that loops and sets the termios settings on that tty. I've
not tried that.

The good news is that all of us here know how nice it is to work from
home, office, etc and not go to a data center. So many other folks look
at crash carts as a viable solution. The only time I ever go to our
rack is when I have to touch hardware. I can control power and get to
the console without leaving the comforts of my man cave at home.

Chris



_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Using conserver with vmware [ In reply to ]
On Tue, 6 Dec 2011, Joe Greco wrote:

> Like to the conserver box. So my obvious first attempt was to use netcat,
> using "nc -l <port>" like so: ...
>
> And this works... at first. But if there's a console that's NOT up, say
> because the VM is down, our system is designed to restart conserver every
> night when it rotates logs, and "nc -l <port>" processes start to stack up,
> because nc doesn't stop waiting for an incoming connection just because its
> stdin has vanished. Foo.

You might want to take a look at the -q option to netcat:

-q seconds after EOF on stdin, wait the specified number of seconds and
then quit. If seconds is negative, wait forever.

I have never used netcat with conserver, but I have used it to listen for
incoming connection in other circumstances. Without that -q option, netcat
hangs even after the "client" has gone away. Guess netcat is waiting for
the next client, but you don't always want that. TTYL.

---------------------------------------------------------------------------
| "For to me to live is Christ, and to die is gain." --- Phil. 1:21 (KJV) |
---------------------------------------------------------------------------
| Ryan Kirkpatrick | Boulder, CO | rkirkpat.net | twitter.com/rkirkpatnet |
---------------------------------------------------------------------------



_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Using conserver with vmware [ In reply to ]
There may be a way you could use dnsmasq on the conserver host.

dnsmasq will first consult /etc/hosts before it uses /etc/resolv.conf.
Consider it a primitive form of DNS server.

If you do a migration you can change /etc/hosts. Netcat does not need
to be restarted if you tweak the kernel's TCP kleep alive settings.
These need to be tuned anyway to guarantee you connected as often as
possible. On our system this is set from 2 hours to 5 minutes. Seems
dramatic, but we consider connections to be critical.

The only other issue would be the issue of ports.

Maybe you write a script that manages net cat on the conserver host and
references an XML file for config. You update the file as needed and
when the nc process dies a manager will execute it again with the proper
port number.

Chris




_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Using conserver with vmware [ In reply to ]
On Wed, 2011-12-07 at 07:59 -0700, Ryan Kirkpatrick wrote:
> Without that -q option, netcat hangs even after the "client" has gone
> away. Guess netcat is waiting for the next client, but you don't
> always want that. TTYL.

TCP_KEEPALIVE?


_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Using conserver with vmware [ In reply to ]
On Wed, 2011-12-07 at 08:58 -0600, Joe Greco wrote:
> [from a message exchange in private e-mail]

> Right. I had a complex question. You suggested to me in private e-mail

Darn, my Android did not do a reply-all I thought I had provided a
suggestion to the list. Sorry about that. I was laying in the bed at
the time I responded. Unfortunately I was trying to go to sleep. :)

>
> > I have expanded my guest usage by using Rackspace and I expect to grow
> > into using them a lot more. They do not support any form of headless
> > monitoring.
>
> I never understood why these places can't be bothered to provide some form
> of headless monitoring.
>

Simple. $$$

> > "Both of THOSE problems would be solved if there was a
> > well-integrated feature in conserver to listen for connections."
> >
> > Years ago I considered myself a "UNIX guru". I thought I knew
> > everything there was about UNIX. I thought I was a UNIX administrator.
> > I was naive. When I purchased "Advanced Programming in the UNIX
> > Environment", read it, and then starting coding. That is when I learned
> > UNIX. I guess it was when I was able to really start telling UNIX what
> > to do instead of it telling me what to do.
>
> Well, that there's the question. I'm looking to arrange the most elegant
> solution, rather than cobbling together something that's a botch. Given
> that conserver *already* has the capability of dealing with physical serial
> ports and remote network ports, which means that it has file descriptors
> for each port, the missing bits are probably some scaffolding for coping
> with a listening socket when there's not an established connection.
>
> > If you were to go down the path of making work what you have instead of
> > modification of VMware and conserver then I would write a program on the
> > host that runs conserver that listens for connections.
>
> I believe I actually said: "I suppose another option is to write a smarter
> version of netcat that does not act stupid. It's trivial to do, but
> watching nc processes stack up in an unexpected manner just suggested to
> me that maybe this was the wrong way to go about it."
>
> It's a fifteen minute program, but it means that there would be lots of
> extra processes and so as a solution it is really not that great. That's
> what prompted me to write to the mailing list. It seemed odd that no one
> else has thought of this.
>
> > I've not thought
> > about your problem in detail so what I suggest may not solve it.
> > Configure conserver to connect to pseudo ttys on the host. Those ttys
> > are created by the process that listens and marries the pseudos to the
> > remote device. I've done this many times when I've written serial tcp
> > code and RFC2217 type stuff.
>
> That's ugly and more complicated because you've just added another layer of
> complexity with a pty layer. I'm trying to go the other direction, getting
> rid of extra processes entirely and just see if conserver can be made to take
> care of it directly.
>
> > Administrators don't like to program because programming is boring and
> > the task of CS grads. No son, programming allows you to be the boss of
> > your systems not a slave. You don't like what your server is doing.
> > Tell it how to do it better. I'm not a CS grad. I'm an administrator
> > that has followed the path of a Jedi.
>
> No, coding is the task of CS grads. Administrators like to be lazy. I find
> the best path to being lazy is to come to a full understanding of the problem
> I want to solve, then find the most elegant and simple solution to fix it,
> then spend a little time coding it, at which point I have been completely
> successful at being lazy because now I never have to worry about whatever I
> just fixed again. CS grads can't and don't do this because they've been
> taught to code, and don't have the experience and insight to be able to
> understand systems as a whole. Understanding the bigger picture is the key
> to making the big bucks. :-)

I was never really taught. I taught myself. It was something to do
when we had PCs that did nothing without writing something. I'm not
very old, but lucky I grew up in the days of DOS. Not many distractions
in that OS (Internet) that kept a kid from wanting to write something.

Whenever I have a problem where I can't change what I'm working with I
enjoy finding external means of making it do what I want it to do.
Coding is one tool of doing that.

I should have said that coding is an option but should be a last resort.
I tend to use it as a last resort myself for stuff like this.

>
> > I have another problem with VMware that I can't fix since I don't have
> > the VM code. VMware does not respect tty settings in the guest and
> > apply those on the host. Not only do I have a serial port in my server
> > I also have an 8 port modem board. VMware treats these ports as raw
> > with no flow control. When old ascii devices dial in and dump alarms I
> > miss data when the software is too busy doing stuff to listen that ms
> > for incoming data. My solution is to remove that part of the software
> > out of the guest and run it on the host directly. Another solution is
> > to go real hardware with that piece. It may be possible to write code
> > on the host that loops and sets the termios settings on that tty. I've
> > not tried that.
>
> That sounds unfortunate. Getting serial right with virtualization is kind
> of stinky. I know we've had pretty good luck using Digi PortServers with
> their Windows "RealPort Driver" software which does a very admirable job
> of making the terminal server's port appear to be a directly connected serial
> port. Not technically a "virtualization" solution but works fine when
> virtualized, even on things like free ESXi, because it doesn't involve VMware
> at all - it's all network.

This is one solution I'm looking at too. Linux does support RFC2217
servers, but I'm not having a lot of luck finding clients. The way you
implement RFC2217 in client form requires an OS driver to create real
character devices. This is where serial->tcp type programs (which run
only in user space) suck. I know very little about Win32 internals so
in Windows it may be possible to implement character devices in user
land. On Linux the only character device you can implement is a pseudo
and many programs are hard coded to check for pseudo.

>
> You might find that something like that, combined with a USR MP/8, might make
> an acceptable replacement for your 8-port modem board. I think we got rid of
> our last MP/8 or I'd offer it to you.
>
> > The good news is that all of us here know how nice it is to work from
> > home, office, etc and not go to a data center. So many other folks look
> > at crash carts as a viable solution. The only time I ever go to our
> > rack is when I have to touch hardware. I can control power and get to
> > the console without leaving the comforts of my man cave at home.
>
> Crash carts are so 1995. You can't even *do* (much) stuff from the console
> of most VM hosts.
>

Yep. If I do go to the DC for my VM I use a browser on my netbook. I
still run my server headless.

> I've watched what used to be a roomful of gear back in the '90's shrink
> down to fit comfortably on two fairly lean VMware hosts. Our core
> support infrastructure, administrative servers, log analyzers, VPN servers,
> DHCP/TFTP servers, and all sorts of other miscellaneous stuff, has never
> been really that resource-intensive, and we've managed to cram 30 or 40
> different servers onto two hosts that idle at about 60 watts, around 100
> full tilt. Still need SAN and network gear, but reducing a 30 amp rack
> down to about 4 is pretty cool.
>
> But you have to be in your man cave? Dude, I can control my network from
> my *phone*. ;-) ;-)

Don't have to be. I like being here. I can do it on the road. :)

>
> How times have changed, eh.
>

I'm glad you are having success with going virtual. I've not done a lot
with it, but see it as a way for my org to add capability without adding
physical size.

Some of my code is pretty intensive. I've seen one guest slow down
others. I think that may just be an issue with my hardware not being
good enough. With the VMware you are running do you have the ability
to control the amount of CPU (not number) that a guest can consume?

I'm a CLI guy so I love headless operation.

Chris

_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Using conserver with vmware [ In reply to ]
I need to add that for the stuff we consider critical I use the PCI
board. For the stuff that is not I use the tty pipe method in VM and
run conserver directly on the VM host.

Not relevant to your problem, but something that may be relevant to
someone else.

Chris




_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users