Mailing List Archive

Console taking 100% CPU
Greetings!

I'm having an issue where the 'console' process is taking 100% of my CPU.

This happens whenever controlling terminal vanishes in an unexpected
manner -- sorry for being vague, but I'm not certain exactly what's
happening TTY-wise.

Anyway, it's super easy to duplicate with:
$ echo | console myport
...or...
$ (sleep 3) | console myport

When the pipeline sends EOF, console spins out of control. I'd like
it to die instead :-)

I noticed this because I'm trying to use xinetd to run console. If
the user's telnet session is aborted I get the same result.

The goal of my project is to make TCP ports on a single server act
like a huge terminal server appliance. The
xinetd/chroot/runuser/console/conserver mess will be a big mux,
allowing me to present any console port in the environment directly on
TCP ports on the server. Users won't need the 'console' binary, any
telnet client will do.

There's probably a better way to do this than having xinetd launch
console, but I'm not sure what it is. This console CPU problem is a
show-stopper.

Thoughts on the CPU issue, or a better way to accomplish my goal?

Thanks very much!

/chris
_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Console taking 100% CPU [ In reply to ]
On Tue, 2010-12-14 at 16:30 -0500, Chris Marget wrote:
>
> The goal of my project is to make TCP ports on a single server act
> like a huge terminal server appliance. The
> xinetd/chroot/runuser/console/conserver mess will be a big mux,
> allowing me to present any console port in the environment directly on
> TCP ports on the server. Users won't need the 'console' binary, any
> telnet client will do.

Meh. Sounds like a big CF and security nightmare.

If you're going to do that maybe you don't need conserver. I have some
"reverse TCP" code I wrote some years back that will do exactly what you
are looking to do. It emulates the behavior of an old school terminal
server.


>
> There's probably a better way to do this than having xinetd launch
> console, but I'm not sure what it is. This console CPU problem is a
> show-stopper.

Fix it. I've not looked at the new code but it seems to me that it is
not catching EOF. I could be wrong.

(sleep 3) | strace console myconsole 2>/tmp/strace.log

What is it doing?

_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Console taking 100% CPU [ In reply to ]
On Tue, Dec 14, 2010 at 4:39 PM, Chris Fowler
<cfowler@outpostsentinel.com> wrote:
> On Tue, 2010-12-14 at 16:30 -0500, Chris Marget wrote:
>>
>> The goal of my project is to make TCP ports on a single server act
>> like a huge terminal server appliance.  The
>> xinetd/chroot/runuser/console/conserver mess will be a big mux,
>> allowing me to present any console port in the environment directly on
>> TCP ports on the server.  Users won't need the 'console' binary, any
>> telnet client will do.
>
> Meh.  Sounds like a big CF and security nightmare.

Yes. Yes it is :-)

Data in flight is meaningless, so the 'telnet' aspects of it are okay.
This will all be front-ended by a web application that knows who's
signed on to which block of ports, and creates iptables policy for
each user. Then, just for good measure, every web user gets his own
chroot enviroment.

It's ugly, but these problems are all figured out.

> If you're going to do that maybe you don't need conserver.  I have some
> "reverse TCP" code I wrote some years back that will do exactly what you
> are looking to do.  It emulates the behavior of an old school terminal
> server.

I might be interested in this!

> (sleep 3) | strace console myconsole 2>/tmp/strace.log
>
> What is it doing?

It's doing this.

# (sleep 3) | strace -f console 20
execve("/usr/bin/console", ["console", "20"], [/* 32 vars */]) = 0
brk(0) = 0x8bfe000
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7802000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18383, ...}) = 0
mmap2(NULL, 18383, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb77fd000
close(3) = 0
open("/lib/libutil.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0
J\332\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=14640, ...}) = 0
mmap2(0xda4000, 12420, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xda4000
mmap2(0xda6000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1) = 0xda6000
close(3) = 0
open("/lib/libcrypt.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260\10\262\0004\0\0\0"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=40292, ...}) = 0
mmap2(0xb20000, 192860, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb20000
mmap2(0xb27000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7) = 0xb27000
mmap2(0xb29000, 155996, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb29000
close(3) = 0
open("/lib/i686/nosegneg/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360]\223\0004\0\0\0"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1785084, ...}) = 0
mmap2(0x91f000, 1550664, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x91f000
mmap2(0xa94000, 12288, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x175) = 0xa94000
mmap2(0xa97000, 10568, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xa97000
close(3) = 0
open("/lib/libfreebl3.so", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`9\265\0004\0\0\0"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=303640, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb77fc000
mmap2(0xb52000, 318252, PROT_READ|PROT_EXEC,
MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb52000
mmap2(0xb9b000, 4096, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x49) = 0xb9b000
mmap2(0xb9c000, 15148, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xb9c000
close(3) = 0
open("/lib/libdl.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0`\312\251\0004\0\0\0"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=19784, ...}) = 0
mmap2(0xa9c000, 16500, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE,
3, 0) = 0xa9c000
mmap2(0xa9f000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2) = 0xa9f000
close(3) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb77fb000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb77fb6c0,
limit:1048575, seg_32bit:1, contents:0, read_exec_only:0,
limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0xda6000, 4096, PROT_READ) = 0
mprotect(0xb27000, 4096, PROT_READ) = 0
mprotect(0xa94000, 8192, PROT_READ) = 0
mprotect(0x91b000, 4096, PROT_READ) = 0
mprotect(0xa9f000, 4096, PROT_READ) = 0
munmap(0xb77fd000, 18383) = 0
getpid() = 8657
brk(0) = 0x8bfe000
brk(0x8c1f000) = 0x8c1f000
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
ioctl(3, SIOCGIFCONF, {64, {{"lo", {AF_INET, inet_addr("127.0.0.1")}},
{"eth0", {AF_INET, inet_addr("10.122.218.33")}}}}) = 0
ioctl(3, SIOCGIFFLAGS, {ifr_name="lo",
ifr_flags=IFF_UP|IFF_LOOPBACK|IFF_RUNNING}) = 0
ioctl(3, SIOCGIFFLAGS, {ifr_name="eth0",
ifr_flags=IFF_UP|IFF_BROADCAST|IFF_RUNNING|IFF_MULTICAST}) = 0
close(3) = 0
open("/etc/console.cf", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=42, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7801000
read(3, "config * { master localhost; por"..., 4096) = 42
read(3, "", 4096) = 0
close(3) = 0
munmap(0xb7801000, 4096) = 0
open("/home/ec2-user/.consolerc", O_RDONLY) = -1 ENOENT (No such file
or directory)
rt_sigaction(SIGPIPE, {SIG_IGN, [], 0}, NULL, 8) = 0
open("/dev/tty", O_RDONLY) = 3
ioctl(3, TIOCGWINSZ, {ws_row=77, ws_col=127, ws_xpixel=762, ws_ypixel=1078}) = 0
close(3) = 0
open("/etc/resolv.conf", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=80, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7801000
read(3, "; generated by /sbin/dhclient-sc"..., 4096) = 80
read(3, "", 4096) = 0
close(3) = 0
munmap(0xb7801000, 4096) = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1
ENOENT (No such file or directory)
close(3) = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
connect(3, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1
ENOENT (No such file or directory)
close(3) = 0
open("/etc/nsswitch.conf", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1688, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7801000
read(3, "#\n# /etc/nsswitch.conf\n#\n# An ex"..., 4096) = 1688
read(3, "", 4096) = 0
close(3) = 0
munmap(0xb7801000, 4096) = 0
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=18383, ...}) = 0
mmap2(NULL, 18383, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb77fd000
close(3) = 0
open("/lib/libnss_files.so.2", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\0\32\0\0004\0\0\0"...,
512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=58532, ...}) = 0
mmap2(NULL, 53956, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3,
0) = 0x332000
mmap2(0x33e000, 8192, PROT_READ|PROT_WRITE,
MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xb) = 0x33e000
close(3) = 0
mprotect(0x33e000, 4096, PROT_READ) = 0
munmap(0xb77fd000, 18383) = 0
open("/etc/host.conf", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=17, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7801000
read(3, "order hosts,bind\n", 4096) = 17
read(3, "", 4096) = 0
close(3) = 0
munmap(0xb7801000, 4096) = 0
open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
fcntl64(3, F_GETFD) = 0x1 (flags FD_CLOEXEC)
fstat64(3, {st_mode=S_IFREG|0644, st_size=44, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7801000
read(3, "127.0.0.1 localhost localhost."..., 4096) = 44
close(3) = 0
munmap(0xb7801000, 4096) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(2000),
sin_addr=inet_addr("127.0.0.1")}, 16) = 0
read(3, "ok\r\n", 1024) = 4
write(3, "login root\r\n", 12) = 12
read(3, "ok\r\n", 1024) = 4
write(3, "call 20\r\n", 9) = 9
read(3, "45769\r\n", 1024) = 7
write(3, "exit\r\n", 6) = 6
read(3, "goodbye\r\n", 1024) = 9
read(3, "", 1024) = 0
close(3) = 0
open("/etc/hosts", O_RDONLY|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=44, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7801000
read(3, "127.0.0.1 localhost localhost."..., 4096) = 44
close(3) = 0
munmap(0xb7801000, 4096) = 0
socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(45769),
sin_addr=inet_addr("127.0.0.1")}, 16) = 0
read(3, "ok\r\n", 1024) = 4
write(3, "login root\r\n", 12) = 12
read(3, "ok\r\n", 1024) = 4
write(3, "call 20\r\n", 9) = 9
read(3, "[attached]\r\n", 1024) = 12
fcntl64(3, F_SETOWN, 8657) = 0
rt_sigaction(SIGCHLD, {0x8049220, [], 0}, NULL, 8) = 0
write(3, "\5c=", 3) = 3
read(3, "[", 1024) = 1
read(3, "up]\r\n", 1024) = 5
write(3, "\5c\326", 3) = 3
read(3, "[", 1024) = 1
read(3, "8001018]\r\n", 1024) = 10
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 1), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1,
0) = 0xb7801000
write(1, "[Enter `^Ec?' for help]\n", 24[Enter `^Ec?' for help]
) = 24
write(3, "\5cm", 3) = 3
read(3, "[", 1024) = 1
read(3, "-- MOTD --]\r\n", 1024) = 13
write(3, "\5c;", 3) = 3
read(3, "[", 1024) = 1
read(3, "connected]\r\n", 1024) = 12
ioctl(0, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfd83868) = -1 EINVAL
(Invalid argument)
fcntl64(3, F_GETFL) = 0x2 (flags O_RDWR)
fcntl64(3, F_SETFL, O_RDWR|O_NONBLOCK) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
<Strace stops here until the sleep expires>
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0
select(4, [0 3], [], NULL, NULL) = 1 (in [0])
read(0, "", 8192) = 0

_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Console taking 100% CPU [ In reply to ]
On Tue, 2010-12-14 at 17:15 -0500, Chris Marget wrote:

>
> I've added two lines here. Seems to do what I need. Am I on the right track?
>
> static int screwy = 0;
> <snip>
> /* anything from stdin? */
> if (FD_ISSET(0, &rmask)) {
> if ((nc = read(0, acMesg, sizeof(acMesg))) <= 0) {
> if ( nc == 0 ) fprintf(stderr, "gotcha!\n"); // added by chris m
> if ( nc == 0 ) break; // added by chris m
> if (screwy)
> break;
> else {
> FD_SET(0, &rinit);
> continue;
> }
> }
>

Does the break terminate the program?

_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Console taking 100% CPU [ In reply to ]
I dug back into that code to see where it came from, etc. There was a
desire to be able to pipe console commands/interaction to the console
client and the code ignores EOF in that case. So, stuff like "echo
'^[.cr^[.c.' | console host" can process the input, and then also send out
the output without bailing prematurely. Why? Well, in this "batch"
case, you don't want the console client to exit before getting back the
output from the server - and there's no way to tell when the data will
be "done".

I should probably fix it to not chew up all the CPU...but it would still
"hang". If you can wrap this thing in a pty, the code will detect it as
such and actually close things down on EOF. Or adjust the 'screwy'
(yeah, pleasant name) variable in console.c so it is always 1 - pretends
all connections are pty-based.

Bryan

On Dec 14, 2010, at 1:59 PM, Chris Fowler wrote:

> On Tue, 2010-12-14 at 16:54 -0500, Chris Marget wrote:
>> read(0, "", 8192) = 0
>> select(4, [0 3], [], NULL, NULL) = 1 (in [0])
>> read(0, "", 8192) = 0
>> select(4, [0 3], [], NULL, NULL) = 1 (in [0])
>> read(0, "", 8192) = 0
>
> select() has seen STDIN ready to be read.
> read() reads 0 bytes. This is an EOF condition.
>
> Easy. Fix the code so that when reading from 0 if 0 bytes are read is
> restores the terminal and exits.
>
>
> _______________________________________________
> users mailing list
> users@conserver.com
> https://www.conserver.com/mailman/listinfo/users
_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Console taking 100% CPU [ In reply to ]
On Tue, 2010-12-14 at 22:31 +0000, Bryan Stansell wrote:
>
> I should probably fix it to not chew up all the CPU...but it would
> still "hang". If you can wrap this thing in a pty, the code will
> detect it as such and actually close things down on EOF. Or adjust
> the 'screwy' (yeah, pleasant name) variable in console.c so it is
> always 1 - pretends all connections are pty-based.
>
>
The pty solution would be a good one and would allow the console code to
remain pristine.

In "APUE" by Richard Stevens (deceased) there is an example on how to
do this.

_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users
Re: Console taking 100% CPU [ In reply to ]
On Tue, Dec 14, 2010 at 10:31:02PM +0000, Bryan Stansell wrote:
> I should probably fix it to not chew up all the CPU...

Found the problem...sorry folks.

--- console.c 19 Oct 2009 06:44:06 -0000 5.185
+++ console.c 14 Dec 2010 23:52:50 -0000
@@ -1352,7 +1352,7 @@
if (screwy)
break;
else {
- FD_SET(0, &rinit);
+ FD_CLR(0, &rinit);
continue;
}
}

Bryan
_______________________________________________
users mailing list
users@conserver.com
https://www.conserver.com/mailman/listinfo/users