--Fig2xvG2VGoz8o/s
Content-Type: multipart/mixed; boundary="NDin8bjvE/0mNLFQ"
Content-Disposition: inline
--NDin8bjvE/0mNLFQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
This may be the bug you're talking about... I'll check
through the cvs tree and see if it's plausible. -sc
--=20
Sean Chittenden
--NDin8bjvE/0mNLFQ
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: sean-freebsd-current@chittenden.org
Received: (qmail 48021 invoked from network); 18 Apr 2001 22:07:51 -0000
Received: from mx2.freebsd.org (216.136.204.119)
by perrin.tgd.net with SMTP; 18 Apr 2001 22:07:51 -0000
Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18])
by mx2.freebsd.org (Postfix) with ESMTP
id B9E6555F53; Wed, 18 Apr 2001 15:07:39 -0700 (PDT)
(envelope-from owner-freebsd-current@FreeBSD.ORG)
Received: by hub.freebsd.org (Postfix, from userid 538)
id 3831737B42C; Wed, 18 Apr 2001 15:07:38 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
by hub.freebsd.org (Postfix) with SMTP
id E64C82E8039; Wed, 18 Apr 2001 15:07:35 -0700 (PDT)
(envelope-from owner-freebsd-current)
Received: by hub.freebsd.org (bulk_mailer v1.12); Wed, 18 Apr 2001 15:07:35 -0700
Delivered-To: freebsd-current@freebsd.org
Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132])
by hub.freebsd.org (Postfix) with ESMTP
id BAF9737B422; Wed, 18 Apr 2001 15:07:26 -0700 (PDT)
(envelope-from tlambert@usr08.primenet.com)
Received: (from daemon@localhost)
by smtp02.primenet.com (8.9.3/8.9.3) id PAA21754;
Wed, 18 Apr 2001 15:00:12 -0700 (MST)
Received: from usr08.primenet.com(206.165.6.208)
via SMTP by smtp02.primenet.com, id smtpdAAA9WaqvQ; Wed Apr 18 15:00:01 2001
Received: (from tlambert@localhost)
by usr08.primenet.com (8.8.5/8.8.5) id PAA11716;
Wed, 18 Apr 2001 15:07:35 -0700 (MST)
From: Terry Lambert <tlambert@primenet.com>
Message-Id: <200104182207.PAA11716@usr08.primenet.com>
Subject: Found BAD BUG: squashed
To: arch@freebsd.org
Date: Wed, 18 Apr 2001 22:07:25 +0000 (GMT)
X-Mailer: ELM [version 2.5 PL2]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-freebsd-current@FreeBSD.ORG
X-Loop: FreeBSD.ORG
Precedence: bulk
I have identified the bug.
It turns out that the bug causes memory corruption by freeing
but continuing to use a credential, and it only occurs in big
resource usage cases, and then seemingly at random.
It is the fact that there are two credentials per socket, one
for the socket, and one for the descriptor itself.
The cr_ref is an unsigned short, and ...
There is no test for overflow.
Thus if I open 65537 files, and close 2, my cred structure is
freed.
If invariants are on, "0xdeadc0de" is spammed over top of the
reference count, and then each time the cread is released, it
is decremented: "0xdeadc0dd", "0xdeadc0dc", "0xdeadc0db", ...
...until another "0xc0de" (49374) closes, at which point it
frees the already freed credential _again_.
The way FreeBSD code is currently structured, this means that
the quick-and-ugly fix is simply to change this from u_short
to u_long. Further, I moved the cr_ref below the cr_groups
declartion -- well out of stomping range of the INVARIANTS
code. I have successfully opened 60,000 socket connections
(240,000 credential references between the client and server
processes combined, and 120,000 each) with this change.
--
The real fix is to do as I suggested on -arch earlier today
(and then some), and change:
crhold(curproc->p_ucred);
sigio->sio_ucred = curproc->p_ucred;
style things into:
sigio->sio_ucred = crhold(curproc->p_ucred);
style things.
But that is only part of it. Additionally, it needs to be
further changed to something like:
sigio->sio_ucred = crhold(&curproc->p_ucred);
-- passing the address of the credential pointer -- and then
(pseudocode), the credential code in crhold() needs to check
for an imminent overflow, and when it sees it:
struct ucred *
crhold(struct ucred **cpp)
{
struct ucred *ret;
if( about_to_overflow) {
ret = *cpp;
*cpp = crdup(save);
} else {
ret = *cpp;
ret ->cr_ref++;
}
return ret;
}
The point of doing this would be to give the FD the overflowing
cred, so that new files opened by the process can start counting
at 1 again.
NOTE: I also suggest chagning free() in kern/kern_malloc.c in
free() in the INVARIANTS case, from:
---
#ifdef INVARIANTS
/*
* Check for multiple frees. Use a quick check to see if
* it looks free before laboriously searching the freelist.
*/
if (freep->spare0 == WEIRD_ADDR) {
fp = (struct freelist *)kbp->kb_next;
while (fp) {
if (fp->spare0 != WEIRD_ADDR)
panic("free: free item %p modified", fp);
else if (addr == (caddr_t)fp)
panic("free: multiple freed item %p", addr);
fp = (struct freelist *)fp->next;
}
}
---
#ifdef INVARIANTS
/*
* Check for multiple frees. We can't use a quick check, since
* that will give us a false negative for something like a
* stomped reference counter or mutex, if it is the first item.
*/
fp = (struct freelist *)kbp->kb_next;
while (fp) {
if (fp->spare0 != WEIRD_ADDR)
panic("free: free item %p modified", fp);
else if (addr == (caddr_t)fp)
panic("free: multiple freed item %p", addr);
fp = (struct freelist *)fp->next;
}
---
Terry Lambert
terry@lambert.org
---
Any opinions in this posting are my own and not those of my present
or previous employers.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
--NDin8bjvE/0mNLFQ--
--Fig2xvG2VGoz8o/s
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Comment: Sean Chittenden <sean@chittenden.org>
iEYEARECAAYFAjreFykACgkQn09c7x7d+q1TSACeIZqoExBVLfu3aNz1icGW+MxG
L3YAnjsdPQ70tL4URpeKYEEhVNHPqb6W
=a4iz
-----END PGP SIGNATURE-----
--Fig2xvG2VGoz8o/s--
Content-Type: multipart/mixed; boundary="NDin8bjvE/0mNLFQ"
Content-Disposition: inline
--NDin8bjvE/0mNLFQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
This may be the bug you're talking about... I'll check
through the cvs tree and see if it's plausible. -sc
--=20
Sean Chittenden
--NDin8bjvE/0mNLFQ
Content-Type: message/rfc822
Content-Disposition: inline
Return-Path: <owner-freebsd-current@FreeBSD.ORG>
Delivered-To: sean-freebsd-current@chittenden.org
Received: (qmail 48021 invoked from network); 18 Apr 2001 22:07:51 -0000
Received: from mx2.freebsd.org (216.136.204.119)
by perrin.tgd.net with SMTP; 18 Apr 2001 22:07:51 -0000
Received: from hub.freebsd.org (hub.freebsd.org [216.136.204.18])
by mx2.freebsd.org (Postfix) with ESMTP
id B9E6555F53; Wed, 18 Apr 2001 15:07:39 -0700 (PDT)
(envelope-from owner-freebsd-current@FreeBSD.ORG)
Received: by hub.freebsd.org (Postfix, from userid 538)
id 3831737B42C; Wed, 18 Apr 2001 15:07:38 -0700 (PDT)
Received: from localhost (localhost [127.0.0.1])
by hub.freebsd.org (Postfix) with SMTP
id E64C82E8039; Wed, 18 Apr 2001 15:07:35 -0700 (PDT)
(envelope-from owner-freebsd-current)
Received: by hub.freebsd.org (bulk_mailer v1.12); Wed, 18 Apr 2001 15:07:35 -0700
Delivered-To: freebsd-current@freebsd.org
Received: from smtp02.primenet.com (smtp02.primenet.com [206.165.6.132])
by hub.freebsd.org (Postfix) with ESMTP
id BAF9737B422; Wed, 18 Apr 2001 15:07:26 -0700 (PDT)
(envelope-from tlambert@usr08.primenet.com)
Received: (from daemon@localhost)
by smtp02.primenet.com (8.9.3/8.9.3) id PAA21754;
Wed, 18 Apr 2001 15:00:12 -0700 (MST)
Received: from usr08.primenet.com(206.165.6.208)
via SMTP by smtp02.primenet.com, id smtpdAAA9WaqvQ; Wed Apr 18 15:00:01 2001
Received: (from tlambert@localhost)
by usr08.primenet.com (8.8.5/8.8.5) id PAA11716;
Wed, 18 Apr 2001 15:07:35 -0700 (MST)
From: Terry Lambert <tlambert@primenet.com>
Message-Id: <200104182207.PAA11716@usr08.primenet.com>
Subject: Found BAD BUG: squashed
To: arch@freebsd.org
Date: Wed, 18 Apr 2001 22:07:25 +0000 (GMT)
X-Mailer: ELM [version 2.5 PL2]
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Sender: owner-freebsd-current@FreeBSD.ORG
X-Loop: FreeBSD.ORG
Precedence: bulk
I have identified the bug.
It turns out that the bug causes memory corruption by freeing
but continuing to use a credential, and it only occurs in big
resource usage cases, and then seemingly at random.
It is the fact that there are two credentials per socket, one
for the socket, and one for the descriptor itself.
The cr_ref is an unsigned short, and ...
There is no test for overflow.
Thus if I open 65537 files, and close 2, my cred structure is
freed.
If invariants are on, "0xdeadc0de" is spammed over top of the
reference count, and then each time the cread is released, it
is decremented: "0xdeadc0dd", "0xdeadc0dc", "0xdeadc0db", ...
...until another "0xc0de" (49374) closes, at which point it
frees the already freed credential _again_.
The way FreeBSD code is currently structured, this means that
the quick-and-ugly fix is simply to change this from u_short
to u_long. Further, I moved the cr_ref below the cr_groups
declartion -- well out of stomping range of the INVARIANTS
code. I have successfully opened 60,000 socket connections
(240,000 credential references between the client and server
processes combined, and 120,000 each) with this change.
--
The real fix is to do as I suggested on -arch earlier today
(and then some), and change:
crhold(curproc->p_ucred);
sigio->sio_ucred = curproc->p_ucred;
style things into:
sigio->sio_ucred = crhold(curproc->p_ucred);
style things.
But that is only part of it. Additionally, it needs to be
further changed to something like:
sigio->sio_ucred = crhold(&curproc->p_ucred);
-- passing the address of the credential pointer -- and then
(pseudocode), the credential code in crhold() needs to check
for an imminent overflow, and when it sees it:
struct ucred *
crhold(struct ucred **cpp)
{
struct ucred *ret;
if( about_to_overflow) {
ret = *cpp;
*cpp = crdup(save);
} else {
ret = *cpp;
ret ->cr_ref++;
}
return ret;
}
The point of doing this would be to give the FD the overflowing
cred, so that new files opened by the process can start counting
at 1 again.
NOTE: I also suggest chagning free() in kern/kern_malloc.c in
free() in the INVARIANTS case, from:
---
#ifdef INVARIANTS
/*
* Check for multiple frees. Use a quick check to see if
* it looks free before laboriously searching the freelist.
*/
if (freep->spare0 == WEIRD_ADDR) {
fp = (struct freelist *)kbp->kb_next;
while (fp) {
if (fp->spare0 != WEIRD_ADDR)
panic("free: free item %p modified", fp);
else if (addr == (caddr_t)fp)
panic("free: multiple freed item %p", addr);
fp = (struct freelist *)fp->next;
}
}
---
#ifdef INVARIANTS
/*
* Check for multiple frees. We can't use a quick check, since
* that will give us a false negative for something like a
* stomped reference counter or mutex, if it is the first item.
*/
fp = (struct freelist *)kbp->kb_next;
while (fp) {
if (fp->spare0 != WEIRD_ADDR)
panic("free: free item %p modified", fp);
else if (addr == (caddr_t)fp)
panic("free: multiple freed item %p", addr);
fp = (struct freelist *)fp->next;
}
---
Terry Lambert
terry@lambert.org
---
Any opinions in this posting are my own and not those of my present
or previous employers.
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-current" in the body of the message
--NDin8bjvE/0mNLFQ--
--Fig2xvG2VGoz8o/s
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Comment: Sean Chittenden <sean@chittenden.org>
iEYEARECAAYFAjreFykACgkQn09c7x7d+q1TSACeIZqoExBVLfu3aNz1icGW+MxG
L3YAnjsdPQ70tL4URpeKYEEhVNHPqb6W
=a4iz
-----END PGP SIGNATURE-----
--Fig2xvG2VGoz8o/s--