Mailing List Archive

Ken Robbins wrote:
> Hi my first gentoo performance came today but it way only a header no body what up with that?
>
<snip>

Mine too and I've been listening for a while. I think this ML may be dead ...
*pokes the darkness*
--
gentoo-performance@gentoo.org mailing list

funny that a "high performance linux" has a dead performance ML... LOL


lnxg33k wrote:

> Ken Robbins wrote:
>
>> Hi my first gentoo performance came today but it way only a header no
>> body what up with that?
>
> <snip>
>
> Mine too and I've been listening for a while. I think this ML may be
> dead ... *pokes the darkness*

--
gentoo-performance@gentoo.org mailing list

Chris wrote:
> funny that a "high performance linux" has a dead performance ML... LOL
<snip>

Could be evidence that the "ricer" crowd doesn't read? (i.e. they post to more
generic lists or use other mediums instead of something specific for their needs)
--
gentoo-performance@gentoo.org mailing list

Hi!

On Fri, Jan 20, 2006 at 03:30:29AM +0100, Chris wrote:
> > Mine too and I've been listening for a while. I think this ML may be
> > dead ... *pokes the darkness*
> funny that a "high performance linux" has a dead performance ML... LOL

It isn't dead because a lot of attentive listeners subscribed to it... :-)

--
WBR, Alex.
--
gentoo-performance@gentoo.org mailing list

lnxg33k wrote:
> Chris wrote:
>
>> funny that a "high performance linux" has a dead performance ML... LOL
>
> <snip>
>
> Could be evidence that the "ricer" crowd doesn't read? (i.e. they post
> to more generic lists or use other mediums instead of something specific
> for their needs)

Time to throw in a performance info post.

Has anybody done any tests on the different between the one core and
dual core opteron processors? I currently have an opteron 242 and a gig
of ram on a tyan mobo, and I'm curious as to which would allow me to
compile programs faster, as I lend the systems out to a lot of groups
that have slow systems

Kyle
--
gentoo-performance@gentoo.org mailing list

lnxg33k wrote:

> Chris wrote:
>
>> funny that a "high performance linux" has a dead performance ML... LOL
>
> <snip>
>
> Could be evidence that the "ricer" crowd doesn't read? (i.e. they post
> to more generic lists or use other mediums instead of something
> specific for their needs)

Since we're all here and saying hello.. Someone have a performance
question or a good tip to add to the list?

The one thing that first pops to my head is some hdparm results I've had
and if maybe it's either my kernel setup or how I'm testing with hdparm...

Anyhow..

Raptors on SATA (Model Number: WDC WD360GD-00FNA0) (-c3 -u1 -A1
-W1 -d1 -a256 -M254 -m16 -X70 )
Kernel config
CONFIG_SCSI_SATA_SIL=y

There's an alternative driver, but haven't tested it...

hdparm -tT /dev/hde

/dev/hde:
Timing cached reads: 956 MB in 2.00 seconds = 477.02 MB/sec
Timing buffered disk reads: 116 MB in 3.03 seconds = 38.26 MB/sec

hdparm -tT --direct /dev/hde

/dev/hde:
Timing O_DIRECT cached reads: 244 MB in 2.01 seconds = 121.26 MB/sec
Timing O_DIRECT disk reads: 124 MB in 3.01 seconds = 41.17 MB/sec

Laptop 7k60 (Model Number: HTS726060M9AT00)
hdparm -tT /dev/hdc

/dev/hdc:
Timing cached reads: 1980 MB in 2.00 seconds = 989.94 MB/sec
Timing buffered disk reads: 118 MB in 3.04 seconds = 38.81 MB/sec


hdparm -tT --direct /dev/hdc

/dev/hdc:
Timing O_DIRECT cached reads: 364 MB in 2.02 seconds = 180.54 MB/sec
Timing O_DIRECT disk reads: 120 MB in 3.04 seconds = 39.42 MB/sec


I also can't set the raptors into UDMA6 tried -X70 with no luck..

Any suggestions?

Thanks

C.
--
gentoo-performance@gentoo.org mailing list

How about speeding up the wait time on updating the portage cache after
a sync.. even on my AMD 64 3500 it takes a number of minutes to chug
through..
are there any known ways to "vrrmmm" this up a little?

Jeremy
--
gentoo-performance@gentoo.org mailing list

Jeremy Brake wrote:
> How about speeding up the wait time on updating the portage cache after
> a sync.. even on my AMD 64 3500 it takes a number of minutes to chug
> through..
> are there any known ways to "vrrmmm" this up a little?
>
> Jeremy

Although not exactly what you're asking, you might want to look at
RSYNC_EXCLUDEFORM. Cuts down on what is checked during rsync. I assume that
this could also cut down on the cache update time since there would be less to
check? Also cuts down on the amount of space eaten up by portage.
--
gentoo-performance@gentoo.org mailing list

quoth the Jeremy Brake:
> How about speeding up the wait time on updating the portage cache after
> a sync.. even on my AMD 64 3500 it takes a number of minutes to chug
> through..
> are there any known ways to "vrrmmm" this up a little?

+1

Mine sped up for all of a day, but is slow as molasses once again. If I did my
syncs during the day it might even peeve me...

> Jeremy

-d
--
darren kirby :: Part of the problem since 1976 :: http://badcomputer.org
"...the number of UNIX installations has grown to 10, with more expected..."
- Dennis Ritchie and Ken Thompson, June 1972

lnxg33k wrote:

> Although not exactly what you're asking, you might want to look at
> RSYNC_EXCLUDEFORM. Cuts down on what is checked during rsync. I assume
> that this could also cut down on the cache update time since there
> would be less to check? Also cuts down on the amount of space eaten up
> by portage.


I'll second RSYNC exclusions. It really does speed up syncing especially
on headless servers which don't need x11-*/*.

--
Michael Liesenfelt
University of Florida
Innovative Nuclear Space Power and Propulsion Institute

darren kirby wrote:

quoth the Jeremy Brake:

How about speeding up the wait time on updating the portage cache after a sync.. even on my AMD 64 3500 it takes a number of minutes to chug through.. are there any known ways to "vrrmmm" this up a little?

+1 Mine sped up for all of a day, but is slow as molasses once again. If I did my syncs during the day it might even peeve me...

What kind of hardware are you guys running on? My laptop isn't on cron and I do it every couple days or so and it finishes in around 15-30 minutes.. I've never really paid any attention.. How long is yours taking?

What I am curious about is... what's it really doing when it says 51-52%.. That 1% seems to take forever..

C.
-- gentoo-performance@gentoo.org mailing list

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jeremy Brake wrote:
> How about speeding up the wait time on updating the portage cache after
> a sync.. even on my AMD 64 3500 it takes a number of minutes to chug
> through..
> are there any known ways to "vrrmmm" this up a little?
>
> Jeremy

Well the current problem is that in the 2.0 portage branch the cache
code sucks. This is fixed in ~arch portage ( the 2.1_pre series ). For
you users that don't want to upgrade to unstable, you should be able to
use cdb to speed up the process.

Setting RSYNC_EXCLUDES will not speed up the second half of the --sync (
the --metadata portion ).

Explanation: The Portage Tree has a ton of ebuilds in it, when you do
something like emerge -pv <blar> portage used to have to go read each
ebuild and be like "oh what are the USE flags, the dependencies, etc.."
This takes a lot of time, especially for something like emerge -uD world
that may touch thousands of packages.

So to mitigate this issue portage has a "metadata cache" where it stores
the ebuild metadata so it doesn't have to source each ebuild every time.
This gives performance increases ( reportedly ) of 100x speed-ups.

However, generating the metadata is time consuming, even on decent
hardware it will probably take 30-45 minutes. To not piss the users
off, Gentoo calculates this metadata serve-side and serves it to you
during sync.

The Rsync portion of emerge --sync is pretty much everything before the
"updating metadata cache". This should be pretty standard as far as
time goes on all boxes. However the second portion is where portage has
to take the generated server-side cache and kind of "meld" it into your
already existant local cache ( stored in /var/cache/edb/dep for those of
you whom are curious ). This didn't used to take a bunch of time..until
KDE split their packages from KDE-monolith to KDE-meta. The X.org
switch probably won't help much in this area either. Particularly the
slowdown at 50-52% is due to this portion of the tree.

There is a new cache system in the 2.1 branch of portage, or using cdb,
or perhaps even an sqlite back-end (patches and modules are out there),
will help until such time as the 2.1 branch is stablized ( probably not
too soon ).

- -Alec Warner (antarus)

Disclaimer: I'm not a developer (yet) but I've worked with the portage
team for some time. Release dates, features, and other things are not
gauranteed to be correct just because I said so :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQ9CExWzglR5RwbyYAQLJEhAAj7i2cRVgv1iBfi61mGkn9q1t/K5JEJcv
zK07bTmMDirviessdKiZVSufXdWV79tW0MDEVqmGI9t+V+uwM77aFbge1VSkEaQB
RWetuQL8tQRUX1KvQ+ItScVtbqKIAQe/Jn+BwSim05jF3+fF15Z6EpPPKypNLQxK
2ef/bcJ91Gctv0xcd6j943uOPFDCt05Ahe06/pQ0wgGdnAcKLOy/RVwDOVfprXim
AwiVsU4aCxRI056RkEj1VuSwSYQEa91WKpTGv81lZVkRW8LRxkgc7UAydMYGjOY5
1prjv2Koyly5ycVvxshKVmLfuaqByY9bBnlklyKDdFW1ZD1udCflCLxmz3GuTCsm
/FHce+Y9smqq3sF1wV7DYXu9vTnLdBqVBlDq4cEtd5XdgQm3TJ5rUGF2Mepicyhg
Bg4ibDExDB5eKWCnGU3ioSCd8TY9cdR9ZXxpm6JLGfr9ztog0/vScsIZbj3dS4RH
WqGklvW0F9N6NjP26WJwtQsmSIZpSJU4yPxneZOdQxGOUfdNPQap+qO+rZaitPKW
yWaikaiSuecPSul0ithpGnCFttPHrBLyNKNl7aom04Bht4KSdaqzriIL/RylWzHW
OgazUV9RuVdI8oEx+q3zKzOgvG3dXeL7HNdhH+j9noIyGVa8ouNHWMGfFUc1baxV
7eyB1buSeQY=
=vWqe
-----END PGP SIGNATURE-----
--
gentoo-performance@gentoo.org mailing list

Alec Warner wrote:

>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Jeremy Brake wrote:
>
>
>>How about speeding up the wait time on updating the portage cache after
>>a sync.. even on my AMD 64 3500 it takes a number of minutes to chug
>>through..
>>are there any known ways to "vrrmmm" this up a little?
>>
>>Jeremy
>>
>>
>
>Well the current problem is that in the 2.0 portage branch the cache
>code sucks. This is fixed in ~arch portage ( the 2.1_pre series ). For
>you users that don't want to upgrade to unstable, you should be able to
>use cdb to speed up the process.
>
>Setting RSYNC_EXCLUDES will not speed up the second half of the --sync (
>the --metadata portion ).
>
>Explanation: *snip*
>
>
Thanks Alec, thats a really awesome explaination :)

My server runs a 5am script which does this, so i'm not too worried
about that machine. For those who are curious, its an Athlon 1800+ on a
10Mbit link, and it takes between 1 and 10 mins to process " emerge
--sync --quiet; emerge -upvD world; glsa-check -t all "

My home pc is on a 2Mbit link, so I only sync when i feel like checking
for updates, or when I want to install something new. This will take
minimum of 10 mins just to update the cache most times, sometimes more.
Being a home pc, I'm happy to have some unstable stuff installed. How
messy would it be to just run ~amd64 portage? would this work, or do I
ideally need to make the entire base system ~amd64? (ugh).

Jeremy


--
gentoo-performance@gentoo.org mailing list

quoth the Christopher Bergström:
> darren kirby wrote:
> quoth the Jeremy Brake:
>
> How about speeding up the wait time on updating the portage cache after
> a sync.. even on my AMD 64 3500 it takes a number of minutes to chug
> through..
> are there any known ways to "vrrmmm" this up a little?
>
>
> +1
>
> Mine sped up for all of a day, but is slow as molasses once again. If I did
> my syncs during the day it might even peeve me...
>
> What kind of hardware are you guys running on?  My laptop isn't on cron
> and I do it every couple days or so and it finishes in around 15-30
> minutes.. I've never really paid any attention.. How long is yours taking?

It doesn't make a difference what hardware. I have 4 boxes that run gentoo
(Athlon 2200, Athlon 3200, Apple G4, Sparc U60) and they all run the actual
sync quite fast, but the 50%-51% takes from 5 minutes on the 3200 to 30
minutes on the G4 and U60.

As I mentioned though, I don't let this bother me as I generally sync ~4:00am
whilst sleeping.

> What I am curious about is... what's it really doing when it says 51-52%..
> That 1% seems to take forever..
>
> C.
> -- gentoo-performance@gentoo.org mailing list
-d
--
darren kirby :: Part of the problem since 1976 :: http://badcomputer.org
"...the number of UNIX installations has grown to 10, with more expected..."
- Dennis Ritchie and Ken Thompson, June 1972

>
> The one thing that first pops to my head is some hdparm results I've had
> and if maybe it's either my kernel setup or how I'm testing with hdparm...
>
> Anyhow..
>
> Raptors on SATA (Model Number: WDC WD360GD-00FNA0) (-c3 -u1 -A1
> -W1 -d1 -a256 -M254 -m16 -X70 )
> Kernel config
> CONFIG_SCSI_SATA_SIL=y
>
> There's an alternative driver, but haven't tested it...
>
> hdparm -tT /dev/hde
>
> /dev/hde:
> Timing cached reads: 956 MB in 2.00 seconds = 477.02 MB/sec
> Timing buffered disk reads: 116 MB in 3.03 seconds = 38.26 MB/sec
>
Your times for the raptor seem awfully slow.

Here the SATA settings for my kernel and my hdparm times.

antec ~ # grep SATA /usr/src/linux/.config
# CONFIG_BLK_DEV_IDE_SATA is not set
CONFIG_SCSI_SATA=y
CONFIG_SCSI_SATA_AHCI=y
# CONFIG_SCSI_SATA_SVW is not set
# CONFIG_SCSI_SATA_MV is not set
# CONFIG_SCSI_SATA_NV is not set
# CONFIG_SCSI_SATA_QSTOR is not set
# CONFIG_SCSI_SATA_PROMISE is not set
# CONFIG_SCSI_SATA_SX4 is not set
# CONFIG_SCSI_SATA_SIL is not set
# CONFIG_SCSI_SATA_SIL24 is not set
# CONFIG_SCSI_SATA_SIS is not set
# CONFIG_SCSI_SATA_ULI is not set
# CONFIG_SCSI_SATA_VIA is not set
# CONFIG_SCSI_SATA_VITESSE is not set
CONFIG_SCSI_SATA_INTEL_COMBINED=y
antec ~ # hdparm -tT /dev/md0

/dev/md0:
Timing cached reads: 2776 MB in 2.00 seconds = 1387.91 MB/sec
Timing buffered disk reads: 398 MB in 3.00 seconds = 132.48 MB/sec
antec ~ #

Note that this is a RAID0 with two disks, so the buffered disk read time
is double what you should expect on a normal install.

Good luck.

Bill Roberts

Bill Roberts wrote:

The one thing that first pops to my head is some hdparm results I've had and if maybe it's either my kernel setup or how I'm testing with hdparm... Anyhow.. Raptors on SATA (Model Number: WDC WD360GD-00FNA0) (-c3 -u1 -A1 -W1 -d1 -a256 -M254 -m16 -X70 ) Kernel config CONFIG_SCSI_SATA_SIL=y There's an alternative driver, but haven't tested it... hdparm -tT /dev/hde /dev/hde: Timing cached reads: 956 MB in 2.00 seconds = 477.02 MB/sec Timing buffered disk reads: 116 MB in 3.03 seconds = 38.26 MB/sec

Your times for the raptor seem awfully slow. Here the SATA settings for my kernel and my hdparm times. antec ~ # grep SATA /usr/src/linux/.config # CONFIG_BLK_DEV_IDE_SATA is not set CONFIG_SCSI_SATA=y CONFIG_SCSI_SATA_AHCI=y # CONFIG_SCSI_SATA_SVW is not set # CONFIG_SCSI_SATA_MV is not set # CONFIG_SCSI_SATA_NV is not set # CONFIG_SCSI_SATA_QSTOR is not set # CONFIG_SCSI_SATA_PROMISE is not set # CONFIG_SCSI_SATA_SX4 is not set # CONFIG_SCSI_SATA_SIL is not set # CONFIG_SCSI_SATA_SIL24 is not set # CONFIG_SCSI_SATA_SIS is not set # CONFIG_SCSI_SATA_ULI is not set # CONFIG_SCSI_SATA_VIA is not set # CONFIG_SCSI_SATA_VITESSE is not set CONFIG_SCSI_SATA_INTEL_COMBINED=y antec ~ # hdparm -tT /dev/md0 /dev/md0: Timing cached reads: 2776 MB in 2.00 seconds = 1387.91 MB/sec Timing buffered disk reads: 398 MB in 3.00 seconds = 132.48 MB/sec antec ~ # Note that this is a RAID0 with two disks, so the buffered disk read time is double what you should expect on a normal install.

I guess I should add that 2nd disk in there then.. Anyhow, I'm bound to what appears to be a not-so-friendly controller..
CONFIG_SCSI_SATA_SIL is not set I'm going to change the sata_sil.c file to enable UDMA6 and I've seen some other minor patches as well.. If it all proves stable.. Maybe someone will actually get it sent upstream. What's further.. I'm on hardened.. So I think you might have some kernel config options I don't have.. I have to look again..

Cheers,

C.
-- gentoo-performance@gentoo.org mailing list

Thanks Alec Warner for the great explanation. It still seems like by not having
portions of the tree by using EXCLUDEFORM and deleting the local dirs that
you'd save some time in the --metadata part of sync as less ebuilds are
available to be checked. Is this simply a wrong misconception?
--
gentoo-performance@gentoo.org mailing list

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

lnxg33k wrote:
> Thanks Alec Warner for the great explanation. It still seems like by not
> having portions of the tree by using EXCLUDEFORM and deleting the local
> dirs that you'd save some time in the --metadata part of sync as less
> ebuilds are available to be checked. Is this simply a wrong misconception?

The portion that "updates metadata cache" has nothing to do with what
ebuilds are in the tree. It simply takes the server-side caches (
pregenerated ) and sync them into your local cache. You could RSYNC
exclude the whole tree and this will still happen for every package,
since the metadata is generated on the server ( and the server has the
whole tree ).

Now if you were to rsync exclude metadata/ you would reduce the
- --metadata portion...because it wouldn't happen ;)

- -Alec Warner (antarus)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQ9GCJ2zglR5RwbyYAQLajw/+Mvfu9+0Sfo8nHo7gbNytHRNL1jVI61RD
/EIiZZwV067X77IP72o0Y7SICRvnooEqtLvQW+rd2c/Kb6W4EtDga94X8EbOrHIm
0/IFqg7OqUa/psU6IkaPk959u7UJTnqlWmzluLbqRTx/X03lpjk6n+V4BOTRhcTA
WHa80quSpd5EkfdFAJ1oWVMn9ck7xSTn3ulzmlznCkLdWK6iR+m+r+wAWMPlcRFG
xE/Ik5uMVemlWuAElhMoB4xr/2hKfcsu/Egw9F+zVL5+3mGMyygHhELAvTVHU/C9
jLX3noNFC+xSOesmC0e+l88Uyz2AYPuhg8S+3qciC+4Ax2QkDhhRxwM7lLF6yPBx
UpDNnTecT1iVuFUhHP08T2GPq8Nyvtzj4oqgjzKq1+l2RdehDM8j0KZrq5ZwDszb
CdKakVUrVXmMOEp16E48k5/66sulgJ5fNdVubJYNdPwsY2dNJ5MC7wbxSwIT46TD
88tYAgco4cH9o4whwL2KZIjCQodKgvNwCnvW3qeXOdD/WqRSvuVbFIZh/+YZMHXi
KVnWrePS2kwukMiL28oB9fwsJomQpwxCJlhZxuLto7kM1vBhlKLOeFfoZ8gm6mrF
gBkDwiyV7aNHL4tFgAGtvDPReQhRS4DcN61EVEOYxMxQIKh1lDSFR1UW+j538S9c
J/7XGJI9CxQ=
=Qike
-----END PGP SIGNATURE-----
--
gentoo-performance@gentoo.org mailing list

Alec Warner wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> lnxg33k wrote:
>> Thanks Alec Warner for the great explanation. It still seems like by not
>> having portions of the tree by using EXCLUDEFORM and deleting the local
>> dirs that you'd save some time in the --metadata part of sync as less
>> ebuilds are available to be checked. Is this simply a wrong misconception?
>
> The portion that "updates metadata cache" has nothing to do with what
> ebuilds are in the tree. It simply takes the server-side caches (
> pregenerated ) and sync them into your local cache. You could RSYNC
> exclude the whole tree and this will still happen for every package,
> since the metadata is generated on the server ( and the server has the
> whole tree ).
>
> Now if you were to rsync exclude metadata/ you would reduce the
> - --metadata portion...because it wouldn't happen ;)
>
<snip>

Ah, ok. Hearing it again makes it sound more reasonable. I'll have to watch my
rsync output a bit more closely. Thanks again for the explanation.
--
gentoo-performance@gentoo.org mailing list