Mailing List Archive: Stats test server running, please check it out

Stats test server running, please check it out

webmaster at hartwork

Aug 10, 2009, 3:08 AM

Post #1 of 19 (1610 views)

Hello again!

I have set up a test server of the current stats code. If you have a
minute to check it out that would rock. I'm very interested in overall
feedback and bug reports.

To check it out please do as following:

0) Make sure you have these packages installed:
sys-apps/portage
dev-util/git
dev-python/rhpl
dev-python/urlgrabber
dev-python/simplejson
dev-python/dbus-python

1) Check out the code
# git clone git://git.goodpoint.de/smolt-gentoo.git
# cd smolt-gentoo/client
# git checkout --track -b gentoo origin/gentoo

2) Verify my server is running at the very moment by browsing to
http://smolt.hartwork.org:45678/static/stats/gentoo.html

3) Run the client (which asks and shows details before submission)
# python sendProfile.py \
--server=http://smolt.hartwork.org:45678/

Please note that a summary update is triggered manually so gentoo.html
will not update instantly. You can ping me on Freenode though, to make
me trigger an update earlier.

To disable certain classes of data for privacy reasons you can put this
in ~/.smolt/client.cfg

[gentoo]
arch_related = True
compile_flags = True
features = True
global_use_flags = True
installed_packages = True
installed_packages_use_flags = True
mirrors_sync = True
mirrors_distfiles = True
package_mask = True
repositories = True
system_profile = True

and turn "True" to "False" where necessary.

Have fun and please share your experience with it.

Sebastian

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 10, 2009, 3:57 AM

Post #2 of 19 (1587 views)

Sebastian Pipping wrote:
> 0) Make sure you have these packages installed:
> sys-apps/portage
> dev-util/git
> dev-python/rhpl
> dev-python/urlgrabber
> dev-python/simplejson
> dev-python/dbus-python

From what I hear not everyone has the HAL daemon installed.
HAL is required, too:

sys-apps/hal

Sebastian

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 10, 2009, 1:46 PM

Post #3 of 19 (1585 views)

Sebastian Pipping wrote:
> 3) Run the client (which asks and shows details before submission)
> # python sendProfile.py \
> --server=http://smolt.hartwork.org:45678/

I forgot to mention you need to create a random machine id one way or
another before you can submit data. An easy and transparent way to do
that would be running this command:

# sudo sh -c 'cat /proc/sys/kernel/random/uuid > /etc/smolt/hw-uuid'

The smolt ebuild is doing the very same.

We have data from 8 machines so far, you can be number 9.
http://smolt.hartwork.org:45678/static/stats/gentoo.html

Sebastian

Re: Stats test server running, please check it out [ In reply to ]

couldbe at soon

Aug 10, 2009, 2:11 PM

Post #4 of 19 (1598 views)

Sebastian Pipping <webmaster <at> hartwork.org> writes:

>
> Hello again!
>
> I have set up a test server of the current stats code. If you have a
> minute to check it out that would rock. I'm very interested in overall
> feedback and bug reports.
>
> To check it out please do as following:
>
> 0) Make sure you have these packages installed:
> sys-apps/portage
> dev-util/git
> dev-python/rhpl
> dev-python/urlgrabber
> dev-python/simplejson
> dev-python/dbus-python
>

emerge rhpl -va

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild N ] net-wireless/wireless-tools-29 USE="nls -multicall" 288 kB
[ebuild N ] dev-python/rhpl-0.213 236 kB

Total: 2 packages (2 new), Size of downloads: 523 kB

Would you like to merge these packages? [Yes/No]

is this correct?

Re: Stats test server running, please check it out [ In reply to ]

dirkjan at ochtman

Aug 10, 2009, 2:16 PM

Post #5 of 19 (1586 views)

On Mon, Aug 10, 2009 at 12:08, Sebastian Pipping<webmaster@hartwork.org> wrote:
> 0) Â Make sure you have these packages installed:
> Â Â Â Â dev-python/rhpl
> Â Â Â Â dev-python/urlgrabber
> Â Â Â Â dev-python/dbus-python

What do you need these for?

Cheers,

Dirkjan

Re: Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 10, 2009, 2:18 PM

Post #6 of 19 (1586 views)

Mark Bateman wrote:
> emerge rhpl -va
>
> These are the packages that would be merged, in order:
>
> Calculating dependencies... done!
> [ebuild N ] net-wireless/wireless-tools-29 USE="nls -multicall" 288 kB
> [ebuild N ] dev-python/rhpl-0.213 236 kB
>
> Total: 2 packages (2 new), Size of downloads: 523 kB
>
> Would you like to merge these packages? [Yes/No]
>
>
>
>
> is this correct?

yes, thanks for asking.

(though my code neither uses that component nor do i know what it does)

sebastian

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 10, 2009, 3:46 PM

Post #7 of 19 (1591 views)

Dirkjan Ochtman wrote:
> On Mon, Aug 10, 2009 at 12:08, Sebastian Pipping<webmaster@hartwork.org> wrote:
>> 0) Make sure you have these packages installed:
>> dev-python/rhpl
>> dev-python/urlgrabber
>> dev-python/dbus-python
>
> What do you need these for?

a short and correct answer would be that smolt 1.2 already depends on
them. i cannot say much more, except that urlgrabber might be
replaceable. i don't use any of them in the gentoo-specific extensions.

sebastian

Re: Stats test server running, please check it out [ In reply to ]

mescalinum at gentoo

Aug 10, 2009, 6:44 PM

Post #8 of 19 (1591 views)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastian Pipping wrote:
> Hello again!
>
>
> I have set up a test server of the current stats code. If you have
> a minute to check it out that would rock. I'm very interested in
> overall feedback and bug reports.

1st: you could make an ebuild for it ;)

2nd: how to improve the output: you could make every data an
hyperlink: that would help understand better the contents
example:
Archs:
hyperlink arch name to wikipedia perhaps

CFLAGS
can link the flag name to GCC user guide relevant section, or to
http://en.gentoo-wiki.com/wiki/CFLAGS#<flagname>
same for other flags and MAKEOPTS

FEATURES
can link feature to make.conf man page:
http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
rendered with no anchors over variable/features, so you could do
better ^__^

USE flags
can link to
http://gentoo-portage.com/Search?search=&use=<useflagname> (now
appears down)

System profiles:
can link to profiles in cvs

Package atoms can link to packages.g.o or to gentoo-portage.com

those percentages are just funny:

33 (275.0 %) 58 (483.3 %) 25 (208.3 %)

Total 12 (100.0 %) [[ yeah, total is 100%, sometimes ]]

kde 3 (25.0 %) others 554 (4616.67 %) Total 1066 (8883.33 %)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkqAzOkACgkQV/B5axfzrPu9YwCfekbUmNJ0LfJ1PxqgAcCuFiD8
STYAn1xjdrMNmWusEV+EZlHPrL5Hr8gH
=ZcgR
-----END PGP SIGNATURE-----

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 11, 2009, 9:44 AM

Post #9 of 19 (1585 views)

Federico Ferri wrote:
> 1st: you could make an ebuild for it ;)

i just made one but it's not that useful yet, as the code is not
runnable from any location yet...

> 2nd: how to improve the output: you could make every data an
> hyperlink: that would help understand better the contents

actually there's a task for that open already:
http://soc.gentooexperimental.org/issues/show/51

i completely agree about it.

> FEATURES
> can link feature to make.conf man page:
> http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
> rendered with no anchors over variable/features, so you could do
> better ^__^

the output of neither

# man2html -r make.conf.5 > make.conf.5.html

nor

# groff -man -Thtml make.conf.5 > make.conf.5.html

make me really happy

do you know any alternative coming with more anchors and external CSS
support?

> those percentages are just funny:
>
> 33 (275.0 %) 58 (483.3 %) 25 (208.3 %)
>
> Total 12 (100.0 %) [[ yeah, total is 100%, sometimes ]]
>
> kde 3 (25.0 %) others 554 (4616.67 %) Total 1066 (8883.33 %)

most of these numbers show how much of something users have on average.
i agree this is not "user friendly". Task now over here:
http://soc.gentooexperimental.org/issues/show/54

sebastian

Re: Stats test server running, please check it out [ In reply to ]

mescalinum at gentoo

Aug 11, 2009, 10:21 AM

Post #10 of 19 (1592 views)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sebastian Pipping wrote:
> Federico Ferri wrote:
>> FEATURES
>> can link feature to make.conf man page:
>> http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
>> rendered with no anchors over variable/features, so you could do
>> better ^__^
>
> the output of neither
>
> # man2html -r make.conf.5 > make.conf.5.html
>
> nor
>
> # groff -man -Thtml make.conf.5 > make.conf.5.html
>
> make me really happy
>
> do you know any alternative coming with more anchors and external CSS
> support?

# nope for css, but you could do something like this:
# imagine you have the feature list in a file
# (I don't know how to extract it automatically, but perhaps someone does)

$ cat featurelist
assume-digests
buildpkg
buildsyspkg
...
userpriv
usersandbox
usersync
webrsync-gpg

$ bzcat /usr/share/man/man5/make.conf.5.bz2 | man2html -r > make.html

# basically now you have to match <DT><B>featurename</B>
# and add a anchor tag to it:

$ eval `echo sed; cat featurelist | sed 's,^\(.*\),-e
'\''s:<DT><B>\1</B>:<DT><B><A NAME="feature_\1">\1</A></B>:'\'',';
echo make.html` > make_anchors.html

this will give you anchor to features, e.g.
make_anchor.html#feature_buildpkg

a bit tricky the sed - could be easier, but this version is fast
because it builds a giant sed expression instead of calling sed n times.

- --
Federico Ferri

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkqBqKwACgkQV/B5axfzrPsegwCdFo95i26IF9+jeUSLntVI4nhS
msgAnRw4I4D1MwPUf1yBY8gJh3PHtX9S
=cx8u
-----END PGP SIGNATURE-----

Re: Stats test server running, please check it out [ In reply to ]

fauli at gentoo

Aug 11, 2009, 12:57 PM

Post #11 of 19 (1588 views)

Hi,

Sebastian Pipping <webmaster@hartwork.org>:
> Have fun and please share your experience with it.

Works fine...really great work. This Summer of code seems to bring
out some results for Gentoo finally.

V-Li

--
Christian Faulhammer, Gentoo Lisp project
<URL:http://www.gentoo.org/proj/en/lisp/>, #gentoo-lisp on FreeNode

<URL:http://gentoo.faulhammer.org/>

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 11, 2009, 9:25 PM

Post #12 of 19 (1583 views)

Federico Ferri wrote:
> 1st: you could make an ebuild for it ;)

Done. Look for app-admin/gentoo-smolt-9999 in the "sping" overlay.

Running

# smoltSendProfile --server=http://smolt.hartwork.org:45678/

should work fine after.

Sebastian

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 12, 2009, 5:31 AM

Post #13 of 19 (1581 views)

Federico Ferri wrote:
> 2nd: how to improve the output: you could make every data an
> hyperlink: that would help understand better the contents
> example:
> Archs:
> hyperlink arch name to wikipedia perhaps

Done, linking to

http://packages.gentoo.org/arch/${arch}

> CFLAGS
> can link the flag name to GCC user guide relevant section, or to
> http://en.gentoo-wiki.com/wiki/CFLAGS#<flagname>
> same for other flags and MAKEOPTS

Later, server seems down.

> FEATURES
> can link feature to make.conf man page:
> http://linuxreviews.org/man/make.conf/ (unfortunately this man page is
> rendered with no anchors over variable/features, so you could do
> better ^__^

Done, linking to
http://smolt.hartwork.org:45678/static/man/man5/make.conf.5.html#${feature}

If anyone feels like playing with manpage-screen.css please do and
come back to me.

Also, the man page does not seem to list these features:

cvs
preserve-libs
split-debug
splitdebug
unmerge-logs

Either people use invalid features (you tell me) or the man page needs
an update. Maybe both ("split-debug" vs "splitdebug")?

> USE flags
> can link to
> http://gentoo-portage.com/Search?search=&use=<useflagname> (now
> appears down)

Still down, linking to

http://gpo.zugaina.org/Search?search=&use=${use}

for now.

> System profiles:
> can link to profiles in cvs

Are you sure about this? Will that be fun to look at?

> Package atoms can link to packages.g.o or to gentoo-portage.com

Done, linking to

http://packages.gentoo.org/package/${CATEGORY}/${PN}

To be able to link all packages properly we'd need a version of
http://gpo.zugaina.org/ that includes the trees "gentoo" and "funtoo".

ycarus, could the server handle that load-wise? would it be easy to
extend the current service or set up an all-packages-version in parallel?

Sebastian

Re: Stats test server running, please check it out [ In reply to ]

1i5t5.duncan at cox

Aug 12, 2009, 8:19 AM

Post #14 of 19 (1578 views)

Sebastian Pipping posted on Wed, 12 Aug 2009 14:31:48 +0200 as excerpted:

> Also, the man page does not seem to list these features:
>
> cvs
> preserve-libs
> split-debug
> splitdebug
> unmerge-logs
>
> Either people use invalid features (you tell me) or the man page needs
> an update. Maybe both ("split-debug" vs "splitdebug")?

Note that some FEATURES may in fact trigger various portage bashrc
customizations. While Ed Catmur disappeared some time ago now, I still
use some of his bashrc customizations, including FEATURES=patchtree, for
instance. There were others, but many of them were more dev focused than
I as a user am, so it's likely that various devs still use a possibly
further customized version of the same scripts, today.

The point being, due to customization it's not reasonable to expect a
full mapping, FEATURES or other variable, tho at least some of the above
would appear to be legitimate mainline portage if perhaps not yet well
documented FEATURES.

--
Duncan - List replies preferred. No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master." Richard Stallman

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 12, 2009, 11:54 AM

Post #15 of 19 (1580 views)

Yannick Chabanois wrote:
> Not really a problem, I can had gentoo and funtoo trees in
> gpo.zugaina.org. This will be available very soon ( next week ?) with
> the new version of the site.
> All source code of gpo.zugaina.org will be made available in the same time.

That's great news!

Please keep me up to date about this.
And thank you for such a great service.

Sebastian

Re: Stats test server running, please check it out [ In reply to ]

Aug 12, 2009, 1:55 PM

Post #16 of 19 (1578 views)

On Wednesday 12 August 2009, Sebastian Pipping wrote:
> cvs

Missing documentation, but valid, see make.conf.example

> split-debug

This is probably invalid.

> preserve-libs
> splitdebug
> unmerge-logs

These are documented in Portage 2.2 make.conf(5).

Robert

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 12, 2009, 3:22 PM

Post #17 of 19 (1578 views)

Robert Buchholz wrote:
>> preserve-libs
>> splitdebug
>> unmerge-logs
>
> These are documented in Portage 2.2 make.conf(5).

Just updated to make.conf from revision 13844.
http://smolt.hartwork.org:45678/static/man/man5/make.conf.5.html

My "man2tidyhtml" wrapper around manServer [1] is now hosted here [2].

An ebuild for manServer is in the pipeline, currently waiting for the
next reply from upstream. An ebuild for man2tidyhtml will follow.

Sebastian

[1] http://www.squarebox.co.uk/users/rolf/download/manServer.shtml
[2] http://git.goodpoint.de/?p=man2tidyhtml.git;a=summary

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 13, 2009, 7:30 AM

Post #18 of 19 (1579 views)

Sebastian Pipping wrote:
> An ebuild for manServer is in the pipeline, currently waiting for the
> next reply from upstream.

manServer ebuild here, new 1.08 release from upstream
http://git.goodpoint.de/?p=overlay-sping.git;a=tree;f=app-text/manserver

Sebastian

Re: Stats test server running, please check it out [ In reply to ]

webmaster at hartwork

Aug 14, 2009, 6:49 PM

Post #19 of 19 (1578 views)

Sebastian Pipping wrote:
> An ebuild for man2tidyhtml will follow.

Here it is:
http://git.goodpoint.de/?p=overlay-sping.git;a=tree;f=app-text/man2tidyhtml

Sebastian