Mailing List Archive

Using Trac as a public bug report interface
Hi,

I am thinking about trying Trac for a big opensource project with lots
of users.

We are currently a bugtracking tool based on a cost sharing idea, and
since we have got ridden of that method in our developement, it is no
longer relevant. As we are using SVN, and since I'm a fervent Trac
supporter at work, I think Trac is a good idea.

However, the problem is that our userbase is about 100K users.
Wouldn't permissions management be a little complex with such a user
base ? Would anyone have some experience in such scaling with Trac ?

Thanks for your comments / remarks / answers / whatever.

--
Best regards,
Katana mailto:katana@katana-inc.com
Using Trac as a public bug report interface [ In reply to ]
Hi,

I am thinking about trying Trac for a big opensource project with lots
of users.

We are currently a bugtracking tool based on a cost sharing idea, and
since we have got ridden of that method in our developement, it is no
longer relevant. As we are using SVN, and since I'm a fervent Trac
supporter at work, I think Trac is a good idea.

However, the problem is that our userbase is about 100K users.
Wouldn't permissions management be a little complex with such a user
base ? Would anyone have some experience in such scaling with Trac ?

Thanks for your comments / remarks / answers / whatever.

--
Best regards,
Katana mailto:katana@katana-inc.com
Using Trac as a public bug report interface [ In reply to ]
>Wouldn't permissions management be a little complex with such a user
>base ? Would anyone have some experience in such scaling with Trac ?
>
>Thanks for your comments / remarks / answers / whatever.
>
>
We use Trac with a potential base of 3K users.

I changed a bit the way Trac authenticates users:
Users are authenticated through LDAP.
User permissions are retrieved from LDAP groups (objectclass: groupOfNames)
Trac permission table contains permission definitions for both single
user and for groups.
I've decided to use the @ prefix to distinguish users from groups.

For example:

...
anonymous CHANGESET_VIEW
@administrators TRAC_ADMIN
@betatesters WIKI_CREATE
...

Note that the small patch I wrote works nice for our user base, but it
should definitely be optimized to give good performances with a larger
base like yours.
One of the problem is that Trac authenticates user on every single
request (I'm not sure if it's still true w/ ModPython, I do not know how
it works).

HTH,
Manu.
Using Trac as a public bug report interface [ In reply to ]
Hello Emmanuel,

> Users are authenticated through LDAP.
I was also thinking about moving the users base to LDAP, so it's a
good start :)

> User permissions are retrieved from LDAP groups (objectclass:
> groupOfNames)
> Trac permission table contains permission definitions for both
> single user and for groups.
[...]
I like your method; easy to read, and theorically easy to
administrate.

> Note that the small patch I wrote works nice for our user base, but
> it should definitely be optimized to give good performances with a
> larger base like yours.
Where can your patch be found ?

> One of the problem is that Trac authenticates user on every single
> request (I'm not sure if it's still true w/ ModPython, I do not know
> how it works).
Could anyone with a possible answer give us a hint there ? I would
appreciate.

Thanks (again) in advance,

--
Best regards,
Katana mailto:katana@katana-inc.com
Using Trac as a public bug report interface [ In reply to ]
Katana wrote:

> Hello Emmanuel,
>
>
>>Users are authenticated through LDAP.
>

I am also interested in this patch to use LDAP to give permission to
users/groups

Regards
--
--
********************************************************************
* *
* Bas van der Vlies e-mail: basv@sara.nl *
* SARA - Academic Computing Services phone: +31 20 592 8012 *
* Kruislaan 415 fax: +31 20 6683167 *
* 1098 SJ Amsterdam *
* *
********************************************************************
Using Trac as a public bug report interface [ In reply to ]
Hi,

I wrote an email yesterday about how you could get the patch, but it
never got distributed to the ML (I think Firefox got an issue), and I do
not have a copy.

Please wait for a couple of days, so I can get rid of my specific server
code, and I will release it (when I wrote the LDAP patch, I did not
though about distributing it, so it is part of our Python Ldap
framework, which is not directly related to Trac)

Manu.
Using Trac as a public bug report interface [ In reply to ]
>>Note that the small patch I wrote works nice for our user base, but
>>it should definitely be optimized to give good performances with a
>>larger base like yours.
>>
>>
>Where can your patch be found ?
>
>
Here it is 8')

Modified files:
* scripts/trac-admin
not directly related to LDAP support, add Wiki default page path selection
* scripts/tracd
modified version to support LDAP authentication. It's mainly for
demonstration purposes.
Be careful: I did not took time to write a secure authentication: LDAP
authentication here is based on 'Basic' HTTP authentication. In other
words, user password is sent to the server with no encryption: password
is Base64 encoded, that is, plain text. You should not use this over an
insecure network, etc. 8')
On my server, I do not use tracd, but Apache2 with mod_ldap and trac.
I've updated tracd if you want to give a try with LDAP without the
Apache2 / Ldap setup overhead.
* trac/core.py
I wrote LDAP authentication so that it can easily adapted to an
existing LDAP installation. Therefore, all LDAP parameters are
optionally defined in the trac.ini config file. In order to acces .ini
file, perm.py needs to get a reference on the 'Environment' instance.
I've changed core.py so that it instanciates a PermissionCache instance
with the current environment
* trac/db_default.py
I've added the default LDAP value to this file, so that trad-admin
creates a new trac.ini file with default LDAP parameters. You need to
edit trac.ini file so that it matches your LDAP server configuration
* trac/ldapgrp.py
Finally, the LDAP authentication class. It performs authentication (it
binds the LDAP connection with the supplied user/password of the remote
user), and retrieves the list of groups the user belongs to.
* trac/perm.py
PermissionCache has been modified so that it optionally support LDAP
(if and only if ldap_auth is set to 'true' in trac.ini). If LDAP is
enabled, PermissionCache retrieved the LDAP configuration parameters,
and instanciates a Ldap connection with the server

NOTES:
* Patch is based on Trac revision r915 (three days old). It is a
deliberate choice. It won't work with official Trac release (0.7.1), as
perm.py has been slighty changed.
* Please note that I'm quite a newbie with Python, so there are probably
a lot of improvements to be done. Security should be improved to (in
order to use challenge mechanism instead of plain text password, etc.) I
do not really care on this point since all access is done through HTTPS
on our server, and LDAP server is on the same host. If you care about
security issues, you should definitely fix up this code.
* I kept the '@' sign to distinguish real username from group name.
Since 0.7.1, Trac developers have introduced groupnames (which are not
different from real username). I prefer to keep the usual syntax for
groups; however, if you want to follow Trac rules, it should not be
difficult to remove the '@' stuff

Please let me know if this patch has been useful. If you have a trouble
using it, I can help you.

Tested with Linux 2.4.22, Open LDAP 2.1.30

Best Regards,
Emmanuel.




-------------- next part --------------
Index: scripts/trac-admin
===================================================================
--- scripts/trac-admin (revision 915)
+++ scripts/trac-admin (working copy)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/python
# -*- coding: iso8859-1 -*-
__author__ = 'Daniel Lundin <daniel@edgewall.com>, Jonas Borgstr?m <jonas@edgewall.com>'
__copyright__ = 'Copyright (c) 2004 Edgewall Software'
@@ -427,7 +427,7 @@

## Initenv
_help_initenv = [.('initenv', 'Create and initialize a new environment interactively'),
- ('initenv <projectname> <repospath> <templatepath>',
+ ('initenv <projectname> <repospath> <templatepath> <wikipath>',
'Create and initialize a new environment from arguments')]

def do_initdb(self, line):
@@ -460,6 +460,13 @@
dt = trac.siteconfig.__default_templates_dir__
prompt = 'Templates directory [%s]> ' % dt
returnvals.append(raw_input(prompt) or dt)
+ print
+ print ' Please enter location of Trac initial wiki pages.'
+ print ' Default is the location of the site-wide wiki pages installed with Trac.'
+ print
+ dw = trac.siteconfig.__default_wiki_dir__
+ prompt = 'Wiki pages directory [%s]> ' % dw
+ returnvals.append(raw_input(prompt) or dw)
return returnvals

def do_initenv(self, line):
@@ -470,18 +477,21 @@
project_name = None
repository_dir = None
templates_dir = None
+ wiki_dir = None
if len(arg) == 1:
returnvals = self.get_initenv_args()
project_name = returnvals[0]
repository_dir = returnvals[1]
templates_dir = returnvals[2]
- elif len(arg)!= 3:
+ wiki_dir = returnvals[3]
+ elif len(arg)!= 4:
print 'Wrong number of arguments to initenv %d' % len(arg)
return
else:
project_name = arg[0]
repository_dir = arg[1]
templates_dir = arg[2]
+ wiki_dir = arg[3]
from svn import util, repos, core
core.apr_initialize()
pool = core.svn_pool_create(None)
@@ -498,6 +508,9 @@
or not os.access(os.path.join(templates_dir, 'ticket.cs'), os.F_OK):
print templates_dir, "doesn't look like a Trac templates directory"
return
+ if not os.access(os.path.join(wiki_dir, 'WikiStart'), os.F_OK):
+ print wiki_dir, "doesn't seem to contain initial Wiki pages"
+ return
try:
print 'Creating and Initializing Project'
self.env_create()
@@ -516,7 +529,7 @@
# Add a few default wiki pages
print ' Installing wiki pages'
cursor = cnx.cursor()
- self._do_wiki_load(trac.siteconfig.__default_wiki_dir__,cursor)
+ self._do_wiki_load(wiki_dir,cursor)

print ' Indexing repository'
sync.sync(cnx, rep, fs_ptr, pool)
@@ -697,6 +710,7 @@
self._do_wiki_export(p, dst)

def _do_wiki_load(self, dir,cursor=None, ignore=[]):
+ print "Wiki load dir %s\n" % dir
for page in os.listdir(dir):
if page in ignore:
continue
Index: scripts/tracd
===================================================================
--- scripts/tracd (revision 915)
+++ scripts/tracd (working copy)
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/python
# -*- coding: iso8859-1 -*-
#
# Copyright (C) 2003, 2004 Edgewall Software
@@ -35,6 +35,7 @@
import urllib
import urllib2
import mimetypes
+import base64
import SocketServer
import BaseHTTPServer

@@ -42,6 +43,7 @@

import trac.core
import trac.util
+import trac.ldapgrp
from trac import Session
from trac.Href import Href
from trac import auth, siteconfig
@@ -129,7 +131,40 @@
self.active_nonces.remove(auth['nonce'])
return auth['username']

+class LdapAuth:
+ def __init__(self, hostport, basedn, realm):
+ host, port = hostport.split(':', 2)
+ self.host = host
+ if None != port:
+ self.port = int(port)
+ else:
+ self.port = 389
+ self.realm = realm
+ self.basedn = basedn

+ def send_auth_request(self, req, stale='false'):
+ req.send_response(401)
+ req.send_header('WWW-Authenticate',
+ 'Basic realm="%s"' % (self.realm))
+ req.end_headers()
+
+ def do_auth(self, req):
+ if not 'Authorization' in req.headers or \
+ req.headers['Authorization'][:5] != 'Basic':
+ self.send_auth_request(req)
+ return None
+ auth64 = urllib2.parse_http_list(req.headers['Authorization'][6:])[0]
+ auth = base64.decodestring(auth64)
+ username, passwd = auth.split(':', 2)
+ lc = trac.ldapgrp.Connection(host=self.host, port=self.port, basedn=self.basedn)
+ rc = lc.open(username,passwd)
+ lc.close()
+ if not rc:
+ self.send_auth_request(req)
+ return None
+ return username
+
+
class TracHTTPServer(SocketServer.ThreadingMixIn, BaseHTTPServer.HTTPServer):
pass

@@ -260,6 +295,7 @@
print 'usage: %s [options] <projenv> [projenv] ...' % sys.argv[0]
print '\nOptions:\n'
print '-a --auth [project],[htdigest_file],[realm]'
+ print '-l --ldapauth [project],[ldaphost:port],[realm],[basedn]'
print '-p --port [port]\t\tPort number to use (default: 80)'
print '-b --hostname [hostname]\tIP to bind to (default: \'\')'
print
@@ -272,8 +308,8 @@
auths = {}
projects = {}
try:
- opts, args = getopt.getopt(sys.argv[1:], "a:p:b:",
- ["auth=", "port=", "hostname="])
+ opts, args = getopt.getopt(sys.argv[1:], "a:l:p:b",
+ ["auth=", "ldapauth=", "port=", "hostname="])
except getopt.GetoptError:
usage()
for o, a in opts:
@@ -283,6 +319,12 @@
usage()
p, h, r = info
auths[p] = DigestAuth(h, r)
+ if o in ("-l", "--ldapauth"):
+ info = a.split(',', 3)
+ if len(info) != 4:
+ usage()
+ p, h, r, b = info
+ auths[p] = LdapAuth(h,b,r)
if o in ("-p", "--port"):
port = int(a)
elif o in ("-b", "--hostname"):
Index: trac/core.py
===================================================================
--- trac/core.py (revision 915)
+++ trac/core.py (working copy)
@@ -162,7 +162,7 @@
module.req = req
module._name = mode
module.db = db
- module.perm = perm.PermissionCache(module.db, req.authname)
+ module.perm = perm.PermissionCache(module.db, module.env, req.authname)
module.perm.add_to_hdf(req.hdf)
module.authzperm = None

Index: trac/db_default.py
===================================================================
--- trac/db_default.py (revision 915)
+++ trac/db_default.py (working copy)
@@ -445,5 +445,13 @@
('notification', 'smtp_from', 'trac@localhost'),
('notification', 'smtp_replyto', 'trac@localhost'),
('timeline', 'changeset_show_files', 'false'),
- ('timeline', 'changeset_files_count', 3))
+ ('timeline', 'changeset_files_count', 3),
+ ('ldapauth', 'ldap_auth', 'false'),
+ ('ldapauth', 'ldap_host', 'localhost'),
+ ('ldapauth', 'ldap_port', 389),
+ ('ldapauth', 'ldap_basedn', 'dc=example,dc=com'),
+ ('ldapauth', 'ldap_groupname', 'groupofnames'),
+ ('ldapauth', 'ldap_groupuid', 'cn'),
+ ('ldapauth', 'ldap_groupmember', 'member'),
+ ('ldapauth', 'ldap_uid', 'uid'))

Index: trac/ldapgrp.py
===================================================================
--- trac/ldapgrp.py (revision 0)
+++ trac/ldapgrp.py (revision 0)
@@ -0,0 +1,98 @@
+# -*- coding: iso8859-1 -*-
+#
+# Copyright (C) 2003, 2004 Edgewall Software
+# Copyright (C) 2003, 2004 Jonas Borgstr??m <jonas@edgewall.com>
+#
+# Trac is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation; either version 2 of the
+# License, or (at your option) any later version.
+#
+# Trac is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.
+#
+# Author: Emmanuel Blot <emmanuel.blot@free.fr>
+
+import ldap
+import re
+
+class Connection:
+ def __init__(self, **ldap):
+ self.host = 'localhost'
+ self.port = 389
+ self.basedn = ''
+ self.groupname = 'groupofnames'
+ self.groupuid = 'cn'
+ self.groupmember = 'member'
+ self.uid = 'uid'
+ if ldap.has_key('host'): self.host = ldap['host']
+ if ldap.has_key('port'): self.port = ldap['port']
+ if ldap.has_key('basedn'): self.basedn = ldap['basedn']
+ if ldap.has_key('groupname'): self.groupname = ldap['groupname']
+ if ldap.has_key('groupuid'): self.groupuid = ldap['groupuid']
+ if ldap.has_key('groupmember'): self.groupmember = ldap['groupmember']
+ if ldap.has_key('uid'): self.uid = ldap['uid']
+
+ def basedn(self):
+ return self.basedn;
+
+ def open(self, uid=None, password=None):
+ try:
+ self.__ds = ldap.open(self.host, self.port)
+ self.__ds.protocol_version = ldap.VERSION3
+ if uid is not None:
+ if ( not uid.startswith('%s=' % self.uid) ):
+ uid = '%s=%s' % (self.uid, uid)
+ self.__ds.simple_bind_s(uid + ',' + self.basedn, password)
+ self.__login = uid
+ return True
+ except ldap.LDAPError, e:
+ return False
+
+ def close(self):
+ self.__ds.unbind_s()
+ self.__login = ''
+
+ def isOwner(self, uid):
+ return self.__login == uid
+
+ def search(self, filter, attributes=None):
+ try:
+ sr = self.__ds.search_s(self.basedn, ldap.SCOPE_SUBTREE, filter, attributes)
+ return sr
+
+ except ldap.LDAPError, e:
+ return False;
+
+ def compare(self, dn, attribute, value):
+ try:
+ cr = self.__ds.compare_s(dn + "," + self.basedn, attribute, value)
+ return cr
+
+ except ldap.LDAPError, e:
+ return False
+
+ def enumerate_groups(self):
+ attributes = ['dn']
+ sr = self.search('objectclass=' + self.groupname, attributes)
+ groups = ['none']
+ if sr:
+ for (dn, attrs) in sr:
+ regex = re.compile('^(\w+)=(\w+)')
+ m = regex.search(dn)
+ if m:
+ groups.append(m.group(2))
+ return groups
+
+ def is_in_group(self, uid, group):
+ dn = self.groupuid + "=" + group
+ value = self.uid + "=" + uid + "," + self.basedn
+ cr = self.compare(dn, self.groupmember, value)
+ return cr
+
Index: trac/perm.py
===================================================================
--- trac/perm.py (revision 915)
+++ trac/perm.py (working copy)
@@ -104,7 +104,7 @@
'authenticated': Permissions granted to this user will apply to
any authenticated (logged in with HTTP_AUTH) user.
"""
- def __init__(self, db, username):
+ def __init__(self, db, env, username):
self.perm_cache = {}
cursor = db.cursor()
cursor.execute ("SELECT username, action FROM permission")
@@ -114,6 +114,26 @@
users = ['anonymous']
if username != 'anonymous':
users += [username, 'authenticated']
+
+ useldap = False
+ if env.get_config('ldapauth', 'ldap_auth', '') == 'true':
+ useldap = True
+ import ldapgrp
+ if useldap and (username[0] !='@'):
+ lc = ldapgrp.Connection(host=env.get_config('ldapauth', 'ldap_host', 'localhost'),
+ port=int(env.get_config('ldapauth', 'ldap_port', '389')),
+ basedn=env.get_config('ldapauth', 'ldap_basedn', ''),
+ groupname=env.get_config('ldapauth', 'ldap_groupname', 'groupofnames'),
+ groupuid=env.get_config('ldapauth', 'ldap_groupuid', 'cn'),
+ groupmember=env.get_config('ldapauth', 'ldap_groupmember', 'member'),
+ uid=env.get_config('ldapauth', 'ldap_uid', 'uid'))
+ if lc.open():
+ groups = lc.enumerate_groups()
+ for group in groups:
+ if lc.is_in_group(username, group):
+ users.append('@' + group)
+ lc.close()
+
while 1:
num_users = len(users)
num_perms = len(perms)