Mailing List Archive

Aaron Stone writes:

for the sake of the ideas, this is getting too far and at the end it will be
too bloated. making everyone happy is not the most important thing here, i'm
happy with what i have and i'd change it the way i like it to suit me, be
that code/conf/whatever. what one code need is a good framework around which
we can hack.. and not to waste time on 'Oh, wtf is this'...

I suppose you understand what I'm saying, sometimes a bit of abstraction
can reduce code and efforts, but too much can cause panic ..

dont get lost in the loop guys.

if my ideas and suggestions will lead to more complexity, please drop them.
i'd prefer to have dbmail the way it is now, than having it bloated with
those stuff below..

c

> This does indeed give everyone what the want wrt config files, but also
> introduces a lot of complexity. That's no necessarily bad, so long as the
> simple cases are handled simply.
>
> So, if you have a current config file, with everything in it, and no database
> config table... works fine.
>
> And, if you have a database only config file, with just the minimum needed to
> "bootstrap" into the database config table... works fine.
>
> Then, for crazy people, crazy things are possible if you're paying attention
> to the order of inclusions... complicated, but possible.
>
> Unless I left anything out, it sounds to me like this satisfies the "simple
> things are simple, complicated things are possible" rule of UNIX Philosophy.
>
> I can already hear Ilja groaning about this one, so if this is the code we
> want, someone's gonna have to do it ;-)
>
> There was one more issue, btw, which was the table names. I don't think that
> fully configurable names are particularly necessary, but one place where we
> might want to look into something like that is the users table. For people who
> might already have a pam_mysql or an apache mod_auth_mysql database, it would
> probably be nice for them to have dbmail use that existing database rather
> than migrating the users into dbmail and reconfiguring everything else. Dunno
> how to do this, just yet, although we should take a look at what pam_mysql and
> mod_auth_mysql use in their config files.
>
> Aaron
>
>
> ""Jesse Norell"" <jesse@kci.net> said:
>
>>
>> An idea that may work to suit everyone's preference of the config
>> layout: define specific section names for config items, like [SMTP],
>> and add that field to Aaron's table example below, and add an
>> [INCLUDE "/path/to/filename"] function to the config file parser.
>>
>> Make the config reader get the db parameters from the config file
>> (also add an (optional) item to the config file to disable
>> database config lookups completely for those who don't use them),
>> read in all the default values from the db, then override with any
>> host specific values from the db, then read through the local config
>> file and override settings with anything found there.
>>
>> Everyone could have the setup they wish - a huge dbmail.conf file,
>> or a minimal one with mostly db config settings, or even break the
>> local config into multiple files (/etc/dbmail/{dbmail,logging,host}.conf)
>> or whatever else you could think of.
>>
>>
>>
>> ---- Original Message ----
>> From: Eelco van Beek - IC&S <dbmail-dev@dbmail.org>
>> To: dbmail-dev@dbmail.org
>> Subject: Re: [Dbmail-dev] Re: table prefix
>> Sent: Thu, 8 Jan 2004 10:34:04 +0100
>>
>> > We could also do both (config on file and in the db) and keep them in
>> > sync. This enables easier administration through interfaces on the DB.
>> >
>> >
>> > On 8-jan-04, at 9:44, Mark Mackay - Orcon wrote:
>> >
>> > > I really like having a local config file [.much better than the old
>> > > days of
>> > > having to compile the database name in, etc]; and I like being
>> able to
>> > > just
>> > > 'vi' a file to enable logging, adjust process levels on one of the
>> > > hosts,
>> > > etc.
>> > >
>> > > If you want to do database configs as well, then the syntax below
>> > > would work
>> > > -- but rather than via hostname, I'd recommend making it some form of
>> > > 'clientid/configid' which could be supplied on command lines (e.g.
>> > > ../dbmail-pop3d -configkey DEFAULT) That would allow configs to be
>> > > experimented with and changed between without having to affect other
>> > > daemons
>> > > on the same host, etc.
>> > >
>> > > Personally I just say keep the config files :)
>> > >
>> > > Cheers,
>> > > Mark.
>> > >
>> > >> dbmail.conf:
>> > >> Is a configuration file in /etc or wherever.
>> > >> - database information
>> > >> - logging info
>> > >> - hostname / clustering info
>> > >>
>> > >> dbmail_conf:
>> > >> Is a table in the database.
>> > >> - Has detailed config info in host/key/value pairs:
>> > >>
>> > >> CREATE TABLE dbmail_conf (
>> > >> hostname VARCHAR(128) NOT NULL,
>> > >> key VARCHAR(32) NOT NULL,
>> > >> value TEXT NULL,
>> > >> UNIQUE INDEX (hostname, key)
>> > >> )
>> > >
>> > > _______________________________________________
>> > > Dbmail-dev mailing list
>> > > Dbmail-dev@dbmail.org
>> > > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>> > >
>> > _________________________
>> > E.J.A. van Beek
>> > ICT Manager
>> > IC&S
>> > T: +31 30 6355730
>> > F: +31 30 6355731
>> >
>> > PGP-key:
>> > www.ic-s.nl/keys/eelco.txt
>> >
>> > _______________________________________________
>> > Dbmail-dev mailing list
>> > Dbmail-dev@dbmail.org
>> > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>> >
>> -- End Original Message --
>>
>>

Mark Mackay - Orcon writes:


mark, i think you're over complicating the thing with this
pop3/imap proxy.. hack pbsp(extend it), make your radius write there the
clients IP when they log, and their user?ip, then do your select ..

you have so much data for a user.
idnr
name
clientid

and then pbsp
ipnumber
+name
timestamp..

you're already using xxx.isp, at the end of the day success is if you dont
have to make your clients change their usernames, just for that..

one optional field more wont be bad.. but why?

you dont need proxy to achive that, this is trivial.

hope that helps.

c

>> 1. configurable prefix. I guess the default should be an empty prefix,
>> to default to the current table names.
>> 2. do tablenames also need to configurable like lou@0xffff.org
>> mentioned in a previous message? If queries get table names from
>> variables, it shouldn't be so hard to make the whole table name
>> configurable.
>>
>> Please shoot at my arguments. I think we need some strict
>> requirements
>> on this before Brian gets to work ;)
>
> I don't think all tablenames need to be configurable , or certainly the ones
> are 'dbmail-specific' [but prefixable may be nice]; but one that may be
> worth considering is the users table. I've said this before, many months
> ago, so forgive my repeat ramblings:
>
> Most of us have a database of all our customers, for billing, radius, etc.
> This to me it make sense to allow dbmail to be able to query our existing
> user databases to ultimately find the dbmail "user_idnr" that we need to
> handle delivery/checking of mail. Thus it may be worth making the "users"
> table totally configurable, so you can hook into any existing table that
> exists.
>
> The real power with above, would be to allow the magic dbmail-specific
> columns to be customisable (so if you already had a "uid" column which
> mimicked the passwd file, you could use that column rather than having to
> add a "user_idnr" column to your database, etc.
>
> Even better -- allow custom queries from that table, to allow greater
> flexibility. There are probably only a handful of queries which actually
> interact with the users' table, so maybe these could be definable [.but not
> required to be defined in a config, eg. default if not present].
>
> Reason: We do wholesale ISP services in NZ and also buy a lot of ISPs, which
> results in the need to support multiple "mark" usernames on our mailserver
> (people over here don't log in with mark@ispa.co.nz and mark@ispb.co.nz --
> usually just 'mark'). Dbmail because it's in theory all numeric post-auth is
> great for this -- I can have mark from ispa with user_idnr 1000 and mark
> from ispb with user_idnr 2000. So all I need to do now is make sure I can
> look up the right user_idnr when 'mark' connects.
>
> Currently we're having to proxy the pop3/imap streams and rewrite the
> username so when someone goes "USER mark" the dbmail server actually gets
> "USER mark.ispa" or similar. We have daemons bound to different IP addresses
> and ports with different configs to do this.
>
> What would be great, is to allow dbmail to do this job for us -- so I don't
> have another intermediate step which doesn't need to be here. I'd love
> dbmail to be able to do the following:
>
> select user_idnr, password from users where userid='mark' and
> SOMEOTHERCOLUMN='VALUE'
>
> This SOMEOTHERCOLUMN=VALUE could be an ip address (host='10.0.0.1') based on
> the IP address of the incoming connection, or perhaps some variable that is
> per-daemon defined (e.g. start ./dbmail-pop3d with some config that says
> ISPID="DEFAULT" or ISPID="ISPB").
>
> Now I know not everyone wants this (or do you? -- there must be people in a
> similar boat to me out there!). Having just the users-related queries
> customisable, and allowing macros to be used such as
>
> %u for username
> %p for password
> %h for incoming hostip
> %{customvariableinconfig}
>
> Would allow a very customisable solution for those people that needed it.
> Then people could easily build up configs which queried their existing
> user/billing databases.
>
> My 2c.
>
> /Mark
>
>
>
>
>
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev

Hi lou@0xffff.org -

Thanks for your comments, but I don't know if I explained myself right,
although I'd love alternate ideas for dealing with our complicated needs:

To start off with we have a database full of users, including say "mark".

Mark connects through to pop3.ispa.co.nz and is used to supplying "mark"
as their username.

If we wholesale to another isp or buy an ISP with "mark" in their system, we
want to be able to allow them to connect seamlessly. [.At the moment we
import them into our central database as say "mark.ispb" but in the near
future a rewrite will allow them to be imported as simply "mark" with a
different account id].

When it was just 'mark' (no second ISP), we simply had:

"mark" <-> dbmail daemons [pop3/imap] [logs into db as "mark"] <->
dbmail-database

When we add the second ISP we currently proxy-rewrite things:

"mark" (from ISPA]
<-> dbmail daemons [pop3/imap] (listening on 10.0.0.1)
<-> dbmail-database
[logs into db as "mark"]

"mark" (from ISPB]
<-> simple pop3/imap proxy (listening on 10.0.0.2)
[rewrites pop stream to say "USER mark.ispb" when user enters "USER
mark"]
<-> dbmail-database
[logs into db as "mark.ispb"]

Thus we point pop3.ispa.co.nz to 10.0.0.1 and pop3.isbp.co.nz to 10.0.0.2.
This means "mark" from ISPB doesn't notice anything, and can keep using
"mark" as their username.

I don't quite understand the radius/pbsp idea you had below. An in any case
if a user is roaming (currently in the US and wants to check their mail), it
wouldn't work I'm guessing.

-------------

It would be great to be able to let Dbmail handle the above -- a very simply
idea would be to have two user tables. One for ISPA and one for ISPB I
guess, and then simply configure the daemons bound to 10.0.0.2 to select
from users_ISPB.

In the end it all just ends up with a user_idnr, so I guess I am looking at
ways that could be made more simple -- whilst allowing other people to do
'fancy' things as well. For example, there was discussion ages ago about how
to deal with suspended accounts, etc.

-------------

I didn't understand your suggestion so if you're willing to explain it to me
on or off list that would be appreciated, and I'd love to hear from anyone
else out there doing similar things as to what ideas they use in their
system.

Also -- what are people doing when they have their own, independent user
database for radius/billing? Are people just running one big database
server and using the dbmail 'users' table with other columns, are they
copying data to and from, updating both databases from the web-apps?

About 6-12 months ago the 'user' table was separated into a different
db-connection, which meant it could be on a different server, and thus be
plugged in with a few hacks of the SQL strings to existing databases. This
was discarded at one point (don't think it was ever posted to the list why).
Maybe this is a good idea for a 2.x release feature now that bigger
installations are starting to use dbmail.

/Mark



> mark, i think you're over complicating the thing with this
> pop3/imap proxy.. hack pbsp(extend it), make your radius
> write there the
> clients IP when they log, and their user?ip, then do your select ..
>
> you have so much data for a user.
> idnr
> name
> clientid
>
> and then pbsp
> ipnumber
> +name
> timestamp..
>
> you're already using xxx.isp, at the end of the day success
> is if you dont
> have to make your clients change their usernames, just for that..
>
> one optional field more wont be bad.. but why?
>
> you dont need proxy to achive that, this is trivial.

Mark Mackay - Orcon writes:

> Hi lou@0xffff.org -
>
> Thanks for your comments, but I don't know if I explained myself right,
> although I'd love alternate ideas for dealing with our complicated needs:
>
> To start off with we have a database full of users, including say "mark".
>
> Mark connects through to pop3.ispa.co.nz and is used to supplying "mark"
> as their username.
>
> If we wholesale to another isp or buy an ISP with "mark" in their system, we
> want to be able to allow them to connect seamlessly. [.At the moment we
> import them into our central database as say "mark.ispb" but in the near
> future a rewrite will allow them to be imported as simply "mark" with a
> different account id].

Well I dont think i understood you properly,i thought you're in control of
radius, also I assumed (a very bad thing though) that you're providing the
internet.

this way you know everything about user mark.

mark is connects through ISPA -> radius update pbsp (add his IP &
his user = mark.ispa)

he connects to dbmail pop3/imap ->
since you know his IP you can simply look for mark.isp coming from that ip,
and then get his idnr.

the whole effort is to change a few sql queries and some code..
and if that's true, that you control the whole isp and your mail clients are
your internet clients, you can even get rid of the . suffix and have 10000
marks usernames..

but if that's not applicable discard it

>
> When it was just 'mark' (no second ISP), we simply had:
>
> "mark" <-> dbmail daemons [pop3/imap] [logs into db as "mark"] <->
> dbmail-database
>
> When we add the second ISP we currently proxy-rewrite things:
>
> "mark" (from ISPA]
> <-> dbmail daemons [pop3/imap] (listening on 10.0.0.1)
> <-> dbmail-database
> [logs into db as "mark"]
>
> "mark" (from ISPB]
> <-> simple pop3/imap proxy (listening on 10.0.0.2)
> [rewrites pop stream to say "USER mark.ispb" when user enters "USER
> mark"]
> <-> dbmail-database
> [logs into db as "mark.ispb"]
>
> Thus we point pop3.ispa.co.nz to 10.0.0.1 and pop3.isbp.co.nz to 10.0.0.2.
> This means "mark" from ISPB doesn't notice anything, and can keep using
> "mark" as their username.

well that's is the same, except that in my example i dont introduce 3d party
proxy. dont forget that dbmail knows the clients IP, therefor you can fetch
his original idnr off pbps. (that's the extended pbsp, not the way you see
it) or if you want dont add anything to pbsp, just use the clientid (if
you're not using it) and add the pbsp idnr of your client's IP i.e.

client Joe Shmoe
username; joe
idnr; 1
clientid = pbsp_idnr = 100
[...]

client Foo
username; joe
idnr; 2
clientid = pbsp_idnr = 101

pbsp
idnr; 100 ; 101
since; timestamp; timestampt
ipnumber; 1.1.1.1; 1.1.1.2
status; static
[...]


get his address from pbsp,
select idnr from pbsp where ipnumber=%ip and status='static';
select username, password from users where user=%user and
clientid=pbsp_idnr;


somewhere here in the middle the whole thing will work, i'd not have time to
rethink the whole implementation, but trust me it's easy.

> I don't quite understand the radius/pbsp idea you had below. An in any case
> if a user is roaming (currently in the US and wants to check their mail), it
> wouldn't work I'm guessing.

no i guess, will yours? or you track your users with gps?;-) j/k

at my work (AFAIK) we do something simple.
pop3 server/smtp.
anyone can connect to the SMTP
for someone to connect to get your email 2 choices:
1 - https webmail
2 - vpn -> pop3


> -------------
>
> It would be great to be able to let Dbmail handle the above -- a very simply
> idea would be to have two user tables. One for ISPA and one for ISPB I
> guess, and then simply configure the daemons bound to 10.0.0.2 to select
> from users_ISPB.
>
> In the end it all just ends up with a user_idnr, so I guess I am looking at
> ways that could be made more simple -- whilst allowing other people to do
> 'fancy' things as well. For example, there was discussion ages ago about how
> to deal with suspended accounts, etc.
>
> -------------
>
> I didn't understand your suggestion so if you're willing to explain it to me
> on or off list that would be appreciated, and I'd love to hear from anyone
> else out there doing similar things as to what ideas they use in their
> system.

There's no much point bloating the list, we can have an offlist discussion
and then you/I can post the discussion.

> Also -- what are people doing when they have their own, independent user
> database for radius/billing? Are people just running one big database
> server and using the dbmail 'users' table with other columns, are they
> copying data to and from, updating both databases from the web-apps?
>
> About 6-12 months ago the 'user' table was separated into a different
> db-connection, which meant it could be on a different server, and thus be
> plugged in with a few hacks of the SQL strings to existing databases. This
> was discarded at one point (don't think it was ever posted to the list why).
> Maybe this is a good idea for a 2.x release feature now that bigger
> installations are starting to use dbmail.
>
> /Mark
>
>
>
>> mark, i think you're over complicating the thing with this
>> pop3/imap proxy.. hack pbsp(extend it), make your radius
>> write there the
>> clients IP when they log, and their user?ip, then do your select ..
>>
>> you have so much data for a user.
>> idnr
>> name
>> clientid
>>
>> and then pbsp
>> ipnumber
>> +name
>> timestamp..
>>
>> you're already using xxx.isp, at the end of the day success
>> is if you dont
>> have to make your clients change their usernames, just for that..
>>
>> one optional field more wont be bad.. but why?
>>
>> you dont need proxy to achive that, this is trivial.
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev

We might be headed off the deep end, but, IMHO, it's an important exercise to
see what's out there. There are certainly some large installations of DBMail
right now, and with 2.x, even larger ones should be expected. Many of these
larger installations will have reasonably complex environments already running
and it would be best if DBMail could adapt a bit to fit.

Aaron


lou@0xffff.org said:

> Aaron Stone writes:
>
> for the sake of the ideas, this is getting too far and at the end it will be
> too bloated. making everyone happy is not the most important thing here, i'm
> happy with what i have and i'd change it the way i like it to suit me, be
> that code/conf/whatever. what one code need is a good framework around which
> we can hack.. and not to waste time on 'Oh, wtf is this'...
>
> I suppose you understand what I'm saying, sometimes a bit of abstraction
> can reduce code and efforts, but too much can cause panic ..
>
> dont get lost in the loop guys.
>
> if my ideas and suggestions will lead to more complexity, please drop them.
> i'd prefer to have dbmail the way it is now, than having it bloated with
> those stuff below..
>
> c
>
> > This does indeed give everyone what the want wrt config files, but also
> > introduces a lot of complexity. That's no necessarily bad, so long as the
> > simple cases are handled simply.
> >
> > So, if you have a current config file, with everything in it, and no database
> > config table... works fine.
> >
> > And, if you have a database only config file, with just the minimum needed to
> > "bootstrap" into the database config table... works fine.
> >
> > Then, for crazy people, crazy things are possible if you're paying attention
> > to the order of inclusions... complicated, but possible.
> >
> > Unless I left anything out, it sounds to me like this satisfies the "simple
> > things are simple, complicated things are possible" rule of UNIX Philosophy.
> >
> > I can already hear Ilja groaning about this one, so if this is the code we
> > want, someone's gonna have to do it ;-)
> >
> > There was one more issue, btw, which was the table names. I don't think that
> > fully configurable names are particularly necessary, but one place where we
> > might want to look into something like that is the users table. For people who
> > might already have a pam_mysql or an apache mod_auth_mysql database, it would
> > probably be nice for them to have dbmail use that existing database rather
> > than migrating the users into dbmail and reconfiguring everything else. Dunno
> > how to do this, just yet, although we should take a look at what pam_mysql and
> > mod_auth_mysql use in their config files.
> >
> > Aaron
> >
> >
> > ""Jesse Norell"" <jesse@kci.net> said:
> >
> >>
> >> An idea that may work to suit everyone's preference of the config
> >> layout: define specific section names for config items, like [SMTP],
> >> and add that field to Aaron's table example below, and add an
> >> [INCLUDE "/path/to/filename"] function to the config file parser.
> >>
> >> Make the config reader get the db parameters from the config file
> >> (also add an (optional) item to the config file to disable
> >> database config lookups completely for those who don't use them),
> >> read in all the default values from the db, then override with any
> >> host specific values from the db, then read through the local config
> >> file and override settings with anything found there.
> >>
> >> Everyone could have the setup they wish - a huge dbmail.conf file,
> >> or a minimal one with mostly db config settings, or even break the
> >> local config into multiple files (/etc/dbmail/{dbmail,logging,host}.conf)
> >> or whatever else you could think of.
> >>
> >>
> >>
> >> ---- Original Message ----
> >> From: Eelco van Beek - IC&S <dbmail-dev@dbmail.org>
> >> To: dbmail-dev@dbmail.org
> >> Subject: Re: [Dbmail-dev] Re: table prefix
> >> Sent: Thu, 8 Jan 2004 10:34:04 +0100
> >>
> >> > We could also do both (config on file and in the db) and keep them in
> >> > sync. This enables easier administration through interfaces on the DB.
> >> >
> >> >
> >> > On 8-jan-04, at 9:44, Mark Mackay - Orcon wrote:
> >> >
> >> > > I really like having a local config file [.much better than the old
> >> > > days of
> >> > > having to compile the database name in, etc]; and I like being
> >> able to
> >> > > just
> >> > > 'vi' a file to enable logging, adjust process levels on one of the
> >> > > hosts,
> >> > > etc.
> >> > >
> >> > > If you want to do database configs as well, then the syntax below
> >> > > would work
> >> > > -- but rather than via hostname, I'd recommend making it some form of
> >> > > 'clientid/configid' which could be supplied on command lines (e.g.
> >> > > ../dbmail-pop3d -configkey DEFAULT) That would allow configs to be
> >> > > experimented with and changed between without having to affect other
> >> > > daemons
> >> > > on the same host, etc.
> >> > >
> >> > > Personally I just say keep the config files :)
> >> > >
> >> > > Cheers,
> >> > > Mark.
> >> > >
> >> > >> dbmail.conf:
> >> > >> Is a configuration file in /etc or wherever.
> >> > >> - database information
> >> > >> - logging info
> >> > >> - hostname / clustering info
> >> > >>
> >> > >> dbmail_conf:
> >> > >> Is a table in the database.
> >> > >> - Has detailed config info in host/key/value pairs:
> >> > >>
> >> > >> CREATE TABLE dbmail_conf (
> >> > >> hostname VARCHAR(128) NOT NULL,
> >> > >> key VARCHAR(32) NOT NULL,
> >> > >> value TEXT NULL,
> >> > >> UNIQUE INDEX (hostname, key)
> >> > >> )
> >> > >
> >> > > _______________________________________________
> >> > > Dbmail-dev mailing list
> >> > > Dbmail-dev@dbmail.org
> >> > > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
> >> > >
> >> > _________________________
> >> > E.J.A. van Beek
> >> > ICT Manager
> >> > IC&S
> >> > T: +31 30 6355730
> >> > F: +31 30 6355731
> >> >
> >> > PGP-key:
> >> > www.ic-s.nl/keys/eelco.txt
> >> >
> >> > _______________________________________________
> >> > Dbmail-dev mailing list
> >> > Dbmail-dev@dbmail.org
> >> > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
> >> >
> >> -- End Original Message --
> >>
> >>
>
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>



--

Hi
On 8 Jan 2004, at 19:23, Mark Mackay - Orcon wrote:

> Rapid development sounds good, but it would also be nice to have a set
> of
> target milestones so that people can know what's coming out and approx
> when.
It would of course be necessary to do more planning with regard to
milestones, feature sets etc. I agree on that.
>
>
> I guess the one thing that stands out to me which worries me a bit is
> schema
> changes, and even the addition of new tables, etc. The change for v1.x
> ->
> 2.0 is not likely to be a trivial one for me. I'm currently sitting on
> about
> 110G of mail and I know the changeover is going to be 'fun'. A
> once-every-so-often schema change is fine, but if you have to do it
> every
> 1-2 months just to keep current with the current version, then it
> starts
> becoming a bit of an admin burden. This is especially true if security
> patches are not applied to the previous versions (e.g if working on
> 2.4 and
> 2.2, 2.3 are not patched, etc).
Security fixes should always be applief to old versions (up to some
point, of course).
>
> There probably needs to be a better way to track schema changes and
> updates
> to databsae required between versions. A "What's changed between v1.2
> and
> 2.0" and "What's changed between v2.0 and 2.1" doc; which lists the
> SQL/scripts that need to be run to upgrade things and the schema
> changes
> made.
> Of course if you can avoid schema changes to more major versions 2.0,
> 2.5,
> 3.0, etc -- even better.
Agreed
>
> Maybe a project plan would help here - and the schema could be set up
> with
> the necessary fields.
Agreed
>
> Don't let the above rain on the development parade :) Just pointing
> out some
> of the concerns from my end relating to the 'cost' of upgrades.


Ilja

--
IC&S
Stadhouderslaan 57
3583 JD Utrecht
telnr. 030-6355730
faxnr. 030-6355731

PGP-key:
http://www.ic-s.nl/keys/ilja.txt

Hi Ilja -

> It would of course be necessary to do more planning with regard to
> milestones, feature sets etc. I agree on that.

Perhaps you could provide a list of the target features you have in mind for
2.0/2.1/etc (and if possible target times (e.g. February 2004 or Q2 2004,
etc) and then the list could pipe up and make their comments as to what else
they'd like to see and priorities/etc.

If there's stuff you'd like to see but need help with, maybe that could be
listed too - and then people could volunteer for the projects, etc.

There's a number of people hacking dbmail, and I'm sure if they knew that
the patches they came up with would make it into the release versions I'm
sure the development could be spread around a bit.

I guess what I'm saying is -- sounds great, how about starting us off so we
know what IC&S is aiming for as a base :)

/Mark

Hi,
On 9 Jan 2004, at 10:46, Mark Mackay - Orcon wrote:

> Hi Ilja -
>
>> It would of course be necessary to do more planning with regard to
>> milestones, feature sets etc. I agree on that.
>
> Perhaps you could provide a list of the target features you have in
> mind for
> 2.0/2.1/etc (and if possible target times (e.g. February 2004 or Q2
> 2004,
> etc) and then the list could pipe up and make their comments as to
> what else
> they'd like to see and priorities/etc.
I'll try to hack together a list next week (I'll also add a
Development page to the website)
>
> If there's stuff you'd like to see but need help with, maybe that
> could be
> listed too - and then people could volunteer for the projects, etc.
>

> There's a number of people hacking dbmail, and I'm sure if they knew
> that
> the patches they came up with would make it into the release versions
> I'm
> sure the development could be spread around a bit.
I know. I just have to adapt to being more of a maintainer, and less of
a coder probably (although I'll of course will keep on doing DBMail
coding, way to much fun!).
>
> I guess what I'm saying is -- sounds great, how about starting us off
> so we
> know what IC&S is aiming for as a base :)
As I mentioned above, I'll publish a (beginning of a) list next week
(here and on the website).

Ilja
--
IC&S
Stadhouderslaan 57
3583 JD Utrecht
telnr. 030-6355730
faxnr. 030-6355731

PGP-key:
http://www.ic-s.nl/keys/ilja.txt

Actually, I think a much better way of doing this is to give DBMail a few
'keywords' that can be used in configurable auth queries.

[AUTH]
auth_user_exists = SELECT name AS %user_name%, uid AS %user_idnr% FROM
radius.usernames WHERE name = %auth_user_name% AND ipaddr = %auth_ip_address%

Something like that. Basically, DBMail would run this query with the
%variable% names replaced with (quoted if necessary) internal values. So the
AS clauses would force the return columns to have the names that DBMail
expects, while the WHERE clause can still use the native naming. (I just
tested this on MySQL 4.0.16).

Aaron


lou@0xffff.org said:

> Mark Mackay - Orcon writes:
>
>
> mark, i think you're over complicating the thing with this
> pop3/imap proxy.. hack pbsp(extend it), make your radius write there the
> clients IP when they log, and their user?ip, then do your select ..
>
> you have so much data for a user.
> idnr
> name
> clientid
>
> and then pbsp
> ipnumber
> +name
> timestamp..
>
> you're already using xxx.isp, at the end of the day success is if you dont
> have to make your clients change their usernames, just for that..
>
> one optional field more wont be bad.. but why?
>
> you dont need proxy to achive that, this is trivial.
>
> hope that helps.
>
> c
>
> >> 1. configurable prefix. I guess the default should be an empty prefix,
> >> to default to the current table names.
> >> 2. do tablenames also need to configurable like lou@0xffff.org
> >> mentioned in a previous message? If queries get table names from
> >> variables, it shouldn't be so hard to make the whole table name
> >> configurable.
> >>
> >> Please shoot at my arguments. I think we need some strict
> >> requirements
> >> on this before Brian gets to work ;)
> >
> > I don't think all tablenames need to be configurable , or certainly the ones
> > are 'dbmail-specific' [but prefixable may be nice]; but one that may be
> > worth considering is the users table. I've said this before, many months
> > ago, so forgive my repeat ramblings:
> >
> > Most of us have a database of all our customers, for billing, radius, etc.
> > This to me it make sense to allow dbmail to be able to query our existing
> > user databases to ultimately find the dbmail "user_idnr" that we need to
> > handle delivery/checking of mail. Thus it may be worth making the "users"
> > table totally configurable, so you can hook into any existing table that
> > exists.
> >
> > The real power with above, would be to allow the magic dbmail-specific
> > columns to be customisable (so if you already had a "uid" column which
> > mimicked the passwd file, you could use that column rather than having to
> > add a "user_idnr" column to your database, etc.
> >
> > Even better -- allow custom queries from that table, to allow greater
> > flexibility. There are probably only a handful of queries which actually
> > interact with the users' table, so maybe these could be definable [.but not
> > required to be defined in a config, eg. default if not present].
> >
> > Reason: We do wholesale ISP services in NZ and also buy a lot of ISPs, which
> > results in the need to support multiple "mark" usernames on our mailserver
> > (people over here don't log in with mark@ispa.co.nz and mark@ispb.co.nz --
> > usually just 'mark'). Dbmail because it's in theory all numeric post-auth is
> > great for this -- I can have mark from ispa with user_idnr 1000 and mark
> > from ispb with user_idnr 2000. So all I need to do now is make sure I can
> > look up the right user_idnr when 'mark' connects.
> >
> > Currently we're having to proxy the pop3/imap streams and rewrite the
> > username so when someone goes "USER mark" the dbmail server actually gets
> > "USER mark.ispa" or similar. We have daemons bound to different IP addresses
> > and ports with different configs to do this.
> >
> > What would be great, is to allow dbmail to do this job for us -- so I don't
> > have another intermediate step which doesn't need to be here. I'd love
> > dbmail to be able to do the following:
> >
> > select user_idnr, password from users where userid='mark' and
> > SOMEOTHERCOLUMN='VALUE'
> >
> > This SOMEOTHERCOLUMN=VALUE could be an ip address (host='10.0.0.1') based on
> > the IP address of the incoming connection, or perhaps some variable that is
> > per-daemon defined (e.g. start ./dbmail-pop3d with some config that says
> > ISPID="DEFAULT" or ISPID="ISPB").
> >
> > Now I know not everyone wants this (or do you? -- there must be people in a
> > similar boat to me out there!). Having just the users-related queries
> > customisable, and allowing macros to be used such as
> >
> > %u for username
> > %p for password
> > %h for incoming hostip
> > %{customvariableinconfig}
> >
> > Would allow a very customisable solution for those people that needed it.
> > Then people could easily build up configs which queried their existing
> > user/billing databases.
> >
> > My 2c.
> >
> > /Mark
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Dbmail-dev mailing list
> > Dbmail-dev@dbmail.org
> > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>



--

Aaron Stone writes:

> We might be headed off the deep end, but, IMHO, it's an important exercise to
> see what's out there. There are certainly some large installations of DBMail
> right now, and with 2.x, even larger ones should be expected. Many of these
> larger installations will have reasonably complex environments already running
> and it would be best if DBMail could adapt a bit to fit.
>

what to adapt? we had a simple suggestion and we created a bloody
moster out of it. there's nothing wrong with brainstorming..

the lack of creativity in some people shouldn't be the reason
to alter dbmail according to their needs.

i'll not comment on this discussion anymore.. there's no point.

cheers
>
> lou@0xffff.org said:
>
>> Aaron Stone writes:
>>
>> for the sake of the ideas, this is getting too far and at the end it will be
>> too bloated. making everyone happy is not the most important thing here, i'm
>> happy with what i have and i'd change it the way i like it to suit me, be
>> that code/conf/whatever. what one code need is a good framework around which
>> we can hack.. and not to waste time on 'Oh, wtf is this'...
>>
>> I suppose you understand what I'm saying, sometimes a bit of abstraction
>> can reduce code and efforts, but too much can cause panic ..
>>
>> dont get lost in the loop guys.
>>
>> if my ideas and suggestions will lead to more complexity, please drop them.
>> i'd prefer to have dbmail the way it is now, than having it bloated with
>> those stuff below..
>>
>> c
>>
>> > This does indeed give everyone what the want wrt config files, but also
>> > introduces a lot of complexity. That's no necessarily bad, so long as the
>> > simple cases are handled simply.
>> >
>> > So, if you have a current config file, with everything in it, and no database
>> > config table... works fine.
>> >
>> > And, if you have a database only config file, with just the minimum needed to
>> > "bootstrap" into the database config table... works fine.
>> >
>> > Then, for crazy people, crazy things are possible if you're paying attention
>> > to the order of inclusions... complicated, but possible.
>> >
>> > Unless I left anything out, it sounds to me like this satisfies the "simple
>> > things are simple, complicated things are possible" rule of UNIX Philosophy.
>> >
>> > I can already hear Ilja groaning about this one, so if this is the code we
>> > want, someone's gonna have to do it ;-)
>> >
>> > There was one more issue, btw, which was the table names. I don't think that
>> > fully configurable names are particularly necessary, but one place where we
>> > might want to look into something like that is the users table. For people who
>> > might already have a pam_mysql or an apache mod_auth_mysql database, it would
>> > probably be nice for them to have dbmail use that existing database rather
>> > than migrating the users into dbmail and reconfiguring everything else. Dunno
>> > how to do this, just yet, although we should take a look at what pam_mysql and
>> > mod_auth_mysql use in their config files.
>> >
>> > Aaron
>> >
>> >
>> > ""Jesse Norell"" <jesse@kci.net> said:
>> >
>> >>
>> >> An idea that may work to suit everyone's preference of the config
>> >> layout: define specific section names for config items, like [SMTP],
>> >> and add that field to Aaron's table example below, and add an
>> >> [INCLUDE "/path/to/filename"] function to the config file parser.
>> >>
>> >> Make the config reader get the db parameters from the config file
>> >> (also add an (optional) item to the config file to disable
>> >> database config lookups completely for those who don't use them),
>> >> read in all the default values from the db, then override with any
>> >> host specific values from the db, then read through the local config
>> >> file and override settings with anything found there.
>> >>
>> >> Everyone could have the setup they wish - a huge dbmail.conf file,
>> >> or a minimal one with mostly db config settings, or even break the
>> >> local config into multiple files (/etc/dbmail/{dbmail,logging,host}.conf)
>> >> or whatever else you could think of.
>> >>
>> >>
>> >>
>> >> ---- Original Message ----
>> >> From: Eelco van Beek - IC&S <dbmail-dev@dbmail.org>
>> >> To: dbmail-dev@dbmail.org
>> >> Subject: Re: [Dbmail-dev] Re: table prefix
>> >> Sent: Thu, 8 Jan 2004 10:34:04 +0100
>> >>
>> >> > We could also do both (config on file and in the db) and keep them in
>> >> > sync. This enables easier administration through interfaces on the DB.
>> >> >
>> >> >
>> >> > On 8-jan-04, at 9:44, Mark Mackay - Orcon wrote:
>> >> >
>> >> > > I really like having a local config file [.much better than the old
>> >> > > days of
>> >> > > having to compile the database name in, etc]; and I like being
>> >> able to
>> >> > > just
>> >> > > 'vi' a file to enable logging, adjust process levels on one of the
>> >> > > hosts,
>> >> > > etc.
>> >> > >
>> >> > > If you want to do database configs as well, then the syntax below
>> >> > > would work
>> >> > > -- but rather than via hostname, I'd recommend making it some form of
>> >> > > 'clientid/configid' which could be supplied on command lines (e.g.
>> >> > > ../dbmail-pop3d -configkey DEFAULT) That would allow configs to be
>> >> > > experimented with and changed between without having to affect other
>> >> > > daemons
>> >> > > on the same host, etc.
>> >> > >
>> >> > > Personally I just say keep the config files :)
>> >> > >
>> >> > > Cheers,
>> >> > > Mark.
>> >> > >
>> >> > >> dbmail.conf:
>> >> > >> Is a configuration file in /etc or wherever.
>> >> > >> - database information
>> >> > >> - logging info
>> >> > >> - hostname / clustering info
>> >> > >>
>> >> > >> dbmail_conf:
>> >> > >> Is a table in the database.
>> >> > >> - Has detailed config info in host/key/value pairs:
>> >> > >>
>> >> > >> CREATE TABLE dbmail_conf (
>> >> > >> hostname VARCHAR(128) NOT NULL,
>> >> > >> key VARCHAR(32) NOT NULL,
>> >> > >> value TEXT NULL,
>> >> > >> UNIQUE INDEX (hostname, key)
>> >> > >> )
>> >> > >
>> >> > > _______________________________________________
>> >> > > Dbmail-dev mailing list
>> >> > > Dbmail-dev@dbmail.org
>> >> > > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>> >> > >
>> >> > _________________________
>> >> > E.J.A. van Beek
>> >> > ICT Manager
>> >> > IC&S
>> >> > T: +31 30 6355730
>> >> > F: +31 30 6355731
>> >> >
>> >> > PGP-key:
>> >> > www.ic-s.nl/keys/eelco.txt
>> >> >
>> >> > _______________________________________________
>> >> > Dbmail-dev mailing list
>> >> > Dbmail-dev@dbmail.org
>> >> > http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>> >> >
>> >> -- End Original Message --
>> >>
>> >>
>>
>>
>> _______________________________________________
>> Dbmail-dev mailing list
>> Dbmail-dev@dbmail.org
>> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>>
>
>
>
> --
>
>
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev

> I'll try to hack together a list next week (I'll also add a
> Development page to the website)

I'm sure we'll all look forward to it :)

Sigh. You know you're a geek when you start getting excited about
programming goals and release dates ;)

/Mark

> Actually, I think a much better way of doing this is to give
> DBMail a few 'keywords' that can be used in configurable auth queries.

Absolutely -- this is exactly what I had in mind. I only thing the queries
that relate to determining a user's user_idnr are the ones that need to be
customizable. The ones relating the internal mail structure shouldn't need
configuring.

I had a fresh look at the 2.0 CVS (been a while since I really examined the
CVS), and the structure with the auth/ looks great, as I'm guessing that
would be the only place we need to play. Worst case, I guess a custom
authsql.c could be written with our custom queries. Only about 10 queries
are used by the looks of it. It would certainly be nice to implement this
into the main tree though.

/Mark


> [AUTH]
> auth_user_exists = SELECT name AS %user_name%, uid AS %user_idnr% FROM
> radius.usernames WHERE name = %auth_user_name% AND ipaddr =
> %auth_ip_address%
>
> Something like that. Basically, DBMail would run this query with the
> %variable% names replaced with (quoted if necessary) internal
> values. So the
> AS clauses would force the return columns to have the names
> that DBMail
> expects, while the WHERE clause can still use the native
> naming. (I just
> tested this on MySQL 4.0.16).