Mailing List Archive

If you're doing that much reworking things, I bet this would be an
ideal time to impliment shared folders and related issues recently
discussed (multiple recipients receiving the same message - rework
the db tables just a bit so you only store a single copy of a
message, and keep track of what mailboxes it belongs too).

If you want to get really ambitious, and end up doing schema
changes anyways, you could stick in the injector side of header
caching, and the pop3/imap daemons could but optimized to use that
later. :)




---- Original Message ----
From: Aaron Stone <dbmail-dev@dbmail.org>
To: dbmail-dev@dbmail.org
Subject: [Dbmail-dev] The wonderful flaws of injector.c, and a plan to move ahead
Sent: Sat, 19 Apr 2003 12:44:18 -0700 (PDT)

I finished reading through main/pipe, injector and mini-injector. Now I
recall the old thread about the gazillion simultaneous pipes chocking
someone's system. injector.c sets up an open sendmail process for each and
every forward, then begins reading in the message from stdin and after
each chunk of data, loops the pipes and sends on them, then loops the
local user id numbers and inserts a block into the database. Repeat until
stdin has finished.

This is an interesting but unworkable architecture because a large set of
forwarding addresses will cause the system to die. A throttled system
would be much more effective. Was this discussed at the time?

As for main/pipe, I was not able to find if there was code to use the
temporary tables. This would be the way out for injector.c, of course.

Where I'm at now is wanting to merge main/pipe with injector. Not sure
about bounce.c and forward.c, although I'd lean toward keeping them. The
code path I see is...

- Start up "new-injector"
- Process command line arguments
- Read the header from stdin (but not hard coded in each function!)
- Set up the lists of local user ids, forwards, bounces (no such user)
- Create a temporary message
- Read in the entire message, storing to the temporary table
- Calculate the message size, and set up the list of quota bounces
- Get the 'SENDMAIL_THROTTLE' config value (whatever we call it)
- Allocate ( fwd.total < THROTTLE ? fwd.total : THROTTLE ) pipes

while( db_get_me_some_data_from_the_temporary_table() )
{
- Get the first/next message block from the database
- For each pipe, send out the data
}

- Repeat this procedure for the local users (Except that it should be
abstracted to the database driver layer. Some versions of the supported
db's can simply use a subselect and not require reading/buffering/writing
back to the database. Much better!)

- Send the no such user bounces and the over quota bounces, again using
the throttle mechanism.


Naturally, the order of a few things can be changed around, but this is
the basic idea. Questions, comments? Eelco and Roel, anything important
that I left out?

Aaron



-- End Original Message --


--
Jesse Norell
jesse (at) kci.net

I'm really just working on the top level of things right now. I still
believe that there's very little reason multiple entries in the messages
table can't point to the same set of messageblks for shared messages,
although actual shared folders will require an additional level somewhere.

Ideally, an appropriate middle layer of code will emerge that can easily
be adapted to new database schemas. Removing this code from the
servers themselves is where I'm at now. Besides, I'm already stealing
enough thunder from Roel, who wanted to do the LMTP server this month! :-P
Perhaps he'll be free now to work on the schema changes and making the
world a better place (or keeping IC&S in business. that, and homework,
seem to be the common limiting factors these days :-)

Aaron


On Sat, 19 Apr 2003, Jesse Norell wrote:

>
>
> If you're doing that much reworking things, I bet this would be an
> ideal time to impliment shared folders and related issues recently
> discussed (multiple recipients receiving the same message - rework
> the db tables just a bit so you only store a single copy of a
> message, and keep track of what mailboxes it belongs too).
>
> If you want to get really ambitious, and end up doing schema
> changes anyways, you could stick in the injector side of header
> caching, and the pop3/imap daemons could but optimized to use that
> later. :)
>
>
>
>
> ---- Original Message ----
> From: Aaron Stone <dbmail-dev@dbmail.org>
> To: dbmail-dev@dbmail.org
> Subject: [Dbmail-dev] The wonderful flaws of injector.c, and a plan to move ahead
> Sent: Sat, 19 Apr 2003 12:44:18 -0700 (PDT)
>
> I finished reading through main/pipe, injector and mini-injector. Now I
> recall the old thread about the gazillion simultaneous pipes chocking
> someone's system. injector.c sets up an open sendmail process for each and
> every forward, then begins reading in the message from stdin and after
> each chunk of data, loops the pipes and sends on them, then loops the
> local user id numbers and inserts a block into the database. Repeat until
> stdin has finished.
>
> This is an interesting but unworkable architecture because a large set of
> forwarding addresses will cause the system to die. A throttled system
> would be much more effective. Was this discussed at the time?
>
> As for main/pipe, I was not able to find if there was code to use the
> temporary tables. This would be the way out for injector.c, of course.
>
> Where I'm at now is wanting to merge main/pipe with injector. Not sure
> about bounce.c and forward.c, although I'd lean toward keeping them. The
> code path I see is...
>
> - Start up "new-injector"
> - Process command line arguments
> - Read the header from stdin (but not hard coded in each function!)
> - Set up the lists of local user ids, forwards, bounces (no such user)
> - Create a temporary message
> - Read in the entire message, storing to the temporary table
> - Calculate the message size, and set up the list of quota bounces
> - Get the 'SENDMAIL_THROTTLE' config value (whatever we call it)
> - Allocate ( fwd.total < THROTTLE ? fwd.total : THROTTLE ) pipes
>
> while( db_get_me_some_data_from_the_temporary_table() )
> {
> - Get the first/next message block from the database
> - For each pipe, send out the data
> }
>
> - Repeat this procedure for the local users (Except that it should be
> abstracted to the database driver layer. Some versions of the supported
> db's can simply use a subselect and not require reading/buffering/writing
> back to the database. Much better!)
>
> - Send the no such user bounces and the over quota bounces, again using
> the throttle mechanism.
>
>
> Naturally, the order of a few things can be changed around, but this is
> the basic idea. Questions, comments? Eelco and Roel, anything important
> that I left out?
>
> Aaron
>
>
>
> -- End Original Message --
>
>
> --
> Jesse Norell
> jesse (at) kci.net
>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>