I finished reading through main/pipe, injector and mini-injector. Now I
recall the old thread about the gazillion simultaneous pipes chocking
someone's system. injector.c sets up an open sendmail process for each and
every forward, then begins reading in the message from stdin and after
each chunk of data, loops the pipes and sends on them, then loops the
local user id numbers and inserts a block into the database. Repeat until
stdin has finished.
This is an interesting but unworkable architecture because a large set of
forwarding addresses will cause the system to die. A throttled system
would be much more effective. Was this discussed at the time?
As for main/pipe, I was not able to find if there was code to use the
temporary tables. This would be the way out for injector.c, of course.
Where I'm at now is wanting to merge main/pipe with injector. Not sure
about bounce.c and forward.c, although I'd lean toward keeping them. The
code path I see is...
- Start up "new-injector"
- Process command line arguments
- Read the header from stdin (but not hard coded in each function!)
- Set up the lists of local user ids, forwards, bounces (no such user)
- Create a temporary message
- Read in the entire message, storing to the temporary table
- Calculate the message size, and set up the list of quota bounces
- Get the 'SENDMAIL_THROTTLE' config value (whatever we call it)
- Allocate ( fwd.total < THROTTLE ? fwd.total : THROTTLE ) pipes
while( db_get_me_some_data_from_the_temporary_table() )
{
- Get the first/next message block from the database
- For each pipe, send out the data
}
- Repeat this procedure for the local users (Except that it should be
abstracted to the database driver layer. Some versions of the supported
db's can simply use a subselect and not require reading/buffering/writing
back to the database. Much better!)
- Send the no such user bounces and the over quota bounces, again using
the throttle mechanism.
Naturally, the order of a few things can be changed around, but this is
the basic idea. Questions, comments? Eelco and Roel, anything important
that I left out?
Aaron
recall the old thread about the gazillion simultaneous pipes chocking
someone's system. injector.c sets up an open sendmail process for each and
every forward, then begins reading in the message from stdin and after
each chunk of data, loops the pipes and sends on them, then loops the
local user id numbers and inserts a block into the database. Repeat until
stdin has finished.
This is an interesting but unworkable architecture because a large set of
forwarding addresses will cause the system to die. A throttled system
would be much more effective. Was this discussed at the time?
As for main/pipe, I was not able to find if there was code to use the
temporary tables. This would be the way out for injector.c, of course.
Where I'm at now is wanting to merge main/pipe with injector. Not sure
about bounce.c and forward.c, although I'd lean toward keeping them. The
code path I see is...
- Start up "new-injector"
- Process command line arguments
- Read the header from stdin (but not hard coded in each function!)
- Set up the lists of local user ids, forwards, bounces (no such user)
- Create a temporary message
- Read in the entire message, storing to the temporary table
- Calculate the message size, and set up the list of quota bounces
- Get the 'SENDMAIL_THROTTLE' config value (whatever we call it)
- Allocate ( fwd.total < THROTTLE ? fwd.total : THROTTLE ) pipes
while( db_get_me_some_data_from_the_temporary_table() )
{
- Get the first/next message block from the database
- For each pipe, send out the data
}
- Repeat this procedure for the local users (Except that it should be
abstracted to the database driver layer. Some versions of the supported
db's can simply use a subselect and not require reading/buffering/writing
back to the database. Much better!)
- Send the no such user bounces and the over quota bounces, again using
the throttle mechanism.
Naturally, the order of a few things can be changed around, but this is
the basic idea. Questions, comments? Eelco and Roel, anything important
that I left out?
Aaron