Mailing List Archive

Got it:

messageblks WHERE physmessage_id = '299658' ORDER BY messageblk_idnr;

Hey! There's that dump utility we were talking about.

Looks like non-ascii chars at the start of the second messageblk (or end
of the first...) is what's causing _ic_fetch to fail.



Blake Mitchell wrote:
> I still get both the unparseable message and empty message problems
> (post rc4 CVS) occasionally, the empty message happens more often, about
> once a day or so. For the empty messages it's a matter of the imap
> server not pulling the whole message, because dbmail-smtp does put it
> all in there if I look with the command line mysql client. What can I
> provide to help solve these problems? Is there a way to have the mysql
> client dump the messageblks as hex or something which will better show
> the non printing characters?
>
> Paul J Stevens wrote:
>
>>
>>
>> Ilja Booij wrote:
>>
>>> OK great.
>>>
>>> Now I think we only need to fix the parser error that Aaron found.
>>> Aaron won't be back until monday, and he's the only one who's had
>>> this problem, which kind of prevents us from doing anything about it
>>> at the moment.
>>
>>
>>
>> There's one other problem that's bugging me. I sometimes get empty
>> messages showing up. I thought it was a problem with thunderbird, but
>> it isn't. Apparently, messages sometimes only get one messageblk for
>> the headers, and that's it.
>>
>> I could be related to Aaron's problem, but I'm not sure. It could also
>> be a problem with my dbmail database. I will investigate further.
>>
>>> Meanwhile, I'll add the IMAP test script to the distribution. Can I
>>> just slam a GPL notice on it, or do you want another license (it's
>>> your code :) )?
>>
>>
>>
>> Feel free to add the GPL notice, but please add my name as well :-)
>>
>> Also, there's one project out there that looks interesting:
>>
>> http://www.mozilla.org/projects/mstone/
>> """
>> Mstone is a multi-protocol stress and performance measurement tool.
>> Mstone can test multiple protocols simultaneously and measures the
>> performance of every transaction. The performance can be graphed
>> throughout the duration of the test.
>> """
>>
>> Very useful as I'll resume work on my preforking patch, and we start
>> thinking about the C10K problem.
>>
>> http://www.kegel.com/c10k.html
>>
> _______________________________________________
> Dbmail-dev mailing list
> Dbmail-dev@dbmail.org
> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>

--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

A final followup for tonight: Those non-ascii chars were not there at
the beginning of the original message. So all this is quite likely
related to the parser bug (or another parser bug) after all.


Paul J Stevens wrote:
> Got it:
>
> mysql> SELECT messageblk into dumpfile '/tmp/messageblks.blob' FROM
> messageblks WHERE physmessage_id = '299658' ORDER BY messageblk_idnr;
>
> Hey! There's that dump utility we were talking about.
>
> Looks like non-ascii chars at the start of the second messageblk (or end
> of the first...) is what's causing _ic_fetch to fail.
>
>
>
> Blake Mitchell wrote:
>
>> I still get both the unparseable message and empty message problems
>> (post rc4 CVS) occasionally, the empty message happens more often,
>> about once a day or so. For the empty messages it's a matter of the
>> imap server not pulling the whole message, because dbmail-smtp does
>> put it all in there if I look with the command line mysql client. What
>> can I provide to help solve these problems? Is there a way to have the
>> mysql client dump the messageblks as hex or something which will
>> better show the non printing characters?
>>
>> Paul J Stevens wrote:
>>
>>>
>>>
>>> Ilja Booij wrote:
>>>
>>>> OK great.
>>>>
>>>> Now I think we only need to fix the parser error that Aaron found.
>>>> Aaron won't be back until monday, and he's the only one who's had
>>>> this problem, which kind of prevents us from doing anything about it
>>>> at the moment.
>>>
>>>
>>>
>>>
>>> There's one other problem that's bugging me. I sometimes get empty
>>> messages showing up. I thought it was a problem with thunderbird, but
>>> it isn't. Apparently, messages sometimes only get one messageblk for
>>> the headers, and that's it.
>>>
>>> I could be related to Aaron's problem, but I'm not sure. It could
>>> also be a problem with my dbmail database. I will investigate further.
>>>
>>>> Meanwhile, I'll add the IMAP test script to the distribution. Can I
>>>> just slam a GPL notice on it, or do you want another license (it's
>>>> your code :) )?
>>>
>>>
>>>
>>>
>>> Feel free to add the GPL notice, but please add my name as well :-)
>>>
>>> Also, there's one project out there that looks interesting:
>>>
>>> http://www.mozilla.org/projects/mstone/
>>> """
>>> Mstone is a multi-protocol stress and performance measurement tool.
>>> Mstone can test multiple protocols simultaneously and measures the
>>> performance of every transaction. The performance can be graphed
>>> throughout the duration of the test.
>>> """
>>>
>>> Very useful as I'll resume work on my preforking patch, and we start
>>> thinking about the C10K problem.
>>>
>>> http://www.kegel.com/c10k.html
>>>
>> _______________________________________________
>> Dbmail-dev mailing list
>> Dbmail-dev@dbmail.org
>> http://twister.fastxs.net/mailman/listinfo/dbmail-dev
>>
>

--
________________________________________________________________
Paul Stevens mailto:paul@nfg.nl
NET FACILITIES GROUP PGP: finger paul@nfg.nl
The Netherlands________________________________http://www.nfg.nl

Excellent! I'll look at mine right now. This reminds me, how hard would
it be to add "X-DBMail-MessageID" and "X-DBMail-PhysMessageID" headers
to the messages, for debugging such things as this?

Paul J Stevens wrote:

> Got it:
>
> mysql> SELECT messageblk into dumpfile '/tmp/messageblks.blob' FROM
> messageblks WHERE physmessage_id = '299658' ORDER BY messageblk_idnr;
>
> Hey! There's that dump utility we were talking about.
>
> Looks like non-ascii chars at the start of the second messageblk (or end
> of the first...) is what's causing _ic_fetch to fail.
>

Blake Mitchell wrote:

> Excellent! I'll look at mine right now. This reminds me, how hard would
> it be to add "X-DBMail-MessageID" and "X-DBMail-PhysMessageID" headers
> to the messages, for debugging such things as this?

That would be very nice! Make it optional of course by adding an extra
option to dbmail.conf like add_debug_mail_headers or something like that.

Matthew T. O'Connor wrote:
> Blake Mitchell wrote:
>
>> Excellent! I'll look at mine right now. This reminds me, how hard
>> would it be to add "X-DBMail-MessageID" and "X-DBMail-PhysMessageID"
>> headers to the messages, for debugging such things as this?
>
>
> That would be very nice! Make it optional of course by adding an extra
> option to dbmail.conf like add_debug_mail_headers or something like that.

Adding the "X-DBMail-PhysMessageID" shouldn't be too hard, since the
physmessage_id of a message won't change after copying (which happens on
delivery). "X-DBMail-MessageID" is harder, it should happen on the fly.
You cannot add the "X-DBMail-MessageID" header to the messageblock in
the database, because it's possible that several different message-id's
point to the same messageblock.

I'm opting for not putting this in just yet, although it seems to be a
nice feature. We should definitely do this later on.

Ilja

Paul J Stevens wrote:
> A final followup for tonight: Those non-ascii chars were not there at
> the beginning of the original message. So all this is quite likely
> related to the parser bug (or another parser bug) after all.

So what you're saying is that there are characters in the message stored
in the database, that should not be there?
If this is correct then the bug is in the message insertion code. It
sounds a lot like a piece of uninitialised or non-cleared (no
memset(buffer, '\0', size) used) memory that's being inserted, giving
random results.

Can anybody send the list such a message? And even better, attach a
trace of the message being inserted (that should give some clues to the
origin of the 'random' characters.

Ilja

Paul J Stevens wrote:
> Ilja Booij wrote:
>> just slam a GPL notice on it, or do you want another license (it's
>> your code :) )?
>
>
> Feel free to add the GPL notice, but please add my name as well :-)

of course.
>
> Also, there's one project out there that looks interesting:
>
> http://www.mozilla.org/projects/mstone/
> """
> Mstone is a multi-protocol stress and performance measurement tool.
> Mstone can test multiple protocols simultaneously and measures the
> performance of every transaction. The performance can be graphed
> throughout the duration of the test.
> """

I'll take a look it.
>
> Very useful as I'll resume work on my preforking patch, and we start
> thinking about the C10K problem.
>
> http://www.kegel.com/c10k.html

Interesting problem. I've been thinking about that for a while now. It
would of course be possible to build a select() or poll() based server
(there are some ready-made frameworks for this). I'm doing some
experiments with building a simple POP server (using DBMail as storage
of course) in python (using asyncore and asynchat). I do this in my free
time, so development is very slow.. (so many other things to do..).

I still wonder if this problem is really relevant for us though, as many
DBMail actions are more CPU (database) bound than I/O-bound.

Ilja

>