Mailing List Archive

better but still errors -> Re: 3.1.16 regression -> Re: DBMail 3.1.16 released (Re: corrupted multi-mime messages)
77d17ca7724c32dba27eb187d4abfa129ccf6c4e is *way better* but still
not error free, can you please fire coverity again because i think
it's more effective than me building and testing a lot of versions
over hours - smells like one remaining buffer overflow / underrun

* "Ostrava Tuesday morning" no longer broken
* "Panel: E-Mail-Debug - IMAP-POP3-Download" 4 out of 1000 broken
* i have attached the 4 diff-files
* all 4 still always identical broken
* two times POP3
* two times IMAP

additional info:

* "Ostrava Tuesday morning" with 23,40 KB *no error* on 100 loops
* "Panel: E-Mail-Debug - IMAP-POP3-Download" is 380 KB large

so it seems larger messages are more likely to reprocude the problem, but that's
a error rate which let me put 77d17ca7724c32dba27eb187d4abfa129ccf6c4e in
production because better than current 8a042214ae1d120581740020f4e73c3cf8d3a6c0
___________________________________________________

my stress-test complained within 25 minutes 5 times

"multipart_message_3", "message_rfc822" and "multipart_message_2"
are from the dbmail-testsuite, "Test 5" is one of my past problem
cases

Jul 20 18:37:13 POP3 24: EXPECTED: 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)

Jul 20 18:41:48 POP3 28: EXPECTED: 60dd43bb886f6c47d89b5dc3ab3f7f764b68eb2e / FOUND:
b99c389d3479d8d407d43045bcd99219e51a8977 (message_rfc822)

Jul 20 18:45:53 POP3 25: EXPECTED: 0e2409bcf6921c5da801e1bf4ca4f57be9f07775 / FOUND:
3ef1aef632e08eef66d77e5a97091797640f3192 (multipart_message_2)

Jul 20 18:47:00 POP3 01: EXPECTED: 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)

Jul 20 18:54:47 POP3 01: EXPECTED: 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)

Am 20.07.2014 17:36, schrieb Reindl Harald (mobile):
> I will give it a try after back at home
>
> if the other reconstruct problem under load still exists i try to get back release version to release version until it is away and want the commits after upwards
>
> that sort of regressions was solved for sure because we both spent a lot of hours last year and finally there where load tests over hours without a single mismatch
>
> However, if we really find a guilty commit the interesting question will be what assumption leaded to that exactly code and how to prevent similar bugs - Autotest are great but can't replay real world mixed load, even my stress tests are synthetic at the end of the day and can't really simulate 50 different clients at the same time acting on completely different messages and what no test ever will reproduce is the memory state with possible leaks part of the game after several days of running
>
> the race bug on the other hand seems to be worst after restart the service - very strange
>
>
> -------- Ursprüngliche Nachricht --------
> Von: Paul J Stevens <paul@nfg.nl>
> Gesendet: 20. Juli 2014 17:00:15 MESZ
> An: DBMail mailinglist <dbmail@dbmail.org>
> Betreff: Re: [Dbmail] 3.1.16 regression -> Re: DBMail 3.1.16 released (Re: corrupted multi-mime messages)
>
> I can't believe this passed the tests, since this exact message is
> part of the test-suite. Seems the test itself was broken :-(
>
> I just pushed a fix. *Please* test!
>
> On 20-07-14 14:12, Reindl Harald wrote:
>> CAUTION
>
>> 3.1.16 at least has a regression in case of the commit below that's
>> a different problem and the testmessage with downgrading is
>> reconstructed proper
>> http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_1&id=8a042214ae1d120581740020f4e73c3cf8d3a6c0
>
>> however, the mime-part-id problem is still the same under load
>
>> i sent Paul some debug data offlist because they contain real
>> messages i am allowed by the mailowner to forward only that way,
>> not in public
>
>> Regards Harry
>
>> Am 20.07.2014 13:38, schrieb Reindl Harald:
>>> *attached* a compressed example *diff* broken / non-broken copy
>>> while compare the same mail IMAP / POP3 in a loop
>>>
>>> 33 out of 1000 always identically broken
>>>
>>> it don't matter on which test message i start the loop the result
>>> is the same in case of multi-mime mails
>>>
>>> fact is there are times while running mixed load for a minute not
>>> a single one get broken and in random time windows any multimime
>>> message for some seconds is damaged the same way
>>> _______________________________________________________________
>>>
>>> that below is the temp-folder where the loop stores diffs look at
>>> the number at the end, that's the loop-comuter
>>>
>>> at the begin of the loop a horrible result with 19 broken copies
>>> in serie and later 118-125 are again 8 in serie and they all have
>>> the same difference as "example-diff.zip"
>>>
>>> the mixed load now get "multipart_message_3" instead
>>> "multipart_message_6" most of the time broken
>>> _______________________________________________________________
>>>
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_002.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_003.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_004.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_005.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_006.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_007.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_009.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_010.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_011.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_012.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_013.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_014.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_015.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_016.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_017.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_018.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_019.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_020.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_021.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_084.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_089.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_118.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_119.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_120.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_121.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_122.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_123.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_124.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_125.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_335.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_364.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_380.diff
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_707.diff
>>> _______________________________________________
>>>
>>> Jul 20 13:26:56 POP3 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Jul 20 13:26:56 IMAP 24: EXPECTED:
>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>>> Jul 20 13:26:56 IMAP 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Jul 20 13:27:23 POP3 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Jul 20 13:27:23 IMAP 24: EXPECTED:
>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>>> Jul 20 13:27:23 IMAP 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Jul 20 13:27:52 POP3 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Jul 20 13:27:52 IMAP 24: EXPECTED:
>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>>> Jul 20 13:27:52 IMAP 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Jul 20 13:28:15 POP3 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Jul 20 13:28:15 IMAP 24: EXPECTED:
>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>>> Jul 20 13:28:15 IMAP 33: EXPECTED:
>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>> morning) Am 20.07.2014 12:56, schrieb Reindl Harald:
>>>> Hi Paul
>>>>
>>>> sorry but still unconfirmed
>>>>
>>>> see attached screenshot of debug-interface as well as
>>>> "stress-test.txt" which normally should not output anything
>>>> because it prcatically makes a lot of noise while check again
>>>> and aagin the same messages against known checksums
>>>>
>>>> the "Ostrava Tuesday morning" and "multipart_message_6" which
>>>> is from the dbmail-autotests imported on my IMAP seems to get
>>>> most likely broken, but others to randomly
>>>>
>>>> "Ostrava Tuesday morning is the one with context to
>>>> "8a042214ae1d120581740020f4e73c3cf8d3a6c0" and was the first
>>>> customer complaint
>>>>
>>>> Regards Harry
>>>>
>>>> Am 20.07.2014 12:30, schrieb Paul J Stevens:
>>>>> It was a buffer overrun which has now been fixed to the best
>>>>> of my abilities.
>>>>>
>>>>> I've done another round of coverity analysis on the master
>>>>> branch, and this doesn't trigger any new warnings with regard
>>>>> to mentioned fix which is also in the dbmail_3_1 branch.
>>>>>
>>>>> I've just tagged and thereby released 3.1.16 which contains
>>>>> just this fix.
>>>>>
>>>>> Being on vacation, I don't have full access to my usual
>>>>> release scripts (home-network is unreachable), so a tag will
>>>>> have to do.
>>>>>
>>>>> But I've also uploaded the tar file seperately to
>>>>> www.dbmail.org, so all the usual download locations should
>>>>> work.
>>>>>
>>>>> On 18-07-14 23:05, Reindl Harald (mobile) wrote:
>>>>>> Interesting - your attachment showed exactly the same bug I
>>>>>> have only randomly for whatever type of messages and only
>>>>>> under load
>>>>>>
>>>>>> I would sell my soul for a predictable reproducer because
>>>>>> in that case debug logging / gdb / strace could shed some
>>>>>> light and since it's always the same want finally happens
>>>>>> it's pretty sure only a few lines of code in dbmail to fix
>>>>>> it
>>>>>>
>>>>>> I am still sure it's exactly the same code path in both of
>>>>>> our cases and only the trigger to went that path is
>>>>>> different
>>>>>>
>>>>>> Maybe Paul has some idea how you can provide debug info's
>>>>>> since you are able to isolate what happens while my trigger
>>>>>> ends in a large mess of mixed debug logs
>>>>>>
>>>>>>
>>>>>> -------- Ursprüngliche Nachricht -------- Von: Guilherme
>>>>>> Souza <gslsouza@gmail.com> Gesendet: 18. Juli 2014 21:20:35
>>>>>> MESZ An: DBMail mailinglist <dbmail@dbmail.org> Betreff:
>>>>>> Re: [Dbmail] corrupted multi-mime messages
>>>>>>
>>>>>> I was no able to reply to your message. In my case is not
>>>>>> random.
>>>>>>
>>>>>> Every message html formatted with an attached file sent by
>>>>>> ensignia webmail shows the same missing boundaries.
>>>>>>
>>>>>> Every message sent by roundcube or thunderbird shows
>>>>>> correctly.
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 18, 2014 at 2:31 PM, Reindl Harald (mobile)
>>>>>> <h.reindl@thelounge.net> wrote:
>>>>>>> That is exactly what i reported a few days ago - that's
>>>>>>> for sure a race condition in reconstruction because i get
>>>>>>> predictable identical broken messages in a loop while
>>>>>>> mixed load on my testserver - at the same machine 1000
>>>>>>> loops receive the same message without other clients no
>>>>>>> broken one
Re: better but still errors -> Re: 3.1.16 regression -> Re: DBMail 3.1.16 released (Re: corrupted multi-mime messages) [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

If you can, test the master-HEAD. It contains all coverity fixes I
could find. I will not run coverity against 3.1

On 20-07-14 19:06, Reindl Harald wrote:
> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e is *way better* but still
> not error free, can you please fire coverity again because i think
> it's more effective than me building and testing a lot of versions
> over hours - smells like one remaining buffer overflow / underrun
>
> * "Ostrava Tuesday morning" no longer broken * "Panel: E-Mail-Debug
> - IMAP-POP3-Download" 4 out of 1000 broken * i have attached the 4
> diff-files * all 4 still always identical broken * two times POP3 *
> two times IMAP
>
> additional info:
>
> * "Ostrava Tuesday morning" with 23,40 KB *no error* on 100 loops *
> "Panel: E-Mail-Debug - IMAP-POP3-Download" is 380 KB large
>
> so it seems larger messages are more likely to reprocude the
> problem, but that's a error rate which let me put
> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e in production because
> better than current 8a042214ae1d120581740020f4e73c3cf8d3a6c0
> ___________________________________________________

> my stress-test complained within 25 minutes 5 times
>
> "multipart_message_3", "message_rfc822" and "multipart_message_2"
> are from the dbmail-testsuite, "Test 5" is one of my past problem
> cases
>
> Jul 20 18:37:13 POP3 24: EXPECTED:
> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>
> Jul 20 18:41:48 POP3 28: EXPECTED:
> 60dd43bb886f6c47d89b5dc3ab3f7f764b68eb2e / FOUND:
> b99c389d3479d8d407d43045bcd99219e51a8977 (message_rfc822)
>
> Jul 20 18:45:53 POP3 25: EXPECTED:
> 0e2409bcf6921c5da801e1bf4ca4f57be9f07775 / FOUND:
> 3ef1aef632e08eef66d77e5a97091797640f3192 (multipart_message_2)
>
> Jul 20 18:47:00 POP3 01: EXPECTED:
> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>
> Jul 20 18:54:47 POP3 01: EXPECTED:
> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>
> Am 20.07.2014 17:36, schrieb Reindl Harald (mobile):
>> I will give it a try after back at home
>>
>> if the other reconstruct problem under load still exists i try to
>> get back release version to release version until it is away and
>> want the commits after upwards
>>
>> that sort of regressions was solved for sure because we both
>> spent a lot of hours last year and finally there where load tests
>> over hours without a single mismatch
>>
>> However, if we really find a guilty commit the interesting
>> question will be what assumption leaded to that exactly code and
>> how to prevent similar bugs - Autotest are great but can't replay
>> real world mixed load, even my stress tests are synthetic at the
>> end of the day and can't really simulate 50 different clients at
>> the same time acting on completely different messages and what no
>> test ever will reproduce is the memory state with possible leaks
>> part of the game after several days of running
>>
>> the race bug on the other hand seems to be worst after restart
>> the service - very strange
>>
>>
>> -------- Ursprüngliche Nachricht -------- Von: Paul J Stevens
>> <paul@nfg.nl> Gesendet: 20. Juli 2014 17:00:15 MESZ An: DBMail
>> mailinglist <dbmail@dbmail.org> Betreff: Re: [Dbmail] 3.1.16
>> regression -> Re: DBMail 3.1.16 released (Re: corrupted
>> multi-mime messages)
>>
>> I can't believe this passed the tests, since this exact message
>> is part of the test-suite. Seems the test itself was broken :-(
>>
>> I just pushed a fix. *Please* test!
>>
>> On 20-07-14 14:12, Reindl Harald wrote:
>>> CAUTION
>>
>>> 3.1.16 at least has a regression in case of the commit below
>>> that's a different problem and the testmessage with downgrading
>>> is reconstructed proper
>>> http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_1&id=8a042214ae1d120581740020f4e73c3cf8d3a6c0
>>
>>>
>>>
however, the mime-part-id problem is still the same under load
>>
>>> i sent Paul some debug data offlist because they contain real
>>> messages i am allowed by the mailowner to forward only that
>>> way, not in public
>>
>>> Regards Harry
>>
>>> Am 20.07.2014 13:38, schrieb Reindl Harald:
>>>> *attached* a compressed example *diff* broken / non-broken
>>>> copy while compare the same mail IMAP / POP3 in a loop
>>>>
>>>> 33 out of 1000 always identically broken
>>>>
>>>> it don't matter on which test message i start the loop the
>>>> result is the same in case of multi-mime mails
>>>>
>>>> fact is there are times while running mixed load for a minute
>>>> not a single one get broken and in random time windows any
>>>> multimime message for some seconds is damaged the same way
>>>> _______________________________________________________________
>>>>
>>>>
>>>>
that below is the temp-folder where the loop stores diffs look at
>>>> the number at the end, that's the loop-comuter
>>>>
>>>> at the begin of the loop a horrible result with 19 broken
>>>> copies in serie and later 118-125 are again 8 in serie and
>>>> they all have the same difference as "example-diff.zip"
>>>>
>>>> the mixed load now get "multipart_message_3" instead
>>>> "multipart_message_6" most of the time broken
>>>> _______________________________________________________________
>>>>
>>>>
>>>>
dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_002.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_003.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_004.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_005.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_006.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_007.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_009.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_010.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_011.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_012.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_013.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_014.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_015.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_016.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_017.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_018.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_019.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_020.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_021.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_084.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_089.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_118.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_119.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_120.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_121.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_122.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_123.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_124.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_125.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_335.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_364.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_380.diff
>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_707.diff
>>>> _______________________________________________
>>>>
>>>> Jul 20 13:26:56 POP3 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Jul 20 13:26:56 IMAP 24: EXPECTED:
>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>> (multipart_message_3) Jul 20 13:26:56 IMAP 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Jul 20 13:27:23 POP3 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Jul 20 13:27:23 IMAP 24: EXPECTED:
>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>> (multipart_message_3) Jul 20 13:27:23 IMAP 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Jul 20 13:27:52 POP3 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Jul 20 13:27:52 IMAP 24: EXPECTED:
>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>> (multipart_message_3) Jul 20 13:27:52 IMAP 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Jul 20 13:28:15 POP3 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Jul 20 13:28:15 IMAP 24: EXPECTED:
>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>> (multipart_message_3) Jul 20 13:28:15 IMAP 33: EXPECTED:
>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>> morning) Am 20.07.2014 12:56, schrieb Reindl Harald:
>>>>> Hi Paul
>>>>>
>>>>> sorry but still unconfirmed
>>>>>
>>>>> see attached screenshot of debug-interface as well as
>>>>> "stress-test.txt" which normally should not output
>>>>> anything because it prcatically makes a lot of noise while
>>>>> check again and aagin the same messages against known
>>>>> checksums
>>>>>
>>>>> the "Ostrava Tuesday morning" and "multipart_message_6"
>>>>> which is from the dbmail-autotests imported on my IMAP
>>>>> seems to get most likely broken, but others to randomly
>>>>>
>>>>> "Ostrava Tuesday morning is the one with context to
>>>>> "8a042214ae1d120581740020f4e73c3cf8d3a6c0" and was the
>>>>> first customer complaint
>>>>>
>>>>> Regards Harry
>>>>>
>>>>> Am 20.07.2014 12:30, schrieb Paul J Stevens:
>>>>>> It was a buffer overrun which has now been fixed to the
>>>>>> best of my abilities.
>>>>>>
>>>>>> I've done another round of coverity analysis on the
>>>>>> master branch, and this doesn't trigger any new warnings
>>>>>> with regard to mentioned fix which is also in the
>>>>>> dbmail_3_1 branch.
>>>>>>
>>>>>> I've just tagged and thereby released 3.1.16 which
>>>>>> contains just this fix.
>>>>>>
>>>>>> Being on vacation, I don't have full access to my usual
>>>>>> release scripts (home-network is unreachable), so a tag
>>>>>> will have to do.
>>>>>>
>>>>>> But I've also uploaded the tar file seperately to
>>>>>> www.dbmail.org, so all the usual download locations
>>>>>> should work.
>>>>>>
>>>>>> On 18-07-14 23:05, Reindl Harald (mobile) wrote:
>>>>>>> Interesting - your attachment showed exactly the same
>>>>>>> bug I have only randomly for whatever type of messages
>>>>>>> and only under load
>>>>>>>
>>>>>>> I would sell my soul for a predictable reproducer
>>>>>>> because in that case debug logging / gdb / strace could
>>>>>>> shed some light and since it's always the same want
>>>>>>> finally happens it's pretty sure only a few lines of
>>>>>>> code in dbmail to fix it
>>>>>>>
>>>>>>> I am still sure it's exactly the same code path in both
>>>>>>> of our cases and only the trigger to went that path is
>>>>>>> different
>>>>>>>
>>>>>>> Maybe Paul has some idea how you can provide debug
>>>>>>> info's since you are able to isolate what happens while
>>>>>>> my trigger ends in a large mess of mixed debug logs
>>>>>>>
>>>>>>>
>>>>>>> -------- Ursprüngliche Nachricht -------- Von:
>>>>>>> Guilherme Souza <gslsouza@gmail.com> Gesendet: 18. Juli
>>>>>>> 2014 21:20:35 MESZ An: DBMail mailinglist
>>>>>>> <dbmail@dbmail.org> Betreff: Re: [Dbmail] corrupted
>>>>>>> multi-mime messages
>>>>>>>
>>>>>>> I was no able to reply to your message. In my case is
>>>>>>> not random.
>>>>>>>
>>>>>>> Every message html formatted with an attached file sent
>>>>>>> by ensignia webmail shows the same missing boundaries.
>>>>>>>
>>>>>>> Every message sent by roundcube or thunderbird shows
>>>>>>> correctly.
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 18, 2014 at 2:31 PM, Reindl Harald (mobile)
>>>>>>> <h.reindl@thelounge.net> wrote:
>>>>>>>> That is exactly what i reported a few days ago -
>>>>>>>> that's for sure a race condition in reconstruction
>>>>>>>> because i get predictable identical broken messages
>>>>>>>> in a loop while mixed load on my testserver - at the
>>>>>>>> same machine 1000 loops receive the same message
>>>>>>>> without other clients no broken one
>
>
>
> _______________________________________________ DBmail mailing
> list DBmail@dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>

- --
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, github, linkedin
www.nfg.nl/info@nfg.nl/+31.85.877.99.97
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlPMDs8ACgkQ8iITvBH4zTHQuwCeIktGVQN1UEdB9iZN0+KsZ10O
WZQAmgKiyMDSiPgOwc5RsRiIX8SYZnQM
=qjVR
-----END PGP SIGNATURE-----
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
Re: better but still errors -> Re: 3.1.16 regression -> Re: DBMail 3.1.16 released (Re: corrupted multi-mime messages) [ In reply to ]
Am 20.07.2014 20:47, schrieb Paul J Stevens:
> If you can, test the master-HEAD. It contains all coverity fixes I
> could find. I will not run coverity against 3.1

the problem is that i can't push MASTER to production and the
problems are affecting dbmail 3.1 stable

i did not mean fix anything coverity lists but maybe
with the diffs what happening to messages coverity
can give a educated hint

since when it happens it's always the same result
having lines with "--" instead "-- mimeid" it is
likely a single specific bug :-(

> On 20-07-14 19:06, Reindl Harald wrote:
>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e is *way better* but still
>> not error free, can you please fire coverity again because i think
>> it's more effective than me building and testing a lot of versions
>> over hours - smells like one remaining buffer overflow / underrun
>
>> * "Ostrava Tuesday morning" no longer broken * "Panel: E-Mail-Debug
>> - IMAP-POP3-Download" 4 out of 1000 broken * i have attached the 4
>> diff-files * all 4 still always identical broken * two times POP3 *
>> two times IMAP
>
>> additional info:
>
>> * "Ostrava Tuesday morning" with 23,40 KB *no error* on 100 loops *
>> "Panel: E-Mail-Debug - IMAP-POP3-Download" is 380 KB large
>
>> so it seems larger messages are more likely to reprocude the
>> problem, but that's a error rate which let me put
>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e in production because
>> better than current 8a042214ae1d120581740020f4e73c3cf8d3a6c0
>> ___________________________________________________
>
>> my stress-test complained within 25 minutes 5 times
>
>> "multipart_message_3", "message_rfc822" and "multipart_message_2"
>> are from the dbmail-testsuite, "Test 5" is one of my past problem
>> cases
>
>> Jul 20 18:37:13 POP3 24: EXPECTED:
>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>
>> Jul 20 18:41:48 POP3 28: EXPECTED:
>> 60dd43bb886f6c47d89b5dc3ab3f7f764b68eb2e / FOUND:
>> b99c389d3479d8d407d43045bcd99219e51a8977 (message_rfc822)
>
>> Jul 20 18:45:53 POP3 25: EXPECTED:
>> 0e2409bcf6921c5da801e1bf4ca4f57be9f07775 / FOUND:
>> 3ef1aef632e08eef66d77e5a97091797640f3192 (multipart_message_2)
>
>> Jul 20 18:47:00 POP3 01: EXPECTED:
>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>
>> Jul 20 18:54:47 POP3 01: EXPECTED:
>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>
>> Am 20.07.2014 17:36, schrieb Reindl Harald (mobile):
>>> I will give it a try after back at home
>>>
>>> if the other reconstruct problem under load still exists i try to
>>> get back release version to release version until it is away and
>>> want the commits after upwards
>>>
>>> that sort of regressions was solved for sure because we both
>>> spent a lot of hours last year and finally there where load tests
>>> over hours without a single mismatch
>>>
>>> However, if we really find a guilty commit the interesting
>>> question will be what assumption leaded to that exactly code and
>>> how to prevent similar bugs - Autotest are great but can't replay
>>> real world mixed load, even my stress tests are synthetic at the
>>> end of the day and can't really simulate 50 different clients at
>>> the same time acting on completely different messages and what no
>>> test ever will reproduce is the memory state with possible leaks
>>> part of the game after several days of running
>>>
>>> the race bug on the other hand seems to be worst after restart
>>> the service - very strange
>>>
>>>
>>> -------- Ursprüngliche Nachricht -------- Von: Paul J Stevens
>>> <paul@nfg.nl> Gesendet: 20. Juli 2014 17:00:15 MESZ An: DBMail
>>> mailinglist <dbmail@dbmail.org> Betreff: Re: [Dbmail] 3.1.16
>>> regression -> Re: DBMail 3.1.16 released (Re: corrupted
>>> multi-mime messages)
>>>
>>> I can't believe this passed the tests, since this exact message
>>> is part of the test-suite. Seems the test itself was broken :-(
>>>
>>> I just pushed a fix. *Please* test!
>>>
>>> On 20-07-14 14:12, Reindl Harald wrote:
>>>> CAUTION
>>>
>>>> 3.1.16 at least has a regression in case of the commit below
>>>> that's a different problem and the testmessage with downgrading
>>>> is reconstructed proper
>>>> http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_1&id=8a042214ae1d120581740020f4e73c3cf8d3a6c0
>>>
>>>>
>>>>
> however, the mime-part-id problem is still the same under load
>>>
>>>> i sent Paul some debug data offlist because they contain real
>>>> messages i am allowed by the mailowner to forward only that
>>>> way, not in public
>>>
>>>> Regards Harry
>>>
>>>> Am 20.07.2014 13:38, schrieb Reindl Harald:
>>>>> *attached* a compressed example *diff* broken / non-broken
>>>>> copy while compare the same mail IMAP / POP3 in a loop
>>>>>
>>>>> 33 out of 1000 always identically broken
>>>>>
>>>>> it don't matter on which test message i start the loop the
>>>>> result is the same in case of multi-mime mails
>>>>>
>>>>> fact is there are times while running mixed load for a minute
>>>>> not a single one get broken and in random time windows any
>>>>> multimime message for some seconds is damaged the same way
>>>>> _______________________________________________________________
>>>>>
>>>>>
>>>>>
> that below is the temp-folder where the loop stores diffs look at
>>>>> the number at the end, that's the loop-comuter
>>>>>
>>>>> at the begin of the loop a horrible result with 19 broken
>>>>> copies in serie and later 118-125 are again 8 in serie and
>>>>> they all have the same difference as "example-diff.zip"
>>>>>
>>>>> the mixed load now get "multipart_message_3" instead
>>>>> "multipart_message_6" most of the time broken
>>>>> _______________________________________________________________
>>>>>
>>>>>
> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_002.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_003.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_004.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_005.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_006.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_007.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_009.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_010.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_011.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_012.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_013.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_014.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_015.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_016.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_017.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_018.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_019.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_020.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_021.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_084.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_089.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_118.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_119.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_120.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_121.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_122.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_123.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_124.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_125.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_335.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_364.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_380.diff
>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_707.diff
>>>>> _______________________________________________
>>>>>
>>>>> Jul 20 13:26:56 POP3 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Jul 20 13:26:56 IMAP 24: EXPECTED:
>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>> (multipart_message_3) Jul 20 13:26:56 IMAP 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Jul 20 13:27:23 POP3 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Jul 20 13:27:23 IMAP 24: EXPECTED:
>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>> (multipart_message_3) Jul 20 13:27:23 IMAP 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Jul 20 13:27:52 POP3 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Jul 20 13:27:52 IMAP 24: EXPECTED:
>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>> (multipart_message_3) Jul 20 13:27:52 IMAP 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Jul 20 13:28:15 POP3 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Jul 20 13:28:15 IMAP 24: EXPECTED:
>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>> (multipart_message_3) Jul 20 13:28:15 IMAP 33: EXPECTED:
>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>> morning) Am 20.07.2014 12:56, schrieb Reindl Harald:
>>>>>> Hi Paul
>>>>>>
>>>>>> sorry but still unconfirmed
>>>>>>
>>>>>> see attached screenshot of debug-interface as well as
>>>>>> "stress-test.txt" which normally should not output
>>>>>> anything because it prcatically makes a lot of noise while
>>>>>> check again and aagin the same messages against known
>>>>>> checksums
>>>>>>
>>>>>> the "Ostrava Tuesday morning" and "multipart_message_6"
>>>>>> which is from the dbmail-autotests imported on my IMAP
>>>>>> seems to get most likely broken, but others to randomly
>>>>>>
>>>>>> "Ostrava Tuesday morning is the one with context to
>>>>>> "8a042214ae1d120581740020f4e73c3cf8d3a6c0" and was the
>>>>>> first customer complaint
>>>>>>
>>>>>> Regards Harry
>>>>>>
>>>>>> Am 20.07.2014 12:30, schrieb Paul J Stevens:
>>>>>>> It was a buffer overrun which has now been fixed to the
>>>>>>> best of my abilities.
>>>>>>>
>>>>>>> I've done another round of coverity analysis on the
>>>>>>> master branch, and this doesn't trigger any new warnings
>>>>>>> with regard to mentioned fix which is also in the
>>>>>>> dbmail_3_1 branch.
>>>>>>>
>>>>>>> I've just tagged and thereby released 3.1.16 which
>>>>>>> contains just this fix.
>>>>>>>
>>>>>>> Being on vacation, I don't have full access to my usual
>>>>>>> release scripts (home-network is unreachable), so a tag
>>>>>>> will have to do.
>>>>>>>
>>>>>>> But I've also uploaded the tar file seperately to
>>>>>>> www.dbmail.org, so all the usual download locations
>>>>>>> should work.
>>>>>>>
>>>>>>> On 18-07-14 23:05, Reindl Harald (mobile) wrote:
>>>>>>>> Interesting - your attachment showed exactly the same
>>>>>>>> bug I have only randomly for whatever type of messages
>>>>>>>> and only under load
>>>>>>>>
>>>>>>>> I would sell my soul for a predictable reproducer
>>>>>>>> because in that case debug logging / gdb / strace could
>>>>>>>> shed some light and since it's always the same want
>>>>>>>> finally happens it's pretty sure only a few lines of
>>>>>>>> code in dbmail to fix it
>>>>>>>>
>>>>>>>> I am still sure it's exactly the same code path in both
>>>>>>>> of our cases and only the trigger to went that path is
>>>>>>>> different
>>>>>>>>
>>>>>>>> Maybe Paul has some idea how you can provide debug
>>>>>>>> info's since you are able to isolate what happens while
>>>>>>>> my trigger ends in a large mess of mixed debug logs
>>>>>>>>
>>>>>>>>
>>>>>>>> -------- Ursprüngliche Nachricht -------- Von:
>>>>>>>> Guilherme Souza <gslsouza@gmail.com> Gesendet: 18. Juli
>>>>>>>> 2014 21:20:35 MESZ An: DBMail mailinglist
>>>>>>>> <dbmail@dbmail.org> Betreff: Re: [Dbmail] corrupted
>>>>>>>> multi-mime messages
>>>>>>>>
>>>>>>>> I was no able to reply to your message. In my case is
>>>>>>>> not random.
>>>>>>>>
>>>>>>>> Every message html formatted with an attached file sent
>>>>>>>> by ensignia webmail shows the same missing boundaries.
>>>>>>>>
>>>>>>>> Every message sent by roundcube or thunderbird shows
>>>>>>>> correctly.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 18, 2014 at 2:31 PM, Reindl Harald (mobile)
>>>>>>>> <h.reindl@thelounge.net> wrote:
>>>>>>>>> That is exactly what i reported a few days ago -
>>>>>>>>> that's for sure a race condition in reconstruction
>>>>>>>>> because i get predictable identical broken messages
>>>>>>>>> in a loop while mixed load on my testserver - at the
>>>>>>>>> same machine 1000 loops receive the same message
>>>>>>>>> without other clients no broken one
Re: better but still errors -> Re: 3.1.16 regression -> Re: DBMail 3.1.16 released (Re: corrupted multi-mime messages) [ In reply to ]
i am out of ideas, customer complaint that two messages
with a pdf invoice are broken and i really don't know
that to answer - i can't tell a customer that get
incoming mails undamaged is some sort of luck

that's exactly the reconstruct problem "--" without mime-id

Date: Mon, 21 Jul 2014 10:43:17 +0200
Date: Mon, 21 Jul 2014 10:43:33 +0200
_______________________________________________________

--
Content-Type: text/html;

--
Content-Type: application/pdf;
_______________________________________________________

Am 20.07.2014 20:56, schrieb Reindl Harald:
> Am 20.07.2014 20:47, schrieb Paul J Stevens:
>> If you can, test the master-HEAD. It contains all coverity fixes I
>> could find. I will not run coverity against 3.1
>
> the problem is that i can't push MASTER to production and the
> problems are affecting dbmail 3.1 stable
>
> i did not mean fix anything coverity lists but maybe
> with the diffs what happening to messages coverity
> can give a educated hint
>
> since when it happens it's always the same result
> having lines with "--" instead "-- mimeid" it is
> likely a single specific bug :-(
>
>> On 20-07-14 19:06, Reindl Harald wrote:
>>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e is *way better* but still
>>> not error free, can you please fire coverity again because i think
>>> it's more effective than me building and testing a lot of versions
>>> over hours - smells like one remaining buffer overflow / underrun
>>
>>> * "Ostrava Tuesday morning" no longer broken * "Panel: E-Mail-Debug
>>> - IMAP-POP3-Download" 4 out of 1000 broken * i have attached the 4
>>> diff-files * all 4 still always identical broken * two times POP3 *
>>> two times IMAP
>>
>>> additional info:
>>
>>> * "Ostrava Tuesday morning" with 23,40 KB *no error* on 100 loops *
>>> "Panel: E-Mail-Debug - IMAP-POP3-Download" is 380 KB large
>>
>>> so it seems larger messages are more likely to reprocude the
>>> problem, but that's a error rate which let me put
>>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e in production because
>>> better than current 8a042214ae1d120581740020f4e73c3cf8d3a6c0
>>> ___________________________________________________
>>
>>> my stress-test complained within 25 minutes 5 times
>>
>>> "multipart_message_3", "message_rfc822" and "multipart_message_2"
>>> are from the dbmail-testsuite, "Test 5" is one of my past problem
>>> cases
>>
>>> Jul 20 18:37:13 POP3 24: EXPECTED:
>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>>
>>> Jul 20 18:41:48 POP3 28: EXPECTED:
>>> 60dd43bb886f6c47d89b5dc3ab3f7f764b68eb2e / FOUND:
>>> b99c389d3479d8d407d43045bcd99219e51a8977 (message_rfc822)
>>
>>> Jul 20 18:45:53 POP3 25: EXPECTED:
>>> 0e2409bcf6921c5da801e1bf4ca4f57be9f07775 / FOUND:
>>> 3ef1aef632e08eef66d77e5a97091797640f3192 (multipart_message_2)
>>
>>> Jul 20 18:47:00 POP3 01: EXPECTED:
>>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
>>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>>
>>> Jul 20 18:54:47 POP3 01: EXPECTED:
>>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
>>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>>
>>> Am 20.07.2014 17:36, schrieb Reindl Harald (mobile):
>>>> I will give it a try after back at home
>>>>
>>>> if the other reconstruct problem under load still exists i try to
>>>> get back release version to release version until it is away and
>>>> want the commits after upwards
>>>>
>>>> that sort of regressions was solved for sure because we both
>>>> spent a lot of hours last year and finally there where load tests
>>>> over hours without a single mismatch
>>>>
>>>> However, if we really find a guilty commit the interesting
>>>> question will be what assumption leaded to that exactly code and
>>>> how to prevent similar bugs - Autotest are great but can't replay
>>>> real world mixed load, even my stress tests are synthetic at the
>>>> end of the day and can't really simulate 50 different clients at
>>>> the same time acting on completely different messages and what no
>>>> test ever will reproduce is the memory state with possible leaks
>>>> part of the game after several days of running
>>>>
>>>> the race bug on the other hand seems to be worst after restart
>>>> the service - very strange
>>>>
>>>>
>>>> -------- Ursprüngliche Nachricht -------- Von: Paul J Stevens
>>>> <paul@nfg.nl> Gesendet: 20. Juli 2014 17:00:15 MESZ An: DBMail
>>>> mailinglist <dbmail@dbmail.org> Betreff: Re: [Dbmail] 3.1.16
>>>> regression -> Re: DBMail 3.1.16 released (Re: corrupted
>>>> multi-mime messages)
>>>>
>>>> I can't believe this passed the tests, since this exact message
>>>> is part of the test-suite. Seems the test itself was broken :-(
>>>>
>>>> I just pushed a fix. *Please* test!
>>>>
>>>> On 20-07-14 14:12, Reindl Harald wrote:
>>>>> CAUTION
>>>>
>>>>> 3.1.16 at least has a regression in case of the commit below
>>>>> that's a different problem and the testmessage with downgrading
>>>>> is reconstructed proper
>>>>> http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_1&id=8a042214ae1d120581740020f4e73c3cf8d3a6c0
>>>>
>>>>>
>>>>>
>> however, the mime-part-id problem is still the same under load
>>>>
>>>>> i sent Paul some debug data offlist because they contain real
>>>>> messages i am allowed by the mailowner to forward only that
>>>>> way, not in public
>>>>
>>>>> Regards Harry
>>>>
>>>>> Am 20.07.2014 13:38, schrieb Reindl Harald:
>>>>>> *attached* a compressed example *diff* broken / non-broken
>>>>>> copy while compare the same mail IMAP / POP3 in a loop
>>>>>>
>>>>>> 33 out of 1000 always identically broken
>>>>>>
>>>>>> it don't matter on which test message i start the loop the
>>>>>> result is the same in case of multi-mime mails
>>>>>>
>>>>>> fact is there are times while running mixed load for a minute
>>>>>> not a single one get broken and in random time windows any
>>>>>> multimime message for some seconds is damaged the same way
>>>>>> _______________________________________________________________
>>>>>>
>>>>>>
>>>>>>
>> that below is the temp-folder where the loop stores diffs look at
>>>>>> the number at the end, that's the loop-comuter
>>>>>>
>>>>>> at the begin of the loop a horrible result with 19 broken
>>>>>> copies in serie and later 118-125 are again 8 in serie and
>>>>>> they all have the same difference as "example-diff.zip"
>>>>>>
>>>>>> the mixed load now get "multipart_message_3" instead
>>>>>> "multipart_message_6" most of the time broken
>>>>>> _______________________________________________________________
>>>>>>
>>>>>>
>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_002.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_003.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_004.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_005.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_006.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_007.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_009.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_010.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_011.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_012.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_013.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_014.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_015.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_016.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_017.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_018.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_019.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_020.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_021.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_084.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_089.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_118.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_119.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_120.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_121.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_122.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_123.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_124.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_125.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_335.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_364.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_380.diff
>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_707.diff
>>>>>> _______________________________________________
>>>>>>
>>>>>> Jul 20 13:26:56 POP3 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Jul 20 13:26:56 IMAP 24: EXPECTED:
>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>> (multipart_message_3) Jul 20 13:26:56 IMAP 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Jul 20 13:27:23 POP3 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Jul 20 13:27:23 IMAP 24: EXPECTED:
>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>> (multipart_message_3) Jul 20 13:27:23 IMAP 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Jul 20 13:27:52 POP3 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Jul 20 13:27:52 IMAP 24: EXPECTED:
>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>> (multipart_message_3) Jul 20 13:27:52 IMAP 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Jul 20 13:28:15 POP3 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Jul 20 13:28:15 IMAP 24: EXPECTED:
>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>> (multipart_message_3) Jul 20 13:28:15 IMAP 33: EXPECTED:
>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>> morning) Am 20.07.2014 12:56, schrieb Reindl Harald:
>>>>>>> Hi Paul
>>>>>>>
>>>>>>> sorry but still unconfirmed
>>>>>>>
>>>>>>> see attached screenshot of debug-interface as well as
>>>>>>> "stress-test.txt" which normally should not output
>>>>>>> anything because it prcatically makes a lot of noise while
>>>>>>> check again and aagin the same messages against known
>>>>>>> checksums
>>>>>>>
>>>>>>> the "Ostrava Tuesday morning" and "multipart_message_6"
>>>>>>> which is from the dbmail-autotests imported on my IMAP
>>>>>>> seems to get most likely broken, but others to randomly
>>>>>>>
>>>>>>> "Ostrava Tuesday morning is the one with context to
>>>>>>> "8a042214ae1d120581740020f4e73c3cf8d3a6c0" and was the
>>>>>>> first customer complaint
>>>>>>>
>>>>>>> Regards Harry
>>>>>>>
>>>>>>> Am 20.07.2014 12:30, schrieb Paul J Stevens:
>>>>>>>> It was a buffer overrun which has now been fixed to the
>>>>>>>> best of my abilities.
>>>>>>>>
>>>>>>>> I've done another round of coverity analysis on the
>>>>>>>> master branch, and this doesn't trigger any new warnings
>>>>>>>> with regard to mentioned fix which is also in the
>>>>>>>> dbmail_3_1 branch.
>>>>>>>>
>>>>>>>> I've just tagged and thereby released 3.1.16 which
>>>>>>>> contains just this fix.
>>>>>>>>
>>>>>>>> Being on vacation, I don't have full access to my usual
>>>>>>>> release scripts (home-network is unreachable), so a tag
>>>>>>>> will have to do.
>>>>>>>>
>>>>>>>> But I've also uploaded the tar file seperately to
>>>>>>>> www.dbmail.org, so all the usual download locations
>>>>>>>> should work.
>>>>>>>>
>>>>>>>> On 18-07-14 23:05, Reindl Harald (mobile) wrote:
>>>>>>>>> Interesting - your attachment showed exactly the same
>>>>>>>>> bug I have only randomly for whatever type of messages
>>>>>>>>> and only under load
>>>>>>>>>
>>>>>>>>> I would sell my soul for a predictable reproducer
>>>>>>>>> because in that case debug logging / gdb / strace could
>>>>>>>>> shed some light and since it's always the same want
>>>>>>>>> finally happens it's pretty sure only a few lines of
>>>>>>>>> code in dbmail to fix it
>>>>>>>>>
>>>>>>>>> I am still sure it's exactly the same code path in both
>>>>>>>>> of our cases and only the trigger to went that path is
>>>>>>>>> different
>>>>>>>>>
>>>>>>>>> Maybe Paul has some idea how you can provide debug
>>>>>>>>> info's since you are able to isolate what happens while
>>>>>>>>> my trigger ends in a large mess of mixed debug logs
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -------- Ursprüngliche Nachricht -------- Von:
>>>>>>>>> Guilherme Souza <gslsouza@gmail.com> Gesendet: 18. Juli
>>>>>>>>> 2014 21:20:35 MESZ An: DBMail mailinglist
>>>>>>>>> <dbmail@dbmail.org> Betreff: Re: [Dbmail] corrupted
>>>>>>>>> multi-mime messages
>>>>>>>>>
>>>>>>>>> I was no able to reply to your message. In my case is
>>>>>>>>> not random.
>>>>>>>>>
>>>>>>>>> Every message html formatted with an attached file sent
>>>>>>>>> by ensignia webmail shows the same missing boundaries.
>>>>>>>>>
>>>>>>>>> Every message sent by roundcube or thunderbird shows
>>>>>>>>> correctly.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 18, 2014 at 2:31 PM, Reindl Harald (mobile)
>>>>>>>>> <h.reindl@thelounge.net> wrote:
>>>>>>>>>> That is exactly what i reported a few days ago -
>>>>>>>>>> that's for sure a race condition in reconstruction
>>>>>>>>>> because i get predictable identical broken messages
>>>>>>>>>> in a loop while mixed load on my testserver - at the
>>>>>>>>>> same machine 1000 loops receive the same message
>>>>>>>>>> without other clients no broken one
Re: better but still errors -> Re: 3.1.16 regression -> Re: DBMail 3.1.16 released (Re: corrupted multi-mime messages) [ In reply to ]
another complaint 5 minutes ago

"i received last week some empty messages, complained
by the sender and they are saying it's in the responsibility
of the receiver"

don't get me wrong but fixes like below in that
context let me despair :-(

- strncat(boundary, s, buflen);
+ strncat(boundary, s, min(i, buflen));

Am 21.07.2014 13:01, schrieb Reindl Harald:
> i am out of ideas, customer complaint that two messages
> with a pdf invoice are broken and i really don't know
> that to answer - i can't tell a customer that get
> incoming mails undamaged is some sort of luck
>
> that's exactly the reconstruct problem "--" without mime-id
>
> Date: Mon, 21 Jul 2014 10:43:17 +0200
> Date: Mon, 21 Jul 2014 10:43:33 +0200
> _______________________________________________________
>
> --
> Content-Type: text/html;
>
> --
> Content-Type: application/pdf;
> _______________________________________________________
>
> Am 20.07.2014 20:56, schrieb Reindl Harald:
>> Am 20.07.2014 20:47, schrieb Paul J Stevens:
>>> If you can, test the master-HEAD. It contains all coverity fixes I
>>> could find. I will not run coverity against 3.1
>>
>> the problem is that i can't push MASTER to production and the
>> problems are affecting dbmail 3.1 stable
>>
>> i did not mean fix anything coverity lists but maybe
>> with the diffs what happening to messages coverity
>> can give a educated hint
>>
>> since when it happens it's always the same result
>> having lines with "--" instead "-- mimeid" it is
>> likely a single specific bug :-(
>>
>>> On 20-07-14 19:06, Reindl Harald wrote:
>>>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e is *way better* but still
>>>> not error free, can you please fire coverity again because i think
>>>> it's more effective than me building and testing a lot of versions
>>>> over hours - smells like one remaining buffer overflow / underrun
>>>
>>>> * "Ostrava Tuesday morning" no longer broken * "Panel: E-Mail-Debug
>>>> - IMAP-POP3-Download" 4 out of 1000 broken * i have attached the 4
>>>> diff-files * all 4 still always identical broken * two times POP3 *
>>>> two times IMAP
>>>
>>>> additional info:
>>>
>>>> * "Ostrava Tuesday morning" with 23,40 KB *no error* on 100 loops *
>>>> "Panel: E-Mail-Debug - IMAP-POP3-Download" is 380 KB large
>>>
>>>> so it seems larger messages are more likely to reprocude the
>>>> problem, but that's a error rate which let me put
>>>> 77d17ca7724c32dba27eb187d4abfa129ccf6c4e in production because
>>>> better than current 8a042214ae1d120581740020f4e73c3cf8d3a6c0
>>>> ___________________________________________________
>>>
>>>> my stress-test complained within 25 minutes 5 times
>>>
>>>> "multipart_message_3", "message_rfc822" and "multipart_message_2"
>>>> are from the dbmail-testsuite, "Test 5" is one of my past problem
>>>> cases
>>>
>>>> Jul 20 18:37:13 POP3 24: EXPECTED:
>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>> 97039dbf12052e527808ca16188971a405fcb529 (multipart_message_3)
>>>
>>>> Jul 20 18:41:48 POP3 28: EXPECTED:
>>>> 60dd43bb886f6c47d89b5dc3ab3f7f764b68eb2e / FOUND:
>>>> b99c389d3479d8d407d43045bcd99219e51a8977 (message_rfc822)
>>>
>>>> Jul 20 18:45:53 POP3 25: EXPECTED:
>>>> 0e2409bcf6921c5da801e1bf4ca4f57be9f07775 / FOUND:
>>>> 3ef1aef632e08eef66d77e5a97091797640f3192 (multipart_message_2)
>>>
>>>> Jul 20 18:47:00 POP3 01: EXPECTED:
>>>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
>>>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>>>
>>>> Jul 20 18:54:47 POP3 01: EXPECTED:
>>>> 04073992ce1c1e75d9bd88e0388486dcc80df627 / FOUND:
>>>> af6836092d12edbb8be1396ad5af08c804fad2fb (Test 5)
>>>
>>>> Am 20.07.2014 17:36, schrieb Reindl Harald (mobile):
>>>>> I will give it a try after back at home
>>>>>
>>>>> if the other reconstruct problem under load still exists i try to
>>>>> get back release version to release version until it is away and
>>>>> want the commits after upwards
>>>>>
>>>>> that sort of regressions was solved for sure because we both
>>>>> spent a lot of hours last year and finally there where load tests
>>>>> over hours without a single mismatch
>>>>>
>>>>> However, if we really find a guilty commit the interesting
>>>>> question will be what assumption leaded to that exactly code and
>>>>> how to prevent similar bugs - Autotest are great but can't replay
>>>>> real world mixed load, even my stress tests are synthetic at the
>>>>> end of the day and can't really simulate 50 different clients at
>>>>> the same time acting on completely different messages and what no
>>>>> test ever will reproduce is the memory state with possible leaks
>>>>> part of the game after several days of running
>>>>>
>>>>> the race bug on the other hand seems to be worst after restart
>>>>> the service - very strange
>>>>>
>>>>>
>>>>> -------- Ursprüngliche Nachricht -------- Von: Paul J Stevens
>>>>> <paul@nfg.nl> Gesendet: 20. Juli 2014 17:00:15 MESZ An: DBMail
>>>>> mailinglist <dbmail@dbmail.org> Betreff: Re: [Dbmail] 3.1.16
>>>>> regression -> Re: DBMail 3.1.16 released (Re: corrupted
>>>>> multi-mime messages)
>>>>>
>>>>> I can't believe this passed the tests, since this exact message
>>>>> is part of the test-suite. Seems the test itself was broken :-(
>>>>>
>>>>> I just pushed a fix. *Please* test!
>>>>>
>>>>> On 20-07-14 14:12, Reindl Harald wrote:
>>>>>> CAUTION
>>>>>
>>>>>> 3.1.16 at least has a regression in case of the commit below
>>>>>> that's a different problem and the testmessage with downgrading
>>>>>> is reconstructed proper
>>>>>> http://git.dbmail.eu/paul/dbmail/commit/?h=dbmail_3_1&id=8a042214ae1d120581740020f4e73c3cf8d3a6c0
>>>>>
>>>>>>
>>>>>>
>>> however, the mime-part-id problem is still the same under load
>>>>>
>>>>>> i sent Paul some debug data offlist because they contain real
>>>>>> messages i am allowed by the mailowner to forward only that
>>>>>> way, not in public
>>>>>
>>>>>> Regards Harry
>>>>>
>>>>>> Am 20.07.2014 13:38, schrieb Reindl Harald:
>>>>>>> *attached* a compressed example *diff* broken / non-broken
>>>>>>> copy while compare the same mail IMAP / POP3 in a loop
>>>>>>>
>>>>>>> 33 out of 1000 always identically broken
>>>>>>>
>>>>>>> it don't matter on which test message i start the loop the
>>>>>>> result is the same in case of multi-mime mails
>>>>>>>
>>>>>>> fact is there are times while running mixed load for a minute
>>>>>>> not a single one get broken and in random time windows any
>>>>>>> multimime message for some seconds is damaged the same way
>>>>>>> _______________________________________________________________
>>>>>>>
>>>>>>>
>>>>>>>
>>> that below is the temp-folder where the loop stores diffs look at
>>>>>>> the number at the end, that's the loop-comuter
>>>>>>>
>>>>>>> at the begin of the loop a horrible result with 19 broken
>>>>>>> copies in serie and later 118-125 are again 8 in serie and
>>>>>>> they all have the same difference as "example-diff.zip"
>>>>>>>
>>>>>>> the mixed load now get "multipart_message_3" instead
>>>>>>> "multipart_message_6" most of the time broken
>>>>>>> _______________________________________________________________
>>>>>>>
>>>>>>>
>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_002.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_003.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_004.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_005.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_006.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_007.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_009.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_010.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_011.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_012.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_013.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_014.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_015.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_016.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_017.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_018.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_019.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_020.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_021.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_084.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_089.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_118.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_119.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_120.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_121.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_122.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_123.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_124.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_125.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_335.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_364.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_380.diff
>>>>>>> dbmail_c6442c520ba8b8a45f1751eb666caeec9ea1ed40_6_707.diff
>>>>>>> _______________________________________________
>>>>>>>
>>>>>>> Jul 20 13:26:56 POP3 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Jul 20 13:26:56 IMAP 24: EXPECTED:
>>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>>> (multipart_message_3) Jul 20 13:26:56 IMAP 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Jul 20 13:27:23 POP3 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Jul 20 13:27:23 IMAP 24: EXPECTED:
>>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>>> (multipart_message_3) Jul 20 13:27:23 IMAP 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Jul 20 13:27:52 POP3 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Jul 20 13:27:52 IMAP 24: EXPECTED:
>>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>>> (multipart_message_3) Jul 20 13:27:52 IMAP 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Jul 20 13:28:15 POP3 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Jul 20 13:28:15 IMAP 24: EXPECTED:
>>>>>>> 2210e351665b9771a6eefab2cb0c8bf43a04fcea / FOUND:
>>>>>>> 97039dbf12052e527808ca16188971a405fcb529
>>>>>>> (multipart_message_3) Jul 20 13:28:15 IMAP 33: EXPECTED:
>>>>>>> 16292abceedfc6f37707cb1adf35e1e1a46d7d2b / FOUND:
>>>>>>> bd17702e6872ebdd60c38f7ef439f9f30a5b9721 (Ostrava Tuesday
>>>>>>> morning) Am 20.07.2014 12:56, schrieb Reindl Harald:
>>>>>>>> Hi Paul
>>>>>>>>
>>>>>>>> sorry but still unconfirmed
>>>>>>>>
>>>>>>>> see attached screenshot of debug-interface as well as
>>>>>>>> "stress-test.txt" which normally should not output
>>>>>>>> anything because it prcatically makes a lot of noise while
>>>>>>>> check again and aagin the same messages against known
>>>>>>>> checksums
>>>>>>>>
>>>>>>>> the "Ostrava Tuesday morning" and "multipart_message_6"
>>>>>>>> which is from the dbmail-autotests imported on my IMAP
>>>>>>>> seems to get most likely broken, but others to randomly
>>>>>>>>
>>>>>>>> "Ostrava Tuesday morning is the one with context to
>>>>>>>> "8a042214ae1d120581740020f4e73c3cf8d3a6c0" and was the
>>>>>>>> first customer complaint
>>>>>>>>
>>>>>>>> Regards Harry
>>>>>>>>
>>>>>>>> Am 20.07.2014 12:30, schrieb Paul J Stevens:
>>>>>>>>> It was a buffer overrun which has now been fixed to the
>>>>>>>>> best of my abilities.
>>>>>>>>>
>>>>>>>>> I've done another round of coverity analysis on the
>>>>>>>>> master branch, and this doesn't trigger any new warnings
>>>>>>>>> with regard to mentioned fix which is also in the
>>>>>>>>> dbmail_3_1 branch.
>>>>>>>>>
>>>>>>>>> I've just tagged and thereby released 3.1.16 which
>>>>>>>>> contains just this fix.
>>>>>>>>>
>>>>>>>>> Being on vacation, I don't have full access to my usual
>>>>>>>>> release scripts (home-network is unreachable), so a tag
>>>>>>>>> will have to do.
>>>>>>>>>
>>>>>>>>> But I've also uploaded the tar file seperately to
>>>>>>>>> www.dbmail.org, so all the usual download locations
>>>>>>>>> should work.
>>>>>>>>>
>>>>>>>>> On 18-07-14 23:05, Reindl Harald (mobile) wrote:
>>>>>>>>>> Interesting - your attachment showed exactly the same
>>>>>>>>>> bug I have only randomly for whatever type of messages
>>>>>>>>>> and only under load
>>>>>>>>>>
>>>>>>>>>> I would sell my soul for a predictable reproducer
>>>>>>>>>> because in that case debug logging / gdb / strace could
>>>>>>>>>> shed some light and since it's always the same want
>>>>>>>>>> finally happens it's pretty sure only a few lines of
>>>>>>>>>> code in dbmail to fix it
>>>>>>>>>>
>>>>>>>>>> I am still sure it's exactly the same code path in both
>>>>>>>>>> of our cases and only the trigger to went that path is
>>>>>>>>>> different
>>>>>>>>>>
>>>>>>>>>> Maybe Paul has some idea how you can provide debug
>>>>>>>>>> info's since you are able to isolate what happens while
>>>>>>>>>> my trigger ends in a large mess of mixed debug logs
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -------- Ursprüngliche Nachricht -------- Von:
>>>>>>>>>> Guilherme Souza <gslsouza@gmail.com> Gesendet: 18. Juli
>>>>>>>>>> 2014 21:20:35 MESZ An: DBMail mailinglist
>>>>>>>>>> <dbmail@dbmail.org> Betreff: Re: [Dbmail] corrupted
>>>>>>>>>> multi-mime messages
>>>>>>>>>>
>>>>>>>>>> I was no able to reply to your message. In my case is
>>>>>>>>>> not random.
>>>>>>>>>>
>>>>>>>>>> Every message html formatted with an attached file sent
>>>>>>>>>> by ensignia webmail shows the same missing boundaries.
>>>>>>>>>>
>>>>>>>>>> Every message sent by roundcube or thunderbird shows
>>>>>>>>>> correctly.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jul 18, 2014 at 2:31 PM, Reindl Harald (mobile)
>>>>>>>>>> <h.reindl@thelounge.net> wrote:
>>>>>>>>>>> That is exactly what i reported a few days ago -
>>>>>>>>>>> that's for sure a race condition in reconstruction
>>>>>>>>>>> because i get predictable identical broken messages
>>>>>>>>>>> in a loop while mixed load on my testserver - at the
>>>>>>>>>>> same machine 1000 loops receive the same message
>>>>>>>>>>> without other clients no broken one