Mailing List Archive: RFE: checksum of original message in database

RFE: checksum of original message in database

Jul 20, 2014, 7:32 AM

Post #1 of 6 (1863 views)

Hi Paul

i would love a additional column in the message table containing the sha1 sum of the message before it get splitted in mime-parts or maybe better a own table with message-id and checksum and ignore that functionality if this table don't exist

that would offer a option for dbmail-util check reconstruction of current stored messages and by ignore old ones not having that value no compatibility break

maybe that verification could go in a small new binary outside dbmail-util with a few options like limit the number of verified messages and specify a date range and produce a csv like output with userid, mailboxid, date and subject and last but not least integrity check of a specific message id

that would greatly improve debugging / verification and in case of a complete raw message by a client that could also optionally calculated / verified and mismatch logged in the maillog to get some valuable Information in case of unpredictable bugs like the current reconstruction under load

surely it needs resources on the server, hence the idea of a separate table to enable that behaviour but at the end of the day it gives us the opportunity to have an insight in overall integrity and early detection of regressions, their impact, easy confirmation of bugfixes by knowing the hash of the original message, and messages known to be broken before the proposed fix

--

Reindl Harald (mobile)
the lounge interactive design GmbH
A-1060 Vienna, HofmÃ¼hlgasse 17
CTO / CISO / Software-Development
+43 (676) 40 221 40
http://www.thelounge.net
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: RFE: checksum of original message in database [ In reply to ]

leithner at itronic

Jul 21, 2014, 4:18 AM

Post #2 of 6 (1841 views)

Permalink

Hi,

I like this Idea and would go a step further, creating a checksum on an
incoming message and save it in the messages table. On reconstruction
check this checksum and don't push the mail to the customer if this
checksum does not match. Maybe retry the reconstruction several times and
log a message.

This shouldn't be too hard to implement and would prevent user complaints.

Paul, how long do you need and how much would it cost?

I'm pretty sure Haralds company and I would pay for this feature.

regards,

Harald

Am .07.2014, 16:32 Uhr, schrieb Reindl Harald (mobile)
<h.reindl@thelounge.net>:

> Hi Paul
>
> i would love a additional column in the message table containing the
> sha1 sum of the message before it get splitted in mime-parts or maybe
> better a own table with message-id and checksum and ignore that
> functionality if this table don't exist
>
> that would offer a option for dbmail-util check reconstruction of
> current stored messages and by ignore old ones not having that value no
> compatibility break
>
> maybe that verification could go in a small new binary outside
> dbmail-util with a few options like limit the number of verified
> messages and specify a date range and produce a csv like output with
> userid, mailboxid, date and subject and last but not least integrity
> check of a specific message id
>
> that would greatly improve debugging / verification and in case of a
> complete raw message by a client that could also optionally calculated /
> verified and mismatch logged in the maillog to get some valuable
> Information in case of unpredictable bugs like the current
> reconstruction under load
>
> surely it needs resources on the server, hence the idea of a separate
> table to enable that behaviour but at the end of the day it gives us the
> opportunity to have an insight in overall integrity and early detection
> of regressions, their impact, easy confirmation of bugfixes by knowing
> the hash of the original message, and messages known to be broken before
> the proposed fix
>
>
>
> --
>
> Reindl Harald (mobile)
> the lounge interactive design GmbH
> A-1060 Vienna, HofmÃ¼hlgasse 17
> CTO / CISO / Software-Development
> +43 (676) 40 221 40
> http://www.thelounge.net
> _______________________________________________
> DBmail mailing list
> DBmail@dbmail.org
> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

--
Harald Leithner

ITronic
Wiedner HauptstraÃŸe 120/5.1, 1050 Wien, Austria
Tel: +43-1-545 0 604
Fax: +43-1-786 23 88 26
Mobil: +43-699-123 78 4 78
Mail: leithner@itronic.at | itronic.at
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: RFE: checksum of original message in database [ In reply to ]

paul at nfg

Jul 21, 2014, 6:43 AM

Post #3 of 6 (1828 views)

Permalink

It's doable, and not very complex. I don't like doing database changes
on 3.1 though.

Good idea to include it in 3.2, I agree.

On 21-07-14 13:18, Harald Leithner wrote:
> Hi,
>
> I like this Idea and would go a step further, creating a checksum on an
> incoming message and save it in the messages table. On reconstruction
> check this checksum and don't push the mail to the customer if this
> checksum does not match. Maybe retry the reconstruction several times
> and log a message.
>
> This shouldn't be too hard to implement and would prevent user complaints.
>
> Paul, how long do you need and how much would it cost?
>
> I'm pretty sure Haralds company and I would pay for this feature.
>
> regards,
>
> Harald
>
> Am .07.2014, 16:32 Uhr, schrieb Reindl Harald (mobile)
> <h.reindl@thelounge.net>:
>
>> Hi Paul
>>
>> i would love a additional column in the message table containing the
>> sha1 sum of the message before it get splitted in mime-parts or maybe
>> better a own table with message-id and checksum and ignore that
>> functionality if this table don't exist
>>
>> that would offer a option for dbmail-util check reconstruction of
>> current stored messages and by ignore old ones not having that value
>> no compatibility break
>>
>> maybe that verification could go in a small new binary outside
>> dbmail-util with a few options like limit the number of verified
>> messages and specify a date range and produce a csv like output with
>> userid, mailboxid, date and subject and last but not least integrity
>> check of a specific message id
>>
>> that would greatly improve debugging / verification and in case of a
>> complete raw message by a client that could also optionally calculated
>> / verified and mismatch logged in the maillog to get some valuable
>> Information in case of unpredictable bugs like the current
>> reconstruction under load
>>
>> surely it needs resources on the server, hence the idea of a separate
>> table to enable that behaviour but at the end of the day it gives us
>> the opportunity to have an insight in overall integrity and early
>> detection of regressions, their impact, easy confirmation of bugfixes
>> by knowing the hash of the original message, and messages known to be
>> broken before the proposed fix
>>
>>
>>
>> --
>>
>> Reindl Harald (mobile)
>> the lounge interactive design GmbH
>> A-1060 Vienna, HofmÃ¼hlgasse 17
>> CTO / CISO / Software-Development
>> +43 (676) 40 221 40
>> http://www.thelounge.net
>> _______________________________________________
>> DBmail mailing list
>> DBmail@dbmail.org
>> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>
>

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, github, linkedin
www.nfg.nl/info@nfg.nl/+31.85.877.99.97
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: RFE: checksum of original message in database [ In reply to ]

leithner at itronic

Jul 21, 2014, 8:11 AM

Post #4 of 6 (1826 views)

Permalink

btw, I looked at the use of find_boundary and its only used on message
reconstruction.

For performance reasons, I would suggest to move the boundary id detection
to the db insert message function and create a new boundary field in
phymessages table.

Maybe with a migration option for dbmail-util to fill this column and
fallback on reconstruction if no boundary is in the database.

this would save some cpu cycles when the user fetches the mail.

what do you think?

Am .07.2014, 15:43 Uhr, schrieb Paul J Stevens <paul@nfg.nl>:

> It's doable, and not very complex. I don't like doing database changes
> on 3.1 though.
>
> Good idea to include it in 3.2, I agree.
>
> On 21-07-14 13:18, Harald Leithner wrote:
>> Hi,
>>
>> I like this Idea and would go a step further, creating a checksum on an
>> incoming message and save it in the messages table. On reconstruction
>> check this checksum and don't push the mail to the customer if this
>> checksum does not match. Maybe retry the reconstruction several times
>> and log a message.
>>
>> This shouldn't be too hard to implement and would prevent user
>> complaints.
>>
>> Paul, how long do you need and how much would it cost?
>>
>> I'm pretty sure Haralds company and I would pay for this feature.
>>
>> regards,
>>
>> Harald
>>
>> Am .07.2014, 16:32 Uhr, schrieb Reindl Harald (mobile)
>> <h.reindl@thelounge.net>:
>>
>>> Hi Paul
>>>
>>> i would love a additional column in the message table containing the
>>> sha1 sum of the message before it get splitted in mime-parts or maybe
>>> better a own table with message-id and checksum and ignore that
>>> functionality if this table don't exist
>>>
>>> that would offer a option for dbmail-util check reconstruction of
>>> current stored messages and by ignore old ones not having that value
>>> no compatibility break
>>>
>>> maybe that verification could go in a small new binary outside
>>> dbmail-util with a few options like limit the number of verified
>>> messages and specify a date range and produce a csv like output with
>>> userid, mailboxid, date and subject and last but not least integrity
>>> check of a specific message id
>>>
>>> that would greatly improve debugging / verification and in case of a
>>> complete raw message by a client that could also optionally calculated
>>> / verified and mismatch logged in the maillog to get some valuable
>>> Information in case of unpredictable bugs like the current
>>> reconstruction under load
>>>
>>> surely it needs resources on the server, hence the idea of a separate
>>> table to enable that behaviour but at the end of the day it gives us
>>> the opportunity to have an insight in overall integrity and early
>>> detection of regressions, their impact, easy confirmation of bugfixes
>>> by knowing the hash of the original message, and messages known to be
>>> broken before the proposed fix
>>>
>>>
>>>
>>> --
>>>
>>> Reindl Harald (mobile)
>>> the lounge interactive design GmbH
>>> A-1060 Vienna, HofmÃ¼hlgasse 17
>>> CTO / CISO / Software-Development
>>> +43 (676) 40 221 40
>>> http://www.thelounge.net
>>> _______________________________________________
>>> DBmail mailing list
>>> DBmail@dbmail.org
>>> http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail
>>
>>
>

--
Harald Leithner

ITronic
Wiedner HauptstraÃŸe 120/5.1, 1050 Wien, Austria
Tel: +43-1-545 0 604
Fax: +43-1-786 23 88 26
Mobil: +43-699-123 78 4 78
Mail: leithner@itronic.at | itronic.at
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: RFE: checksum of original message in database [ In reply to ]

paul at nfg

Jul 21, 2014, 8:31 AM

Post #5 of 6 (1826 views)

Permalink

On 21-07-14 17:11, Harald Leithner wrote:
> btw, I looked at the use of find_boundary and its only used on message
> reconstruction.

correct.

> For performance reasons, I would suggest to move the boundary id
> detection to the db insert message function and create a new boundary
> field in phymessages table.

Alas, not possible. Mime is a recursive system. Boundaries can occur
also in included messages, or messages included in included messages, etc.

--
________________________________________________________________
Paul J Stevens pjstevns @ gmail, twitter, github, linkedin
www.nfg.nl/info@nfg.nl/+31.85.877.99.97
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail

Re: RFE: checksum of original message in database [ In reply to ]

leithner at itronic

Jul 22, 2014, 12:44 AM

Post #6 of 6 (1834 views)

Permalink

Am .07.2014, 17:31 Uhr, schrieb Paul J Stevens <paul@nfg.nl>:

>
>
> On 21-07-14 17:11, Harald Leithner wrote:
>> btw, I looked at the use of find_boundary and its only used on message
>> reconstruction.
>
> correct.
>
>> For performance reasons, I would suggest to move the boundary id
>> detection to the db insert message function and create a new boundary
>> field in phymessages table.
>
> Alas, not possible. Mime is a recursive system. Boundaries can occur
> also in included messages, or messages included in included messages,
> etc.
>

ok, most messages would hit the cache, messages with messages would have
to use the current method. But on the other side it adds complexity and
more problems.

Its a bad idea, thx for comment.

--
Harald Leithner

ITronic
Wiedner HauptstraÃŸe 120/5.1, 1050 Wien, Austria
Tel: +43-1-545 0 604
Fax: +43-1-786 23 88 26
Mobil: +43-699-123 78 4 78
Mail: leithner@itronic.at | itronic.at
_______________________________________________
DBmail mailing list
DBmail@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail