A NOTE has been added to this issue.
======================================================================
http://www.dbmail.org/mantis/view.php?id=1020
======================================================================
Reported By: igorbelykh86
Assigned To:
======================================================================
Project: DBMail
Issue ID: 1020
Category: IMAP daemon
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 17-Aug-13 09:37 CEST
Last Modified: 12-Sep-14 09:39 CEST
======================================================================
Summary: dbmail 3.1.2. Imap server returns wrong decoded
header fields.
Description:
Hi
After upgrading the dbmail to 3.1.2 the IMAP server has began to return
wrong decoded email headers. I've thought that it does it only for old
emails which were been received before upgrading, but no. New mails have
the problem too. This problem appears only when application try to get a
header field. Example:
CLIENT:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {98}
From: Igor Belykh <myemail@address.com>
Subject: ????? ??????
To: Igor <anotheremail@address.com>
)
A003 OK UID FETCH completed
The subject is in russian. It is encoded correctly. If get full header of
the message the server returns:
....
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
....
what I can do to always get encoded header fields?
Thanks.
======================================================================
----------------------------------------------------------------------
(0003559) paul (administrator) - 17-Aug-13 10:33
http://www.dbmail.org/mantis/view.php?id=1020#c3559
----------------------------------------------------------------------
Question is: what is in the database? Was it during insertion or during
retrieval that something went wrong.
The message_idnr is 1017 in your case. Try following:
select m.message_idnr,n.headername,v.headervalue from dbmail_header h join
dbmail_headername n on h.headername_id=n.id join dbmail_headervalue v on
h.headervalue_id=v.id join dbmail_physmessage p on p.id=h.physmessage_id
join dbmail_messages m on m.physmessage_id=p.id where m.message_idnr=1017;
This should give you a dump of the cached headers. If your database is
UTF-8 encoded, and your terminal is as well, it should give you nicely
readable values.
Of course, the order of the headers returned is also wrong, but that was
fixed already and will be ok in 3.1.3.
----------------------------------------------------------------------
(0003560) igorbelykh86 (reporter) - 17-Aug-13 13:15
http://www.dbmail.org/mantis/view.php?id=1020#c3560
----------------------------------------------------------------------
Hi Paul,
All data is readable in the database. All tables is utf-8 encoded. And
system (and terminal too) default encoding is en_US.UTF-8.
Any idea?
----------------------------------------------------------------------
(0003561) igorbelykh86 (reporter) - 19-Aug-13 12:02
http://www.dbmail.org/mantis/view.php?id=1020#c3561
----------------------------------------------------------------------
Hi Paul,
I think that the problem is in receiving data from database? I've checked
the mysql server and mysql client settings. Server and client default
character set is utf8 too:
[client]
......
default-character-set=utf8
......
[mysql]
......
default-character-set=utf8
......
[mysqld]
......
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
......
What do you think what's the problem? Maybe some thing in dbmail settings
or mysql or something else?
----------------------------------------------------------------------
(0003562) jasb (reporter) - 19-Aug-13 12:04
http://www.dbmail.org/mantis/view.php?id=1020#c3562
----------------------------------------------------------------------
The [client] section of my.ini/cnf is for the mysql console client.. not
3rd party software.
----------------------------------------------------------------------
(0003563) igorbelykh86 (reporter) - 23-Sep-13 18:24
http://www.dbmail.org/mantis/view.php?id=1020#c3563
----------------------------------------------------------------------
just tried to comment line "encoding = utf8". Now it works fine.
CLIENT REQUEST:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER RESPONSE:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {119}
From: Igor Belykh <example@example.com>
Subject: =?iso-8859-5?b?vd7S3tUg39jh7Nze?=
To: Igor <igor@example.com>
)
A003 OK UID FETCH completed
How I understand these feader fields are received from source code of the
message.
----------------------------------------------------------------------
(0003564) igorbelykh86 (reporter) - 19-Aug-13 12:29
http://www.dbmail.org/mantis/view.php?id=1020#c3564
----------------------------------------------------------------------
no, in the source code the subject header field is different:
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
So the imap server works fine if the line "encoding = utf8" is commented.
----------------------------------------------------------------------
(0003565) paul (administrator) - 19-Aug-13 12:54
http://www.dbmail.org/mantis/view.php?id=1020#c3565
----------------------------------------------------------------------
the value of the 'encoding' parameter must match the collation of the
database. Apparently your database is not in UTF8.
try:
mysql dbmail
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | utf8_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
if the collation of the database and connection are *not* the same, you
need to set the encoding of the connection to match the collation of the
database:
mysql> SET NAMES koi8r;
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | koi8r_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
this means in this example the 'encoding' parameter should be 'koi8r';
Without the encoding paramater, it falls back to your system's default
locale, I syspect.
----------------------------------------------------------------------
(0003566) igorbelykh86 (reporter) - 19-Aug-13 13:07
http://www.dbmail.org/mantis/view.php?id=1020#c3566
----------------------------------------------------------------------
This is result of the query:
mysql> show variables like 'collation_%';
+----------------------+-----------------+
| Variable_name | Value |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
+----------------------+-----------------+
3 rows in set (0.00 sec)
Thus the encoding parameter should be 'utf8'. Right?
----------------------------------------------------------------------
(0003580) igorbelykh86 (reporter) - 21-Sep-13 19:44
http://www.dbmail.org/mantis/view.php?id=1020#c3580
----------------------------------------------------------------------
Hi Paul,
I've found one issue with encoding in data base. There is `sortfield` cell
in the `dbmail_headervalue` table. This cell has utf8_general_ci collation
but in rows I see following:
ßрøòõт ø÷ à þÑÂÑÂøø (Hello from
Russia)
instead
Привет из РоÑÑии (Hello from Russia)
But the `headervalue` cell contains the right value in right encoding
Regards,
Igor
----------------------------------------------------------------------
(0003581) Lionel (reporter) - 22-Sep-13 18:07
http://www.dbmail.org/mantis/view.php?id=1020#c3581
----------------------------------------------------------------------
Hi Paul,
The problem is not only with russian characters but with any language that
needs UTF-8 support.
I've done tests with russian, hebrew, arabic and greek characters, they
all have the same issue.
I suspect the problem to be in the place where the sortfield is being
created/processed when there is a new email because the sortfield is not
inserted in the database correctly. The headervalue is inserted correctly,
but not the sortfield. The way the sortfield appears in the database looks
like a typical utf-8 meta tag missing in html. However, I do not know where
in the src files this is handled.
Your input on this in order to fix it is welcomed as I think this is a
major issue for all dbmail users trying to run dbmail 3.x as with this
issue dbmail is totally useless for anyone receiving or sending emails with
characters other than english/dutch.
Thanks.
----------------------------------------------------------------------
(0003582) paul (administrator) - 23-Sep-13 16:44
http://www.dbmail.org/mantis/view.php?id=1020#c3582
----------------------------------------------------------------------
I wonder how does this affect the issue mentioned here, since the sortfield
only affects 'sorting'?
----------------------------------------------------------------------
(0003583) paul (administrator) - 23-Sep-13 16:48
http://www.dbmail.org/mantis/view.php?id=1020#c3583
----------------------------------------------------------------------
can you please provide a simple test message to verify?
----------------------------------------------------------------------
(0003584) Lionel (reporter) - 23-Sep-13 16:49
http://www.dbmail.org/mantis/view.php?id=1020#c3584
----------------------------------------------------------------------
yes you are right. We (me & Igor) figured this out yesterday while looking
at the source files.
However, we did not find a fix for the issue regarding the headers and the
subject that is totally unreadable (???? ???? ??????). Any idea what might
be the issue? or where we should be looking at to try to figure it out?
Thanks.
----------------------------------------------------------------------
(0003585) Lionel (reporter) - 23-Sep-13 16:52
http://www.dbmail.org/mantis/view.php?id=1020#c3585
----------------------------------------------------------------------
just send an email and put in the subject the following: Привет из
РоÑÑии (Hello from Russia)
Same problem as well with hebrew: ×©×œ×•× ×‘×¢×‘×¨×™×ª (Hello in Hebrew)
In arabic: مرØبا باللغة العربية (Hellow in Arabic)
Thanks.
----------------------------------------------------------------------
(0003586) paul (administrator) - 23-Sep-13 16:57
http://www.dbmail.org/mantis/view.php?id=1020#c3586
----------------------------------------------------------------------
ignore that last message
----------------------------------------------------------------------
(0003587) Lionel (reporter) - 23-Sep-13 16:58
http://www.dbmail.org/mantis/view.php?id=1020#c3587
----------------------------------------------------------------------
If you want to try other languages like chinese or japanese for example,
translate (Hellow in Chinese) with google translate. This issue did not
exist with dbmail 2.x
----------------------------------------------------------------------
(0003589) paul (administrator) - 23-Sep-13 17:28
http://www.dbmail.org/mantis/view.php?id=1020#c3589
----------------------------------------------------------------------
When I test above subjects with Thunderbird, everything is working just
fine as far as I can tell.
Are you using messages with raw 8bit headers? If so, that would violate
RFC 2822
If you are *not* using 8bit headers please provide one or more test
messages with encoded headers
----------------------------------------------------------------------
(0003591) Lionel (reporter) - 23-Sep-13 17:49
http://www.dbmail.org/mantis/view.php?id=1020#c3591
----------------------------------------------------------------------
Thundrbird is the only email client that we tested that shows the subject
properly in the message itself; not in the list of emails. I also made
tests with Apple Mail and it does not show up properly.
----------------------------------------------------------------------
(0003592) Lionel (reporter) - 23-Sep-13 17:50
http://www.dbmail.org/mantis/view.php?id=1020#c3592
----------------------------------------------------------------------
By the way; on my side, even when I send the emails with thunderbird the
subject is unreadable in the emails list. All settings are the default one.
----------------------------------------------------------------------
(0003593) igorbelykh86 (reporter) - 23-Sep-13 18:21
http://www.dbmail.org/mantis/view.php?id=1020#c3593
----------------------------------------------------------------------
Hi Paul
Let me show you example of headers:
Received: from mail-vb0-f49.google.com (mail-vb0-f49.google.com
[209.85.212.49]) (using TLSv1
with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate
requested)
by mail.example.com (Postfix) with ESMTPS id 1D156781587
for <igor@example.com>; Sun, 22 Sep 2013 12:58:31 -0500 (CDT)
Received: by mail-vb0-f49.google.com with SMTP id w16so1572291vbb.22
for <igor@example.com>; Sun, 22 Sep 2013 10:58:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=example.com;
s=20120113;
h=mime-version:date:message-id:subject:from:to:content-type;
bh=H8rshYhYYDgKX2iy+I9JDjA3nltOr4SWC6SGSMGbS58=;
b=uG7fs/xEvi7mptyqURCvDefwIItiq+cIwgynj/8FHwcIJyThwH1N3z6oGiJ89KDK2n
NDbrII/GkQuQ3radtjRX3R9+8D52f0+JN8kxmaLPlrYYzSz2vu0QE+1QNh5ec/sNuF3y
TULQJpFm6hELvN63yyItAACH1zGSJvfIA4P9bI+SxHZjJMt30Xuh6mgcMNFlfyEdZ5WF
vJbZ/I62+hfzTz5vmO3spsow+lNUclXmalUrMcFWb8GytMunVIVqQ9kuCkRNyQej/en0
7IIjkcHLrXH0wgXEIZrK+qmpvJczYR53TcyPn6QJkn00//FuJ5qor3bWUo+BepKYhef5
TV6Q==
MIME-Version: 1.0
X-Received: by 10.221.51.206 with SMTP id
vj14mr17765026vcb.17.1379872710605; Sun, 22 Sep 2013 10:58:30 -0700
(PDT)
Received: by 10.52.191.40 with HTTP; Sun, 22 Sep 2013 10:58:30 -0700
(PDT)
Date: Mon, 23 Sep 2013 00:58:30 +0700
Message-ID:
<CA+jC7cWEg30O6UP4nKfy9BbbBBzH3_4w4g9=4OQYqfNpKmjniQ@mail.example.com>
Subject:
=?UTF-8?B?0KHQsNC80L7QtSDQvdC+0LLQvtC1INGB0L7QvtCx0YnQtdC90LjQtSDQvtGCINCY0LPQvg==?=
=?UTF-8?B?0YDRjw==?=
From: Igor Belykh <example@example.com>
To: Igor <igor@example.com>
Return-Path: example@example.com
Content-Type: multipart/alternative;
boundary=001a11332252c042b904e6fca6f2
all fields is readable in database except sortfield
----------------------------------------------------------------------
(0003594) Lionel (reporter) - 23-Sep-13 18:33
http://www.dbmail.org/mantis/view.php?id=1020#c3594
----------------------------------------------------------------------
Paul, both Igor and myself are running the dbmail on the same machine. We
both of us checked all the mysql encodings seperately and everything is
fine; it should be running.
We used to run dbmail 2.x and did not have any issue related to encoding.
The only major problem was the memory leaks killing the process that
appears to be fixed in dbmail 3.x. This is the reason why we found it a
great idea to update to dbmail 3.x.
Yesterday, we decided to install a fresh dbmail server, rather than an
upgrade; thinking something might have messed up during the update. We
reinstalled the mysql database and everything from scratch but the issue is
still there. As said before, only in Thunderbird when you open the message
you can see the subject correctly. All the other mail clients replace the
letter by question marks (???)
----------------------------------------------------------------------
(0003595) Lionel (reporter) - 23-Sep-13 19:04
http://www.dbmail.org/mantis/view.php?id=1020#c3595
----------------------------------------------------------------------
If you have Skype, or other, we can maybe arrange a teamviewer session so
that you can see by yourself. I think that would be the easiest thing to
figure out the issue. Let me know. Thanks for your help.
----------------------------------------------------------------------
(0003603) Oleg Lapshin (reporter) - 30-Oct-13 18:37
http://www.dbmail.org/mantis/view.php?id=1020#c3603
----------------------------------------------------------------------
Hello
I have the same issue in new installation.
After some debugging, I saw, that this "questions" are from database.
So, I put
db_query(c, "set names utf8");
in dm_db.c after testing tables and this solved the problem: I see real
subjects.
Paul, can you put this query in wright place, so that connect collation is
set according to real settings (may be with using db_charset)
Thanks.
----------------------------------------------------------------------
(0003677) santa (reporter) - 12-Sep-14 09:39
http://www.dbmail.org/mantis/view.php?id=1020#c3677
----------------------------------------------------------------------
Hello Paul. I have almost the same issue with 3.1.7. Almost because I do
not observe ??? chars in UID FETCH result but rather raw 8bit values.
Steps to reproduce:
1. take a message with non-ascii chars in any of the fields (encoded, RFC
2822 compliant). you can find an example (message.tar.xz attached) with
polish subject.
2. send this message with dbmail-deliver as:
cat message.eml | dbmail-deliver -u 123001@porta-um.intra -m 'INBOX'
-v
dbmail-deliver pushes raw message parts into
dbmail_mimeparts/dbmail_partlists and decoded utf8 headers into
dbmail_headername, dbmail_headervalue
dbmail-imapd returns raw message for UID fetch 213 (UID RFC822.SIZE
BODY.PEEK[]) command as:
From: Pavlo Lavrenenko <Pavlo.Lavrenenko@portaone.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101
Thunderbird/24.6.0
MIME-Version: 1.0
To: santa@portaone.com
Subject: Faks nie =?ISO-8859-2?Q?zosta=B3_wys=B3any?=
Content-Type: text/plain; charset=ISO-8859-2; format=flowed
dbmail-imapd returns decoded utf8 headers for UID fetch 213 (UID
RFC822.SIZE FLAGS BODY.PEEK[.HEADER.FIELDS (From To Cc Bcc Subject Date
Message-ID Priority X-Priority References Newsgroups In-Reply-To
Content-Type Reply-To)]) command as:
* 1 FETCH (RFC822.SIZE 2277 FLAGS (\Seen NonJunk) UID 213
BODY[.HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)] {200}
From: Pavlo Lavrenenko <Pavlo.Lavrenenko@portaone.com>
To: santa@portaone.com
Subject: Faks nie zosta? wys?any
Date: Thu, 11 Sep 2014 14:58:50 +0300
Message-id: <54118E7A.8070007@portaone.com>
)
001 OK UID fetch completed
I expected returned headers to be RFC 2822 encoded.
Issue History
Date Modified Username Field Change
======================================================================
17-Aug-13 09:37 igorbelykh86 New Issue
17-Aug-13 10:33 paul Note Added: 0003559
17-Aug-13 13:15 igorbelykh86 Note Added: 0003560
19-Aug-13 12:02 igorbelykh86 Note Added: 0003561
19-Aug-13 12:04 jasb Note Added: 0003562
19-Aug-13 12:24 igorbelykh86 Note Added: 0003563
19-Aug-13 12:29 igorbelykh86 Note Added: 0003564
19-Aug-13 12:54 paul Note Added: 0003565
19-Aug-13 13:07 igorbelykh86 Note Added: 0003566
21-Sep-13 19:44 igorbelykh86 Note Added: 0003580
22-Sep-13 18:07 Lionel Note Added: 0003581
23-Sep-13 16:44 paul Note Added: 0003582
23-Sep-13 16:48 paul Note Added: 0003583
23-Sep-13 16:49 Lionel Note Added: 0003584
23-Sep-13 16:52 Lionel Note Added: 0003585
23-Sep-13 16:57 paul Note Added: 0003586
23-Sep-13 16:58 Lionel Note Added: 0003587
23-Sep-13 17:28 paul Note Added: 0003589
23-Sep-13 17:49 Lionel Note Added: 0003591
23-Sep-13 17:50 Lionel Note Added: 0003592
23-Sep-13 18:12 igorbelykh86 Note Added: 0003593
23-Sep-13 18:21 igorbelykh86 Note Edited: 0003593
23-Sep-13 18:24 igorbelykh86 Note Edited: 0003563
23-Sep-13 18:33 Lionel Note Added: 0003594
23-Sep-13 19:04 Lionel Note Added: 0003595
30-Oct-13 18:37 Oleg Lapshin Note Added: 0003603
12-Sep-14 09:20 santa File Added: message.tar.xz
12-Sep-14 09:39 santa Note Added: 0003677
======================================================================
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev
======================================================================
http://www.dbmail.org/mantis/view.php?id=1020
======================================================================
Reported By: igorbelykh86
Assigned To:
======================================================================
Project: DBMail
Issue ID: 1020
Category: IMAP daemon
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 17-Aug-13 09:37 CEST
Last Modified: 12-Sep-14 09:39 CEST
======================================================================
Summary: dbmail 3.1.2. Imap server returns wrong decoded
header fields.
Description:
Hi
After upgrading the dbmail to 3.1.2 the IMAP server has began to return
wrong decoded email headers. I've thought that it does it only for old
emails which were been received before upgrading, but no. New mails have
the problem too. This problem appears only when application try to get a
header field. Example:
CLIENT:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {98}
From: Igor Belykh <myemail@address.com>
Subject: ????? ??????
To: Igor <anotheremail@address.com>
)
A003 OK UID FETCH completed
The subject is in russian. It is encoded correctly. If get full header of
the message the server returns:
....
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
....
what I can do to always get encoded header fields?
Thanks.
======================================================================
----------------------------------------------------------------------
(0003559) paul (administrator) - 17-Aug-13 10:33
http://www.dbmail.org/mantis/view.php?id=1020#c3559
----------------------------------------------------------------------
Question is: what is in the database? Was it during insertion or during
retrieval that something went wrong.
The message_idnr is 1017 in your case. Try following:
select m.message_idnr,n.headername,v.headervalue from dbmail_header h join
dbmail_headername n on h.headername_id=n.id join dbmail_headervalue v on
h.headervalue_id=v.id join dbmail_physmessage p on p.id=h.physmessage_id
join dbmail_messages m on m.physmessage_id=p.id where m.message_idnr=1017;
This should give you a dump of the cached headers. If your database is
UTF-8 encoded, and your terminal is as well, it should give you nicely
readable values.
Of course, the order of the headers returned is also wrong, but that was
fixed already and will be ok in 3.1.3.
----------------------------------------------------------------------
(0003560) igorbelykh86 (reporter) - 17-Aug-13 13:15
http://www.dbmail.org/mantis/view.php?id=1020#c3560
----------------------------------------------------------------------
Hi Paul,
All data is readable in the database. All tables is utf-8 encoded. And
system (and terminal too) default encoding is en_US.UTF-8.
Any idea?
----------------------------------------------------------------------
(0003561) igorbelykh86 (reporter) - 19-Aug-13 12:02
http://www.dbmail.org/mantis/view.php?id=1020#c3561
----------------------------------------------------------------------
Hi Paul,
I think that the problem is in receiving data from database? I've checked
the mysql server and mysql client settings. Server and client default
character set is utf8 too:
[client]
......
default-character-set=utf8
......
[mysql]
......
default-character-set=utf8
......
[mysqld]
......
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
......
What do you think what's the problem? Maybe some thing in dbmail settings
or mysql or something else?
----------------------------------------------------------------------
(0003562) jasb (reporter) - 19-Aug-13 12:04
http://www.dbmail.org/mantis/view.php?id=1020#c3562
----------------------------------------------------------------------
The [client] section of my.ini/cnf is for the mysql console client.. not
3rd party software.
----------------------------------------------------------------------
(0003563) igorbelykh86 (reporter) - 23-Sep-13 18:24
http://www.dbmail.org/mantis/view.php?id=1020#c3563
----------------------------------------------------------------------
just tried to comment line "encoding = utf8". Now it works fine.
CLIENT REQUEST:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER RESPONSE:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {119}
From: Igor Belykh <example@example.com>
Subject: =?iso-8859-5?b?vd7S3tUg39jh7Nze?=
To: Igor <igor@example.com>
)
A003 OK UID FETCH completed
How I understand these feader fields are received from source code of the
message.
----------------------------------------------------------------------
(0003564) igorbelykh86 (reporter) - 19-Aug-13 12:29
http://www.dbmail.org/mantis/view.php?id=1020#c3564
----------------------------------------------------------------------
no, in the source code the subject header field is different:
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
So the imap server works fine if the line "encoding = utf8" is commented.
----------------------------------------------------------------------
(0003565) paul (administrator) - 19-Aug-13 12:54
http://www.dbmail.org/mantis/view.php?id=1020#c3565
----------------------------------------------------------------------
the value of the 'encoding' parameter must match the collation of the
database. Apparently your database is not in UTF8.
try:
mysql dbmail
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | utf8_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
if the collation of the database and connection are *not* the same, you
need to set the encoding of the connection to match the collation of the
database:
mysql> SET NAMES koi8r;
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | koi8r_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
this means in this example the 'encoding' parameter should be 'koi8r';
Without the encoding paramater, it falls back to your system's default
locale, I syspect.
----------------------------------------------------------------------
(0003566) igorbelykh86 (reporter) - 19-Aug-13 13:07
http://www.dbmail.org/mantis/view.php?id=1020#c3566
----------------------------------------------------------------------
This is result of the query:
mysql> show variables like 'collation_%';
+----------------------+-----------------+
| Variable_name | Value |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
+----------------------+-----------------+
3 rows in set (0.00 sec)
Thus the encoding parameter should be 'utf8'. Right?
----------------------------------------------------------------------
(0003580) igorbelykh86 (reporter) - 21-Sep-13 19:44
http://www.dbmail.org/mantis/view.php?id=1020#c3580
----------------------------------------------------------------------
Hi Paul,
I've found one issue with encoding in data base. There is `sortfield` cell
in the `dbmail_headervalue` table. This cell has utf8_general_ci collation
but in rows I see following:
ßрøòõт ø÷ à þÑÂÑÂøø (Hello from
Russia)
instead
Привет из РоÑÑии (Hello from Russia)
But the `headervalue` cell contains the right value in right encoding
Regards,
Igor
----------------------------------------------------------------------
(0003581) Lionel (reporter) - 22-Sep-13 18:07
http://www.dbmail.org/mantis/view.php?id=1020#c3581
----------------------------------------------------------------------
Hi Paul,
The problem is not only with russian characters but with any language that
needs UTF-8 support.
I've done tests with russian, hebrew, arabic and greek characters, they
all have the same issue.
I suspect the problem to be in the place where the sortfield is being
created/processed when there is a new email because the sortfield is not
inserted in the database correctly. The headervalue is inserted correctly,
but not the sortfield. The way the sortfield appears in the database looks
like a typical utf-8 meta tag missing in html. However, I do not know where
in the src files this is handled.
Your input on this in order to fix it is welcomed as I think this is a
major issue for all dbmail users trying to run dbmail 3.x as with this
issue dbmail is totally useless for anyone receiving or sending emails with
characters other than english/dutch.
Thanks.
----------------------------------------------------------------------
(0003582) paul (administrator) - 23-Sep-13 16:44
http://www.dbmail.org/mantis/view.php?id=1020#c3582
----------------------------------------------------------------------
I wonder how does this affect the issue mentioned here, since the sortfield
only affects 'sorting'?
----------------------------------------------------------------------
(0003583) paul (administrator) - 23-Sep-13 16:48
http://www.dbmail.org/mantis/view.php?id=1020#c3583
----------------------------------------------------------------------
can you please provide a simple test message to verify?
----------------------------------------------------------------------
(0003584) Lionel (reporter) - 23-Sep-13 16:49
http://www.dbmail.org/mantis/view.php?id=1020#c3584
----------------------------------------------------------------------
yes you are right. We (me & Igor) figured this out yesterday while looking
at the source files.
However, we did not find a fix for the issue regarding the headers and the
subject that is totally unreadable (???? ???? ??????). Any idea what might
be the issue? or where we should be looking at to try to figure it out?
Thanks.
----------------------------------------------------------------------
(0003585) Lionel (reporter) - 23-Sep-13 16:52
http://www.dbmail.org/mantis/view.php?id=1020#c3585
----------------------------------------------------------------------
just send an email and put in the subject the following: Привет из
РоÑÑии (Hello from Russia)
Same problem as well with hebrew: ×©×œ×•× ×‘×¢×‘×¨×™×ª (Hello in Hebrew)
In arabic: مرØبا باللغة العربية (Hellow in Arabic)
Thanks.
----------------------------------------------------------------------
(0003586) paul (administrator) - 23-Sep-13 16:57
http://www.dbmail.org/mantis/view.php?id=1020#c3586
----------------------------------------------------------------------
ignore that last message
----------------------------------------------------------------------
(0003587) Lionel (reporter) - 23-Sep-13 16:58
http://www.dbmail.org/mantis/view.php?id=1020#c3587
----------------------------------------------------------------------
If you want to try other languages like chinese or japanese for example,
translate (Hellow in Chinese) with google translate. This issue did not
exist with dbmail 2.x
----------------------------------------------------------------------
(0003589) paul (administrator) - 23-Sep-13 17:28
http://www.dbmail.org/mantis/view.php?id=1020#c3589
----------------------------------------------------------------------
When I test above subjects with Thunderbird, everything is working just
fine as far as I can tell.
Are you using messages with raw 8bit headers? If so, that would violate
RFC 2822
If you are *not* using 8bit headers please provide one or more test
messages with encoded headers
----------------------------------------------------------------------
(0003591) Lionel (reporter) - 23-Sep-13 17:49
http://www.dbmail.org/mantis/view.php?id=1020#c3591
----------------------------------------------------------------------
Thundrbird is the only email client that we tested that shows the subject
properly in the message itself; not in the list of emails. I also made
tests with Apple Mail and it does not show up properly.
----------------------------------------------------------------------
(0003592) Lionel (reporter) - 23-Sep-13 17:50
http://www.dbmail.org/mantis/view.php?id=1020#c3592
----------------------------------------------------------------------
By the way; on my side, even when I send the emails with thunderbird the
subject is unreadable in the emails list. All settings are the default one.
----------------------------------------------------------------------
(0003593) igorbelykh86 (reporter) - 23-Sep-13 18:21
http://www.dbmail.org/mantis/view.php?id=1020#c3593
----------------------------------------------------------------------
Hi Paul
Let me show you example of headers:
Received: from mail-vb0-f49.google.com (mail-vb0-f49.google.com
[209.85.212.49]) (using TLSv1
with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate
requested)
by mail.example.com (Postfix) with ESMTPS id 1D156781587
for <igor@example.com>; Sun, 22 Sep 2013 12:58:31 -0500 (CDT)
Received: by mail-vb0-f49.google.com with SMTP id w16so1572291vbb.22
for <igor@example.com>; Sun, 22 Sep 2013 10:58:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=example.com;
s=20120113;
h=mime-version:date:message-id:subject:from:to:content-type;
bh=H8rshYhYYDgKX2iy+I9JDjA3nltOr4SWC6SGSMGbS58=;
b=uG7fs/xEvi7mptyqURCvDefwIItiq+cIwgynj/8FHwcIJyThwH1N3z6oGiJ89KDK2n
NDbrII/GkQuQ3radtjRX3R9+8D52f0+JN8kxmaLPlrYYzSz2vu0QE+1QNh5ec/sNuF3y
TULQJpFm6hELvN63yyItAACH1zGSJvfIA4P9bI+SxHZjJMt30Xuh6mgcMNFlfyEdZ5WF
vJbZ/I62+hfzTz5vmO3spsow+lNUclXmalUrMcFWb8GytMunVIVqQ9kuCkRNyQej/en0
7IIjkcHLrXH0wgXEIZrK+qmpvJczYR53TcyPn6QJkn00//FuJ5qor3bWUo+BepKYhef5
TV6Q==
MIME-Version: 1.0
X-Received: by 10.221.51.206 with SMTP id
vj14mr17765026vcb.17.1379872710605; Sun, 22 Sep 2013 10:58:30 -0700
(PDT)
Received: by 10.52.191.40 with HTTP; Sun, 22 Sep 2013 10:58:30 -0700
(PDT)
Date: Mon, 23 Sep 2013 00:58:30 +0700
Message-ID:
<CA+jC7cWEg30O6UP4nKfy9BbbBBzH3_4w4g9=4OQYqfNpKmjniQ@mail.example.com>
Subject:
=?UTF-8?B?0KHQsNC80L7QtSDQvdC+0LLQvtC1INGB0L7QvtCx0YnQtdC90LjQtSDQvtGCINCY0LPQvg==?=
=?UTF-8?B?0YDRjw==?=
From: Igor Belykh <example@example.com>
To: Igor <igor@example.com>
Return-Path: example@example.com
Content-Type: multipart/alternative;
boundary=001a11332252c042b904e6fca6f2
all fields is readable in database except sortfield
----------------------------------------------------------------------
(0003594) Lionel (reporter) - 23-Sep-13 18:33
http://www.dbmail.org/mantis/view.php?id=1020#c3594
----------------------------------------------------------------------
Paul, both Igor and myself are running the dbmail on the same machine. We
both of us checked all the mysql encodings seperately and everything is
fine; it should be running.
We used to run dbmail 2.x and did not have any issue related to encoding.
The only major problem was the memory leaks killing the process that
appears to be fixed in dbmail 3.x. This is the reason why we found it a
great idea to update to dbmail 3.x.
Yesterday, we decided to install a fresh dbmail server, rather than an
upgrade; thinking something might have messed up during the update. We
reinstalled the mysql database and everything from scratch but the issue is
still there. As said before, only in Thunderbird when you open the message
you can see the subject correctly. All the other mail clients replace the
letter by question marks (???)
----------------------------------------------------------------------
(0003595) Lionel (reporter) - 23-Sep-13 19:04
http://www.dbmail.org/mantis/view.php?id=1020#c3595
----------------------------------------------------------------------
If you have Skype, or other, we can maybe arrange a teamviewer session so
that you can see by yourself. I think that would be the easiest thing to
figure out the issue. Let me know. Thanks for your help.
----------------------------------------------------------------------
(0003603) Oleg Lapshin (reporter) - 30-Oct-13 18:37
http://www.dbmail.org/mantis/view.php?id=1020#c3603
----------------------------------------------------------------------
Hello
I have the same issue in new installation.
After some debugging, I saw, that this "questions" are from database.
So, I put
db_query(c, "set names utf8");
in dm_db.c after testing tables and this solved the problem: I see real
subjects.
Paul, can you put this query in wright place, so that connect collation is
set according to real settings (may be with using db_charset)
Thanks.
----------------------------------------------------------------------
(0003677) santa (reporter) - 12-Sep-14 09:39
http://www.dbmail.org/mantis/view.php?id=1020#c3677
----------------------------------------------------------------------
Hello Paul. I have almost the same issue with 3.1.7. Almost because I do
not observe ??? chars in UID FETCH result but rather raw 8bit values.
Steps to reproduce:
1. take a message with non-ascii chars in any of the fields (encoded, RFC
2822 compliant). you can find an example (message.tar.xz attached) with
polish subject.
2. send this message with dbmail-deliver as:
cat message.eml | dbmail-deliver -u 123001@porta-um.intra -m 'INBOX'
-v
dbmail-deliver pushes raw message parts into
dbmail_mimeparts/dbmail_partlists and decoded utf8 headers into
dbmail_headername, dbmail_headervalue
dbmail-imapd returns raw message for UID fetch 213 (UID RFC822.SIZE
BODY.PEEK[]) command as:
From: Pavlo Lavrenenko <Pavlo.Lavrenenko@portaone.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:24.0) Gecko/20100101
Thunderbird/24.6.0
MIME-Version: 1.0
To: santa@portaone.com
Subject: Faks nie =?ISO-8859-2?Q?zosta=B3_wys=B3any?=
Content-Type: text/plain; charset=ISO-8859-2; format=flowed
dbmail-imapd returns decoded utf8 headers for UID fetch 213 (UID
RFC822.SIZE FLAGS BODY.PEEK[.HEADER.FIELDS (From To Cc Bcc Subject Date
Message-ID Priority X-Priority References Newsgroups In-Reply-To
Content-Type Reply-To)]) command as:
* 1 FETCH (RFC822.SIZE 2277 FLAGS (\Seen NonJunk) UID 213
BODY[.HEADER.FIELDS (From To Cc Bcc Subject Date Message-ID Priority
X-Priority References Newsgroups In-Reply-To Content-Type Reply-To)] {200}
From: Pavlo Lavrenenko <Pavlo.Lavrenenko@portaone.com>
To: santa@portaone.com
Subject: Faks nie zosta? wys?any
Date: Thu, 11 Sep 2014 14:58:50 +0300
Message-id: <54118E7A.8070007@portaone.com>
)
001 OK UID fetch completed
I expected returned headers to be RFC 2822 encoded.
Issue History
Date Modified Username Field Change
======================================================================
17-Aug-13 09:37 igorbelykh86 New Issue
17-Aug-13 10:33 paul Note Added: 0003559
17-Aug-13 13:15 igorbelykh86 Note Added: 0003560
19-Aug-13 12:02 igorbelykh86 Note Added: 0003561
19-Aug-13 12:04 jasb Note Added: 0003562
19-Aug-13 12:24 igorbelykh86 Note Added: 0003563
19-Aug-13 12:29 igorbelykh86 Note Added: 0003564
19-Aug-13 12:54 paul Note Added: 0003565
19-Aug-13 13:07 igorbelykh86 Note Added: 0003566
21-Sep-13 19:44 igorbelykh86 Note Added: 0003580
22-Sep-13 18:07 Lionel Note Added: 0003581
23-Sep-13 16:44 paul Note Added: 0003582
23-Sep-13 16:48 paul Note Added: 0003583
23-Sep-13 16:49 Lionel Note Added: 0003584
23-Sep-13 16:52 Lionel Note Added: 0003585
23-Sep-13 16:57 paul Note Added: 0003586
23-Sep-13 16:58 Lionel Note Added: 0003587
23-Sep-13 17:28 paul Note Added: 0003589
23-Sep-13 17:49 Lionel Note Added: 0003591
23-Sep-13 17:50 Lionel Note Added: 0003592
23-Sep-13 18:12 igorbelykh86 Note Added: 0003593
23-Sep-13 18:21 igorbelykh86 Note Edited: 0003593
23-Sep-13 18:24 igorbelykh86 Note Edited: 0003563
23-Sep-13 18:33 Lionel Note Added: 0003594
23-Sep-13 19:04 Lionel Note Added: 0003595
30-Oct-13 18:37 Oleg Lapshin Note Added: 0003603
12-Sep-14 09:20 santa File Added: message.tar.xz
12-Sep-14 09:39 santa Note Added: 0003677
======================================================================
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev