A NOTE has been added to this issue.
======================================================================
http://www.dbmail.org/mantis/view.php?id=1020
======================================================================
Reported By: igorbelykh86
Assigned To:
======================================================================
Project: DBMail
Issue ID: 1020
Category: IMAP daemon
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 17-Aug-13 09:37 CEST
Last Modified: 21-Sep-13 19:44 CEST
======================================================================
Summary: dbmail 3.1.2. Imap server returns wrong decoded
header fields.
Description:
Hi
After upgrading the dbmail to 3.1.2 the IMAP server has began to return
wrong decoded email headers. I've thought that it does it only for old
emails which were been received before upgrading, but no. New mails have
the problem too. This problem appears only when application try to get a
header field. Example:
CLIENT:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {98}
From: Igor Belykh <myemail@address.com>
Subject: ????? ??????
To: Igor <anotheremail@address.com>
)
A003 OK UID FETCH completed
The subject is in russian. It is encoded correctly. If get full header of
the message the server returns:
....
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
....
what I can do to always get encoded header fields?
Thanks.
======================================================================
----------------------------------------------------------------------
(0003559) paul (administrator) - 17-Aug-13 10:33
http://www.dbmail.org/mantis/view.php?id=1020#c3559
----------------------------------------------------------------------
Question is: what is in the database? Was it during insertion or during
retrieval that something went wrong.
The message_idnr is 1017 in your case. Try following:
select m.message_idnr,n.headername,v.headervalue from dbmail_header h join
dbmail_headername n on h.headername_id=n.id join dbmail_headervalue v on
h.headervalue_id=v.id join dbmail_physmessage p on p.id=h.physmessage_id
join dbmail_messages m on m.physmessage_id=p.id where m.message_idnr=1017;
This should give you a dump of the cached headers. If your database is
UTF-8 encoded, and your terminal is as well, it should give you nicely
readable values.
Of course, the order of the headers returned is also wrong, but that was
fixed already and will be ok in 3.1.3.
----------------------------------------------------------------------
(0003560) igorbelykh86 (reporter) - 17-Aug-13 13:15
http://www.dbmail.org/mantis/view.php?id=1020#c3560
----------------------------------------------------------------------
Hi Paul,
All data is readable in the database. All tables is utf-8 encoded. And
system (and terminal too) default encoding is en_US.UTF-8.
Any idea?
----------------------------------------------------------------------
(0003561) igorbelykh86 (reporter) - 19-Aug-13 12:02
http://www.dbmail.org/mantis/view.php?id=1020#c3561
----------------------------------------------------------------------
Hi Paul,
I think that the problem is in receiving data from database? I've checked
the mysql server and mysql client settings. Server and client default
character set is utf8 too:
[client]
......
default-character-set=utf8
......
[mysql]
......
default-character-set=utf8
......
[mysqld]
......
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
......
What do you think what's the problem? Maybe some thing in dbmail settings
or mysql or something else?
----------------------------------------------------------------------
(0003562) jasb (reporter) - 19-Aug-13 12:04
http://www.dbmail.org/mantis/view.php?id=1020#c3562
----------------------------------------------------------------------
The [client] section of my.ini/cnf is for the mysql console client.. not
3rd party software.
----------------------------------------------------------------------
(0003563) igorbelykh86 (reporter) - 19-Aug-13 12:24
http://www.dbmail.org/mantis/view.php?id=1020#c3563
----------------------------------------------------------------------
just tried to comment line "encoding = utf8". Now it works fine.
CLIENT REQUEST:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER RESPONSE:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {119}
From: Igor Belykh <igorbelykh86@gmail.com>
Subject: =?iso-8859-5?b?vd7S3tUg39jh7Nze?=
To: Igor <igor@zubybox.com>
)
A003 OK UID FETCH completed
How I understand these feader fields are received from source code of the
message.
----------------------------------------------------------------------
(0003564) igorbelykh86 (reporter) - 19-Aug-13 12:29
http://www.dbmail.org/mantis/view.php?id=1020#c3564
----------------------------------------------------------------------
no, in the source code the subject header field is different:
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
So the imap server works fine if the line "encoding = utf8" is commented.
----------------------------------------------------------------------
(0003565) paul (administrator) - 19-Aug-13 12:54
http://www.dbmail.org/mantis/view.php?id=1020#c3565
----------------------------------------------------------------------
the value of the 'encoding' parameter must match the collation of the
database. Apparently your database is not in UTF8.
try:
mysql dbmail
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | utf8_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
if the collation of the database and connection are *not* the same, you
need to set the encoding of the connection to match the collation of the
database:
mysql> SET NAMES koi8r;
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | koi8r_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
this means in this example the 'encoding' parameter should be 'koi8r';
Without the encoding paramater, it falls back to your system's default
locale, I syspect.
----------------------------------------------------------------------
(0003566) igorbelykh86 (reporter) - 19-Aug-13 13:07
http://www.dbmail.org/mantis/view.php?id=1020#c3566
----------------------------------------------------------------------
This is result of the query:
mysql> show variables like 'collation_%';
+----------------------+-----------------+
| Variable_name | Value |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
+----------------------+-----------------+
3 rows in set (0.00 sec)
Thus the encoding parameter should be 'utf8'. Right?
----------------------------------------------------------------------
(0003580) igorbelykh86 (reporter) - 21-Sep-13 19:44
http://www.dbmail.org/mantis/view.php?id=1020#c3580
----------------------------------------------------------------------
Hi Paul,
I've found one issue with encoding in data base. There is `sortfield` cell
in the `dbmail_headervalue` table. This cell has utf8_general_ci collation
but in rows I see following:
ßрøòõт ø÷ à þÑÂÑÂøø (Hello from
Russia)
instead
Привет из РоÑÑии (Hello from Russia)
But the `headervalue` cell contains the right value in right encoding
Regards,
Igor
Issue History
Date Modified Username Field Change
======================================================================
17-Aug-13 09:37 igorbelykh86 New Issue
17-Aug-13 10:33 paul Note Added: 0003559
17-Aug-13 13:15 igorbelykh86 Note Added: 0003560
19-Aug-13 12:02 igorbelykh86 Note Added: 0003561
19-Aug-13 12:04 jasb Note Added: 0003562
19-Aug-13 12:24 igorbelykh86 Note Added: 0003563
19-Aug-13 12:29 igorbelykh86 Note Added: 0003564
19-Aug-13 12:54 paul Note Added: 0003565
19-Aug-13 13:07 igorbelykh86 Note Added: 0003566
21-Sep-13 19:44 igorbelykh86 Note Added: 0003580
======================================================================
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev
======================================================================
http://www.dbmail.org/mantis/view.php?id=1020
======================================================================
Reported By: igorbelykh86
Assigned To:
======================================================================
Project: DBMail
Issue ID: 1020
Category: IMAP daemon
Reproducibility: always
Severity: minor
Priority: normal
Status: new
target:
======================================================================
Date Submitted: 17-Aug-13 09:37 CEST
Last Modified: 21-Sep-13 19:44 CEST
======================================================================
Summary: dbmail 3.1.2. Imap server returns wrong decoded
header fields.
Description:
Hi
After upgrading the dbmail to 3.1.2 the IMAP server has began to return
wrong decoded email headers. I've thought that it does it only for old
emails which were been received before upgrading, but no. New mails have
the problem too. This problem appears only when application try to get a
header field. Example:
CLIENT:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {98}
From: Igor Belykh <myemail@address.com>
Subject: ????? ??????
To: Igor <anotheremail@address.com>
)
A003 OK UID FETCH completed
The subject is in russian. It is encoded correctly. If get full header of
the message the server returns:
....
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
....
what I can do to always get encoded header fields?
Thanks.
======================================================================
----------------------------------------------------------------------
(0003559) paul (administrator) - 17-Aug-13 10:33
http://www.dbmail.org/mantis/view.php?id=1020#c3559
----------------------------------------------------------------------
Question is: what is in the database? Was it during insertion or during
retrieval that something went wrong.
The message_idnr is 1017 in your case. Try following:
select m.message_idnr,n.headername,v.headervalue from dbmail_header h join
dbmail_headername n on h.headername_id=n.id join dbmail_headervalue v on
h.headervalue_id=v.id join dbmail_physmessage p on p.id=h.physmessage_id
join dbmail_messages m on m.physmessage_id=p.id where m.message_idnr=1017;
This should give you a dump of the cached headers. If your database is
UTF-8 encoded, and your terminal is as well, it should give you nicely
readable values.
Of course, the order of the headers returned is also wrong, but that was
fixed already and will be ok in 3.1.3.
----------------------------------------------------------------------
(0003560) igorbelykh86 (reporter) - 17-Aug-13 13:15
http://www.dbmail.org/mantis/view.php?id=1020#c3560
----------------------------------------------------------------------
Hi Paul,
All data is readable in the database. All tables is utf-8 encoded. And
system (and terminal too) default encoding is en_US.UTF-8.
Any idea?
----------------------------------------------------------------------
(0003561) igorbelykh86 (reporter) - 19-Aug-13 12:02
http://www.dbmail.org/mantis/view.php?id=1020#c3561
----------------------------------------------------------------------
Hi Paul,
I think that the problem is in receiving data from database? I've checked
the mysql server and mysql client settings. Server and client default
character set is utf8 too:
[client]
......
default-character-set=utf8
......
[mysql]
......
default-character-set=utf8
......
[mysqld]
......
collation-server = utf8_unicode_ci
init-connect='SET NAMES utf8'
character-set-server = utf8
......
What do you think what's the problem? Maybe some thing in dbmail settings
or mysql or something else?
----------------------------------------------------------------------
(0003562) jasb (reporter) - 19-Aug-13 12:04
http://www.dbmail.org/mantis/view.php?id=1020#c3562
----------------------------------------------------------------------
The [client] section of my.ini/cnf is for the mysql console client.. not
3rd party software.
----------------------------------------------------------------------
(0003563) igorbelykh86 (reporter) - 19-Aug-13 12:24
http://www.dbmail.org/mantis/view.php?id=1020#c3563
----------------------------------------------------------------------
just tried to comment line "encoding = utf8". Now it works fine.
CLIENT REQUEST:
UID FETCH 1017 (BODY.PEEK[HEADER.FIELDS (SUBJECT FROM CC TO)] INTERNALDATE
BODY FLAGS)
SERVER RESPONSE:
* 127 FETCH (INTERNALDATE "16-Aug-2013 21:27:40 +0000" FLAGS (\Seen) UID
1017 BODY (("text" "plain" ("charset" "UTF-8") NIL NIL "base64" 60
1)("text" "html" ("charset" "UTF-8") NIL NIL "base64" 90 2) "alternative")
BODY[HEADER.FIELDS (SUBJECT FROM CC TO)] {119}
From: Igor Belykh <igorbelykh86@gmail.com>
Subject: =?iso-8859-5?b?vd7S3tUg39jh7Nze?=
To: Igor <igor@zubybox.com>
)
A003 OK UID FETCH completed
How I understand these feader fields are received from source code of the
message.
----------------------------------------------------------------------
(0003564) igorbelykh86 (reporter) - 19-Aug-13 12:29
http://www.dbmail.org/mantis/view.php?id=1020#c3564
----------------------------------------------------------------------
no, in the source code the subject header field is different:
Subject: =?UTF-8?B?0J3QvtCy0L7QtSDQv9C40YHRjNC80L4=?=
So the imap server works fine if the line "encoding = utf8" is commented.
----------------------------------------------------------------------
(0003565) paul (administrator) - 19-Aug-13 12:54
http://www.dbmail.org/mantis/view.php?id=1020#c3565
----------------------------------------------------------------------
the value of the 'encoding' parameter must match the collation of the
database. Apparently your database is not in UTF8.
try:
mysql dbmail
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | utf8_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
if the collation of the database and connection are *not* the same, you
need to set the encoding of the connection to match the collation of the
database:
mysql> SET NAMES koi8r;
mysql> show variables like 'collation_%';
+----------------------+-------------------+
| Variable_name | Value |
+----------------------+-------------------+
| collation_connection | koi8r_general_ci |
| collation_database | koi8r_general_ci |
| collation_server | koi8r_general_ci |
+----------------------+-------------------+
3 rows in set (0.00 sec)
this means in this example the 'encoding' parameter should be 'koi8r';
Without the encoding paramater, it falls back to your system's default
locale, I syspect.
----------------------------------------------------------------------
(0003566) igorbelykh86 (reporter) - 19-Aug-13 13:07
http://www.dbmail.org/mantis/view.php?id=1020#c3566
----------------------------------------------------------------------
This is result of the query:
mysql> show variables like 'collation_%';
+----------------------+-----------------+
| Variable_name | Value |
+----------------------+-----------------+
| collation_connection | utf8_general_ci |
| collation_database | utf8_general_ci |
| collation_server | utf8_general_ci |
+----------------------+-----------------+
3 rows in set (0.00 sec)
Thus the encoding parameter should be 'utf8'. Right?
----------------------------------------------------------------------
(0003580) igorbelykh86 (reporter) - 21-Sep-13 19:44
http://www.dbmail.org/mantis/view.php?id=1020#c3580
----------------------------------------------------------------------
Hi Paul,
I've found one issue with encoding in data base. There is `sortfield` cell
in the `dbmail_headervalue` table. This cell has utf8_general_ci collation
but in rows I see following:
ßрøòõт ø÷ à þÑÂÑÂøø (Hello from
Russia)
instead
Привет из РоÑÑии (Hello from Russia)
But the `headervalue` cell contains the right value in right encoding
Regards,
Igor
Issue History
Date Modified Username Field Change
======================================================================
17-Aug-13 09:37 igorbelykh86 New Issue
17-Aug-13 10:33 paul Note Added: 0003559
17-Aug-13 13:15 igorbelykh86 Note Added: 0003560
19-Aug-13 12:02 igorbelykh86 Note Added: 0003561
19-Aug-13 12:04 jasb Note Added: 0003562
19-Aug-13 12:24 igorbelykh86 Note Added: 0003563
19-Aug-13 12:29 igorbelykh86 Note Added: 0003564
19-Aug-13 12:54 paul Note Added: 0003565
19-Aug-13 13:07 igorbelykh86 Note Added: 0003566
21-Sep-13 19:44 igorbelykh86 Note Added: 0003580
======================================================================
_______________________________________________
Dbmail-dev mailing list
Dbmail-dev@dbmail.org
http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev