Mailing List Archive

Encoding - UTF-8 on LATIN1-server
Dear all

on my servers, I need to have a LATIN1 encoding for various reasons
which I can influence only partially.

When I want to select some data from davical on the "command-line", I
run usually into an encoding problem, as shown next:

postgres=# \c davical You are now connected to database "davical" as
user "postgres". davical=# \d caldav_data; davical=# show
client_encoding; ?client_encoding ----------------- ?LATIN1 (1 row)
davical=# show server_encoding; ?server_encoding ----------------- ?UTF8
(1 row) davical=# select user_no,dav_name,caldav_data from caldav_data;
ERROR:? character 0xe28093 of encoding "UTF8" has no equivalent in "LATIN1"

Is there any chance to circumvent this problem?? I'd appreciate any hint
how I can achieve that best.

Thanks in advance.

Best regards

Lukas
Re: Encoding - UTF-8 on LATIN1-server [ In reply to ]
Hi Lukas,

> on my servers, I need to have a LATIN1 encoding for various reasons
> which I can influence only partially.
>
> When I want to select some data from davical on the "command-line", I
> run usually into an encoding problem, as shown next:
>
> postgres=# \c davical You are now connected to database "davical" as
> user "postgres". davical=# \d caldav_data; davical=# show
> client_encoding; ?client_encoding ----------------- ?LATIN1 (1 row)
> davical=# show server_encoding; ?server_encoding ----------------- ?UTF8
> (1 row) davical=# select user_no,dav_name,caldav_data from caldav_data;
> ERROR:? character 0xe28093 of encoding "UTF8" has no equivalent in "LATIN1"
>
> Is there any chance to circumvent this problem?? I'd appreciate any hint
> how I can achieve that best.

calendar and addressbook data is UTF-8 encoded by default, so clients
can send and store data on your davical server which is impossible to
represent in latin1 (think a personal name in a non-latin script or
typographic quotes or ...). So it is not possible to prevent your
database to contain such data, which is why the davical installation
mandates utf-8 as the database encoding for postgres.

How you deal with this depends on what you want to do. For database
dumps, you want files encoded in UTF-8, to faithfully preserve user
data. When exploring the database interactively in search of some other
problem, you may not care about seeing funny characters where umlauts
are supposed to be. In that case, change the client encoding to UTF-8,
for example by setting LC_CTYPE=de_CH.UTF-8 or PGCLIENTENCODING when
calling psql, and thus shift the burden of converting from UTF-8 to
latin1 to the terminal, which will usually try to carry on and might
even do the right thing if you also adjust your terminal's encoding...

In general, to work with UTF-8 characters doesn't force you to change
the default encoding on a machine, however it often helps when
applications are aware of the correct encoding of the data they have to
deal with, and you're careful not to introduce unintended character
conversions when writing or displaying UTF-8 data.

Florian



_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Encoding - UTF-8 on LATIN1-server [ In reply to ]
Dear Florian


On 05.12.2018 17:33, Florian Schlichting wrote:

>> on my servers, I need to have a LATIN1 encoding for various reasons
>> which I can influence only partially.
>>
>> When I want to select some data from davical on the "command-line", I
>> run usually into an encoding problem, as shown next:
>>
>> postgres=# \c davical You are now connected to database "davical" as
>> user "postgres". davical=# \d caldav_data; davical=# show
>> client_encoding; ?client_encoding ----------------- ?LATIN1 (1 row)
>> davical=# show server_encoding; ?server_encoding ----------------- ?UTF8
>> (1 row) davical=# select user_no,dav_name,caldav_data from caldav_data;
>> ERROR:? character 0xe28093 of encoding "UTF8" has no equivalent in "LATIN1"
>>
>> Is there any chance to circumvent this problem?? I'd appreciate any hint
>> how I can achieve that best.
> calendar and addressbook data is UTF-8 encoded by default, so clients
> can send and store data on your davical server which is impossible to
> represent in latin1 (think a personal name in a non-latin script or
> typographic quotes or ...). So it is not possible to prevent your
> database to contain such data, which is why the davical installation
> mandates utf-8 as the database encoding for postgres.
>
> How you deal with this depends on what you want to do. For database
> dumps, you want files encoded in UTF-8, to faithfully preserve user
> data. When exploring the database interactively in search of some other
> problem, you may not care about seeing funny characters where umlauts
> are supposed to be. In that case, change the client encoding to UTF-8,
> for example by setting LC_CTYPE=de_CH.UTF-8 or PGCLIENTENCODING when
> calling psql, and thus shift the burden of converting from UTF-8 to
> latin1 to the terminal, which will usually try to carry on and might
> even do the right thing if you also adjust your terminal's encoding...
>
> In general, to work with UTF-8 characters doesn't force you to change
> the default encoding on a machine, however it often helps when
> applications are aware of the correct encoding of the data they have to
> deal with, and you're careful not to introduce unintended character
> conversions when writing or displaying UTF-8 data.

Thanks for your hints.

I followed your advise with the following results:

Supported locales updated.
Updated view: dav_principal.sql applied.
CalDAV functions updated.
RRULE functions updated.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_acl" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 61.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_acl" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 61.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_bind" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 62.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_bind" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 62.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_delete" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 63.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_delete" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 63.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_delticket" does
not exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 64.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_delticket" does
not exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 64.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_get" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 65.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_get" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 65.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_head" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 66.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_head" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 66.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_lock" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 67.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_lock" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 67.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_mkcalendar" does
not exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 68.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_mkcalendar" does
not exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 68.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_mkcol" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 69.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_mkcol" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 69.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_mkticket" does
not exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 70.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_mkticket" does
not exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 70.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_move" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 71.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_move" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 71.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_options" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 72.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_options" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 72.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_post" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 73.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_post" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 73.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_propfind" does
not exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 74.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_propfind" does
not exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 74.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_proppatch" does
not exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 75.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_proppatch" does
not exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 75.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_put" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 76.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_put" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 76.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_report" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 77.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_report" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 77.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_unknown" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 78.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_unknown" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 78.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_unlock" does not
exist at /usr/share/davical/dba/update-davical-database line 400,
<PERMS> line 79.
DBD::Pg::db do failed: ERROR:? relation "metrics_count_unlock" does not
exist at /usr/share/davical/dba/update-davical-database line 410,
<PERMS> line 79.
Database permissions updated.


Do you? or anybody else know how to proceed from here?

Thanks in advance

Lukas
Re: Encoding - UTF-8 on LATIN1-server [ In reply to ]
Hi Lukas,

> I followed your advise with the following results:
>
> Supported locales updated.
> Updated view: dav_principal.sql applied.
> CalDAV functions updated.
> RRULE functions updated.
> DBD::Pg::db do failed: ERROR:? relation "metrics_count_acl" does not
> exist at /usr/share/davical/dba/update-davical-database line 400,
> <PERMS> line 61.

metrics_count_acl etc. were added in c5c0421ca, released in davical
1.1.5; they are added to the database with the version 1.3.2 patch.

The funny thing is that dba/update-davical-database should first apply
all the patches to the database (including dba/patches/1.3.2.sql) before
updating locales/views/.../PERMS. And it should print a line about that,
like "Successfully applied 3 patches." or "No patches were applied.",
unless you called it with --nopatch??

Florian


_______________________________________________
Davical-general mailing list
Davical-general@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/davical-general
Re: Encoding - UTF-8 on LATIN1-server [ In reply to ]
Salut Florian

Thanks for your reply.

On 29.12.2018 00:33, Florian Schlichting wrote:
>> Supported locales updated.
>> Updated view: dav_principal.sql applied.
>> CalDAV functions updated.
>> RRULE functions updated.
>> DBD::Pg::db do failed: ERROR:? relation "metrics_count_acl" does not
>> exist at /usr/share/davical/dba/update-davical-database line 400,
>> <PERMS> line 61.
> metrics_count_acl etc. were added in c5c0421ca, released in davical
> 1.1.5; they are added to the database with the version 1.3.2 patch.
>
> The funny thing is that dba/update-davical-database should first apply
> all the patches to the database (including dba/patches/1.3.2.sql) before
> updating locales/views/.../PERMS. And it should print a line about that,
> like "Successfully applied 3 patches." or "No patches were applied.",
> unless you called it with --nopatch??

Thanks for pointing this out.

When debugging my previous problem, I found, somewhere on the web, a
recommendation to run dba/update-davical-database with the --nopatch
parameter.? Removing this erroneously added parameter made the update
work as expected:

The database is version 9.6 currently at revision 1.2.11.
Applying patch 1.2.12.sql ... succeeded.
Applying patch 1.3.1.sql ... succeeded.
Applying patch 1.3.2.sql ... succeeded.
Successfully applied 3 patches.
Supported locales updated.
Updated view: dav_principal.sql applied.
CalDAV functions updated.
RRULE functions updated.
Database permissions updated.

Thanks again.

I wish you a happy New Year!

Lukas