Mailing List Archive: Problemwith unicode

Problemwith unicode

Jun 28, 2016, 2:56 PM

Post #1 of 3 (1526 views)

Hi list, my system is Zope 2.11.4 with default enconding iso-8859-1 for
in/out and SQL Server 2008.

I'm trying to fix a bad desing with a sql server database. it has a mix of
encodings, traditional_spanish and modern_spanish. And some tables with
both type of fields, unicode and normal (nvarchar and varchar) .

The new database is Latin-1 enconding (SQL_Latin1_General_CP1_CI_AI) with
only varchar fields.

Using the ZMI if I check the "Z SQL Method" the data is displayed
correctly, but using a "page template" it gives me this error

*Expression: <StringExpr u' ${repeat/items/number}. ${items/nombre}'>*

{'container': <Folder at /control-emisiones/campos used for
/control-emisiones/oficio/alta>,
'context': <Folder at /control-emisiones/campos used for
/control-emisiones/oficio/alta>,
'default': <object object at 0x00ADC528>,
'here': <Folder at /control-emisiones/campos used for
/control-emisiones/oficio/alta>,
'loop': {u'items': <Products.PageTemplates.Expressions.PathIterator
object at 0x078877D0>},
'nothing': None,
'options': {'busqueda': 1, 'args': ()},
'repeat': <Products.PageTemplates.Expressions.SafeMapping object at
0x07802530>,
'request': <HTTPRequest,
URL=http://10.28.1.2:91/control-emisiones/oficio/alta/index_html>,
'root': <Application at >,
'template': <ZopePageTemplate at /control-emisiones/campos/campo__oficio_tipo>,
'traverse_subpath': [],
'user': <User 'admin'>}

- Module zope.tales.expressions, line 263, in __call__UnicodeDecodeError:
'ascii' codec can't decode byte 0xf3 in position 10: ordinal not in
range(128)

The data transfer was made using database tools (DTWizard).

why it is happening this?

--
________________________________________
Lo bueno de vivir un dia mas
es saber que nos queda un dia menos de vida

Re: Problemwith unicode [ In reply to ]

yourpadre at gmail

Jun 29, 2016, 1:50 PM

Post #2 of 3 (1502 views)

Permalink

After search a while, someone can confirm if I undestand it well? the
problem is because it is trying to decode to "ascii" instead of "iso-8859-1"

if it's true, why is doing it? I set to use iso-8859-1 in my conf file

2016-06-28 16:56 GMT-05:00 Miguel Beltran R. <yourpadre@gmail.com>:

> Hi list, my system is Zope 2.11.4 with default enconding iso-8859-1 for
> in/out and SQL Server 2008.
>
> I'm trying to fix a bad desing with a sql server database. it has a mix of
> encodings, traditional_spanish and modern_spanish. And some tables with
> both type of fields, unicode and normal (nvarchar and varchar) .
>
> The new database is Latin-1 enconding (SQL_Latin1_General_CP1_CI_AI) with
> only varchar fields.
>
> Using the ZMI if I check the "Z SQL Method" the data is displayed
> correctly, but using a "page template" it gives me this error
>
>
> *Expression: <StringExpr u' ${repeat/items/number}. ${items/nombre}'>*
>
> {'container': <Folder at /control-emisiones/campos used for /control-emisiones/oficio/alta>,
> 'context': <Folder at /control-emisiones/campos used for /control-emisiones/oficio/alta>,
> 'default': <object object at 0x00ADC528>,
> 'here': <Folder at /control-emisiones/campos used for /control-emisiones/oficio/alta>,
> 'loop': {u'items': <Products.PageTemplates.Expressions.PathIterator object at 0x078877D0>},
> 'nothing': None,
> 'options': {'busqueda': 1, 'args': ()},
> 'repeat': <Products.PageTemplates.Expressions.SafeMapping object at 0x07802530>,
> 'request': <HTTPRequest, URL=http://10.28.1.2:91/control-emisiones/oficio/alta/index_html>,
> 'root': <Application at >,
> 'template': <ZopePageTemplate at /control-emisiones/campos/campo__oficio_tipo>,
> 'traverse_subpath': [],
> 'user': <User 'admin'>}
>
> - Module zope.tales.expressions, line 263, in __call__UnicodeDecodeError:
> 'ascii' codec can't decode byte 0xf3 in position 10: ordinal not in
> range(128)
>
>
> The data transfer was made using database tools (DTWizard).
>
> why it is happening this?
>
>
> --
> ________________________________________
> Lo bueno de vivir un dia mas
> es saber que nos queda un dia menos de vida
>

--
________________________________________
Lo bueno de vivir un dia mas
es saber que nos queda un dia menos de vida

Re: Problemwith unicode [ In reply to ]

yourpadre at gmail

Jul 4, 2016, 10:49 AM

Post #3 of 3 (1489 views)

Permalink

2016-06-29 20:46 GMT-05:00 Alberto Berti <azazel@metapensiero.it>:

> >>>>> "Miguel" == Miguel Beltran R <yourpadre@gmail.com> writes:
>
>
> Miguel> After search a while, someone can confirm if I undestand it
> well? the
> Miguel> problem is because it is trying to decode to "ascii" instead
> of "iso-8859-1"
>
> Miguel> if it's true, why is doing it? I set to use iso-8859-1 in my
> conf file
>
> Miguel> 2016-06-28 16:56 GMT-05:00 Miguel Beltran R. <
> yourpadre@gmail.com>:
>
> >> Using the ZMI if I check the "Z SQL Method" the data is displayed
> >> correctly, but using a "page template" it gives me this error
> >>
> >>
> >> *Expression: <StringExpr u' ${repeat/items/number}.
> ${items/nombre}'>*
> >>
> >> - Module zope.tales.expressions, line 263, in
> __call__UnicodeDecodeError:
> >> 'ascii' codec can't decode byte 0xf3 in position 10: ordinal not in
> >> range(128)
> >>
>
> this is due probably because the template engine tries to convert a
> string into an unicode, but it doesn't know the charset of your data, so
> it's trying to use the default encoding as set in
> /usr/lib/python2.7/site.py during interpreter bootstrap. If you are
> using a good database you can trust your data and and let the database
> connector (the encoding should be settable somewhere in the connector or
> Z SQL method conf, i don't remember since i don't use them anymore)
> convert the strings to unicodes for you.
>
> If you have something like MySQL you can't always trust that the data in
> the db is in the expected encoding because some of its table engine
> simply don't do any check on the input. If this is your case, it will be
> more problematic.
>
> In my experience, setting a default encoding in the config file didn't
> always fix the problem.
>
> As a last resort you can tweak the site.py, but really if there issues
> with data's encoding this doesn't mean you solved the problem. Likely
> the user will see wrong charachters where non-ascii ones are supposed to
> be.
>

Thanks Alberto!!! :D

Changing the enconding in the site.py file made the trick.

To some zope developer in the list, why is not taking the enconding from
the zope.conf file? Do I must open a ticket?

--
________________________________________
Lo bueno de vivir un dia mas
es saber que nos queda un dia menos de vida