Mailing List Archive: UTF8 problems with plugin::encoding

UTF8 problems with plugin::encoding

Jul 21, 2014, 3:57 AM

Post #1 of 9 (3430 views)

Hey all,

on most of my website running on (latest catalyst: 5.90065) i always get utf8 related errors.
the usually appear if a spider
Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
comes accross.

the error is:
Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 167.

It took me while to get the actual parameters the spiders sends because the debug-message of catalyst do not tell that much :...

—————————————
[2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/…./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s) [10682] [Wed Jul 16 15:08:47 2014] ***
[2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim /usr/local/…./lib/perl5/Catalyst.pm +2309: Response Code: 400; Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
[2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s (154.059/s)
.---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
| Action | Time |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
'---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
—————————————

i changed to Plugin::Unicode::Encoding plugin a bit to find out what the client sends … the results are these:
UTF8 trash arrives - and the module seems unable to deal with it…

————————————
Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 170.
-

URL: notice/list

PARAMS:$VAR1 = {
'X*Ö^K^@^@^@^@¸®ä
^@^@^@^@8<83>^H^K^@^@^@^@h¡ä
^@^@^@^@Hµä
^@^@^@^@X^Z^N^Q^@^@^@^@ø<91>^F^Q^@^@^@^@Ø^F^N^Q^@^@^@^@¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@¸úÝ^P^@^@^@^@^X%q^G^@^@^@^@Ø§ñ^O^@^@^@^@ØøB.^@^@^@^@èâÝ^P^@^@^@^@XÛ_^L^@^@^@^@ÈíÝ^P^@^@^@^@¸~P^S^@^@^@^@èåÝ^P^@^@^@^@Øný^O^@^@^@^@<88>úÝ^P^@^@^@^@^Xá( ^@^@^@^@Ø¦Æ
^@^@^@^@Øï*^Q^@^@^@^@^X' => '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[.^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Ð¢^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@'
};

// value: $VAR1 = '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Ð¢^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@';

headers: Connection: close
Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Encoding: gzip, deflate
Accept-Language: zh;q=0.9,en;q=0.8
Host: wbc-inco.net
User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
Content-Length: 927
Content-Type: application/x-www-form-urlencoded
REFER: http://b------.net/“

————————————

to understand the logging above: this is what i added /changed in the Catalyst::Plugin::Unicode::Encoding

————————————————————
around line 168:

my $val;
eval {
$val = Encode::is_utf8( $value ) ? $value : $enc->decode( $value, $CHECK );
};
if ($@){
# UPS !
# get request infos
use Data::Dumper;
my $params = $self->req->parameters;
my $headers= $self->req->headers->as_string;
die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" . Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " . $headers;
….
————————————————————

I guess my Catalyst Apps are not the only ones with these errors ?

about my App settings / config:

app-config has
encoding UTF-8

App.pm does not load Unicode::Encoding anymore (since this is not need when using latest Catalyst: 5.90065)

i am using postgres with
pg_enable_utf8 1
(but the error about is far away from any DB related problem i guess)

using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with catalyxt)

i just checked out the tracker for catalyst on cpan, there is an UTF8 issue ticket
https://rt.cpan.org/Public/Bug/Display.html?id=94957
but i does not look as it was this problem ...

Any ideas what todo ?
Add a issue/ticket ?

thanks for feedback,
bernhard bauch

—
Bernhard Bauch, Webdevelopment
ZSI - Zentrum für soziale Innovation
bauch@zsi.at
Skype: berni-zsi

Re: UTF8 problems with plugin::encoding [ In reply to ]

list at fossilgroup

Jul 21, 2014, 5:23 AM

Post #2 of 9 (3384 views)

Permalink

On 07/21/2014 12:57 PM, Bernhard Bauch wrote:
> Hey all,
>
> on most of my website running on (latest catalyst: 5.90065) i always get
> utf8 related errors.

Hello,

I have no answer to your question, I just want to add something related.
Several days ago I created application with Catalyst, TT, FormFu and
PostgreSQL. Immediately after I put [% form %] into my template file,
encoding was broken. I spent several hours trying to fix it. Also, there
is Catalyst::Plugin::Unicode which seems to be deprecated,
Catalyst::Plugin::Unicode::Encoding. Maybe it is just because I'm
relatively new to Catalyst and other stuff, but I think it is confusing.

It was necessary to 'use utf8', add 'ENCODING UTF-8' both for
Controller::HTML::FormFu and View::HTML into config file and set
pg_enable_utf8 => 1 for DBIC. Maybe it is possible to change Catalyst in
order to read/write encoding information only once and use it for whole
site/templates/forms/etc?

Karol

_______________________________________________
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/

Re: UTF8 problems with plugin::encoding [ In reply to ]

winfinit at gmail

Jul 21, 2014, 5:36 AM

Post #3 of 9 (3382 views)

Permalink

Hello,

How are you running your application? Ie: mod_perl, fcgi, fcgi +
httpd/nginx, plack + ... also what version of perl are you using and what
os?

-roman
On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <bauch@zsi.at> wrote:

> Hey all,
>
> on most of my website running on (latest catalyst: 5.90065) i always get
> utf8 related errors.
> the usually appear if a spider
> Mozilla/5.0 (compatible; EasouSpider; +
> http://www.easou.com/search/spider.html)
> comes accross.
>
> the error is:
> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to
> Unicode at /usr/local/â€¦./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line
> 167.
>
> It took me while to get the actual parameters the spiders sends because
> the debug-message of catalyst do not tell that much :...
>
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim
> /usr/local/â€¦./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s)
> [10682] [Wed Jul 16 15:08:47 2014] ***
> [2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim
> /usr/local/â€¦./lib/perl5/Catalyst.pm +2309: Response Code: 400;
> Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim
> /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s
> (154.059/s)
>
> .---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
> | Action
>
> | Time |
>
> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
>
> '---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>
> i changed to Plugin::Unicode::Encoding plugin a bit to find out what the
> client sends â€¦ the results are these:
> UTF8 trash arrives - and the module seems unable to deal with itâ€¦
>
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to
> Unicode at /usr/local/â€¦../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm
> line 170.
> -
>
> URL: notice/list
>
> PARAMS:$VAR1 = {
> 'X*Ã–^K^@^@^@^@Â¸Â®Ã¤
> ^@^@^@^@8<83>^H^K^@^@^@^@hÂ¡Ã¤
> ^@^@^@^@HÂµÃ¤
> ^@^@^@^@X^Z^N^Q^@^@^@^@Ã¸<91>^F^Q^@^@^@^@Ã˜^F^N^Q^@^@^@^@Â¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@Â¸ÃºÃ^P^@^@^@^@^X%q^G^@^@^@^@Ã˜Â§Ã±^O^@^@^@^@Ã˜Ã¸B.^@^@^@^@Ã¨Ã¢Ã^P^@^@^@^@XÃ›_^L^@^@^@^@ÃˆÃÃ^P^@^@^@^@Â¸~P^S^@^@^@^@Ã¨Ã¥Ã^P^@^@^@^@Ã˜nÃ½^O^@^@^@^@<88>ÃºÃ^P^@^@^@^@^XÃ¡(
> ^@^@^@^@Ã˜Â¦Ã†
> ^@^@^@^@Ã˜Ã¯*^Q^@^@^@^@^X'
> => '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@
> J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@
> <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@
> <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@'
> };
>
>
> // value: $VAR1
> = '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@
> J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@
> <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@
> <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@';
>
>
> headers: Connection: close
> Accept: text/html, application/xml;q=0.9, application/xhtml+xml,
> image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
> Accept-Encoding: gzip, deflate
> Accept-Language: zh;q=0.9,en;q=0.8
> Host: wbc-inco.net
> User-Agent: Mozilla/5.0 (compatible; EasouSpider; +
> http://www.easou.com/search/spider.html)
> Content-Length: 927
> Content-Type: application/x-www-form-urlencoded
> REFER: http://b------.net/â€œ
>
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>
> to understand the logging above: this is what i added /changed in
> the Catalyst::Plugin::Unicode::Encoding
>
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
> around line 168:
>
> my $val;
> eval {
> $val = Encode::is_utf8( $value ) ? $value : $enc->decode(
> $value, $CHECK );
> };
> if ($@){
> # UPS !
> # get request infos
> use Data::Dumper;
> my $params = $self->req->parameters;
> my $headers= $self->req->headers->as_string;
> die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" .
> Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " .
> $headers;
> â€¦.
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>
> I guess my Catalyst Apps are not the only ones with these errors ?
>
>
> about my App settings / config:
>
> app-config has
> encoding UTF-8
>
> App.pm does not load Unicode::Encoding anymore (since this is not need
> when using latest Catalyst: 5.90065)
>
> i am using postgres with
> pg_enable_utf8 1
> (but the error about is far away from any DB related problem i guess)
>
> using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with
> catalyxt)
>
> i just checked out the tracker for catalyst on cpan, there is an UTF8
> issue ticket
> https://rt.cpan.org/Public/Bug/Display.html?id=94957
> but i does not look as it was this problem ...
>
> Any ideas what todo ?
> Add a issue/ticket ?
>
> thanks for feedback,
> bernhard bauch
>
>
>
> â€”
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum fÃ¼r soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>

Re: UTF8 problems with plugin::encoding [ In reply to ]

bauch at zsi

Jul 21, 2014, 11:41 AM

Post #4 of 9 (3391 views)

Permalink

hey roman and all,

On 21 Jul 2014, at 14:36, Roman Winfinit <winfinit@gmail.com> wrote:

> Hello,
>
> How are you running your application? Ie: mod_perl, fcgi, fcgi + httpd/nginx, plack + …
>
apache2 + fastcgi
> also what version of perl are you using and what os ?
>
>
debian wheezy,
perl 5.20.0
(but the errors also occure on another system with an older perl version 5.12.something)

need more infos ?
bye, berni

> -roman
>
> On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <bauch@zsi.at> wrote:
> Hey all,
>
> on most of my website running on (latest catalyst: 5.90065) i always get utf8 related errors.
> the usually appear if a spider
> Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
> comes accross.
>
> the error is:
> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 167.
>
> It took me while to get the actual parameters the spiders sends because the debug-message of catalyst do not tell that much :...
>
> —————————————
> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/…./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s) [10682] [Wed Jul 16 15:08:47 2014] ***
> [2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim /usr/local/…./lib/perl5/Catalyst.pm +2309: Response Code: 400; Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s (154.059/s)
> .---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
> | Action | Time |
> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
> '---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
> —————————————
>
> i changed to Plugin::Unicode::Encoding plugin a bit to find out what the client sends … the results are these:
> UTF8 trash arrives - and the module seems unable to deal with it…
>
> ————————————
> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 170.
> -
>
> URL: notice/list
>
> PARAMS:$VAR1 = {
> 'X*Ö^K^@^@^@^@¸®ä
> ^@^@^@^@8<83>^H^K^@^@^@^@h¡ä
> ^@^@^@^@Hµä
> ^@^@^@^@X^Z^N^Q^@^@^@^@ø<91>^F^Q^@^@^@^@Ø^F^N^Q^@^@^@^@¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@¸úÝ^P^@^@^@^@^X%q^G^@^@^@^@Ø§ñ^O^@^@^@^@ØøB.^@^@^@^@èâÝ^P^@^@^@^@XÛ_^L^@^@^@^@ÈíÝ^P^@^@^@^@¸~P^S^@^@^@^@èåÝ^P^@^@^@^@Øný^O^@^@^@^@<88>úÝ^P^@^@^@^@^Xá( ^@^@^@^@Ø¦Æ
> ^@^@^@^@Øï*^Q^@^@^@^@^X' => '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[.^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Ð¢^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@'
> };
>
>
> // value: $VAR1 = '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Ð¢^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@';
>
>
> headers: Connection: close
> Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
> Accept-Encoding: gzip, deflate
> Accept-Language: zh;q=0.9,en;q=0.8
> Host: wbc-inco.net
> User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
> Content-Length: 927
> Content-Type: application/x-www-form-urlencoded
> REFER: http://b------.net/“
>
> ————————————
>
> to understand the logging above: this is what i added /changed in the Catalyst::Plugin::Unicode::Encoding
>
> ————————————————————
> around line 168:
>
> my $val;
> eval {
> $val = Encode::is_utf8( $value ) ? $value : $enc->decode( $value, $CHECK );
> };
> if ($@){
> # UPS !
> # get request infos
> use Data::Dumper;
> my $params = $self->req->parameters;
> my $headers= $self->req->headers->as_string;
> die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" . Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " . $headers;
> ….
> ————————————————————
>
> I guess my Catalyst Apps are not the only ones with these errors ?
>
>
> about my App settings / config:
>
> app-config has
> encoding UTF-8
>
> App.pm does not load Unicode::Encoding anymore (since this is not need when using latest Catalyst: 5.90065)
>
> i am using postgres with
> pg_enable_utf8 1
> (but the error about is far away from any DB related problem i guess)
>
> using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with catalyxt)
>
> i just checked out the tracker for catalyst on cpan, there is an UTF8 issue ticket
> https://rt.cpan.org/Public/Bug/Display.html?id=94957
> but i does not look as it was this problem ...
>
> Any ideas what todo ?
> Add a issue/ticket ?
>
> thanks for feedback,
> bernhard bauch
>
>
>
> —
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum für soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
> !DSPAM:53cd09a3104511692032419! _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> !DSPAM:53cd09a3104511692032419!

—
Bernhard Bauch, Webdevelopment
ZSI - Zentrum für soziale Innovation
bauch@zsi.at
Skype: berni-zsi

Re: UTF8 problems with plugin::encoding [ In reply to ]

bauch at zsi

Jul 21, 2014, 1:19 PM

Post #5 of 9 (3382 views)

Permalink

more news..

the crawler/searcheinge that triggers these errors is
http://easou.com

this searchengine delivers their pages not in UTF8 — but in “gb2312” which is “simple chinese”
if i open the “wrong utf8” parameters from the faulty requests with “gb2312” some readable signs appear.
>> this leads me to: catalyst does not handle requests with gb2312 encoded parameters (because they are not utf8) -and the request does not promote that it is encoded in other than utf8.

any ideas what to do ?

bye, bernhard

On 21 Jul 2014, at 14:36, Roman Winfinit <winfinit@gmail.com> wrote:

> Hello,
>
> How are you running your application? Ie: mod_perl, fcgi, fcgi + httpd/nginx, plack + ... also what version of perl are you using and what os?
>
> -roman
>
> On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <bauch@zsi.at> wrote:
> Hey all,
>
> on most of my website running on (latest catalyst: 5.90065) i always get utf8 related errors.
> the usually appear if a spider
> Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
> comes accross.
>
> the error is:
> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 167.
>
> It took me while to get the actual parameters the spiders sends because the debug-message of catalyst do not tell that much :...
>
> —————————————
> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/…./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s) [10682] [Wed Jul 16 15:08:47 2014] ***
> [2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim /usr/local/…./lib/perl5/Catalyst.pm +2309: Response Code: 400; Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s (154.059/s)
> .---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
> | Action | Time |
> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
> '---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
> —————————————
>
> i changed to Plugin::Unicode::Encoding plugin a bit to find out what the client sends … the results are these:
> UTF8 trash arrives - and the module seems unable to deal with it…
>
> ————————————
> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/…../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 170.
> -
>
> URL: notice/list
>
> PARAMS:$VAR1 = {
> 'X*Ö^K^@^@^@^@¸®ä
> ^@^@^@^@8<83>^H^K^@^@^@^@h¡ä
> ^@^@^@^@Hµä
> ^@^@^@^@X^Z^N^Q^@^@^@^@ø<91>^F^Q^@^@^@^@Ø^F^N^Q^@^@^@^@¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@¸úÝ^P^@^@^@^@^X%q^G^@^@^@^@Ø§ñ^O^@^@^@^@ØøB.^@^@^@^@èâÝ^P^@^@^@^@XÛ_^L^@^@^@^@ÈíÝ^P^@^@^@^@¸~P^S^@^@^@^@èåÝ^P^@^@^@^@Øný^O^@^@^@^@<88>úÝ^P^@^@^@^@^Xá( ^@^@^@^@Ø¦Æ
> ^@^@^@^@Øï*^Q^@^@^@^@^X' => '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[.^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Ð¢^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@'
> };
>
>
> // value: $VAR1 = '^F^L^@^@^@^@<98>Ûø^O^@^@^@^@Ø~^A^N^@^@^@^@<98>=H>^@^@^@^@ø<99>ó^K^@^@^@^@hÔu^R^@^@^@^@¸<8e>ó^K^@^@^@^@^Xä_^L^@^@^@^@Ø<90>a^G^@^@^@^@hðÉ^O^@^@^@^@8ã*^G^@^@^@^@ØØý^M^@^@^@^@Xùë^F^@^@^@^@^HÜý^M^@^@^@^@8W6^H^@^@^@^@øÐý^M^@^@^@^@xÿÃ^K^@^@^@^@X]i^O^@^@^@^@8^Mÿ^H^@^@^@^@Xû<98>^Q^@^@^@^@x¦h^H^@^@^@^@Xý<98>^Q^@^@^@^@^X=5^H^@^@^@^@^X¦ú^K^@^@^@^@^XVQ^P^@^@^@^@^H^Yû^N^@^@^@^@x¤h^H^@^@^@^@^Xå<98>^Q^@^@^@^@ø¤h^H^@^@^@^@Xé<98>^Q^@^@^@^@X¼h^H^@^@^@^@Ø¡h^H^@^@^@^@øf<82>^Q^@^@^@^@^X>éH^@^@^@^@xv<82>^Q^@^@^@^@X6éH^@^@^@^@xl<82>^Q^@^@^@^@83Ì^G^@^@^@^@Xl<82>^Q^@^@^@^@¸Ñý^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@¸Ë¢^K^@^@^@^@¸u<82>^Q^@^@^@^@<98>Á¢^K^@^@^@^@Øp<82>^Q^@^@^@^@8Í¢^K^@^@^@^@Øl<82>^Q^@^@^@^@XË¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@¸Å¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>Ð¢^K^@^@^@^@¨fÐ^Q^@^@^@^@ØÉ=^R^@^@^@^@ÀC<95>^M^@^@^@^@°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@À\\<95>^M^@^@^@^@ðE<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@àU<95>^M^@^@^@^@àF<95>^M^@^@^@^@àA<95>^M^@^@^@^@^@<9e>ô^P^@^@^@^@°<9d>ô^P^@^@^@^@0<91>ô^P^@^@^@^@ <9e>ô^P^@^@^@^@^P<8e>ô^P^@^@^@^@ <88>ô^P^@^@^@^@Ð<82>ô^P^@^@^@^@ <8d>ô^P^@^@^@^@<90><95>ô^P^@^@^@^@à<90>ô^P^@^@^@^@@<95>ô^P^@^@^@^@P<8f>ô^P^@^@^@^@<90><81>ô^P^@^@^@^@ <97>ô^P^@^@^@^@Ð<8c>ô^P^@^@^@^@p<88>ô^P^@^@^@^@P<99>ô^P^@^@^@^@<90><90>ô^P^@^@^@^@@<9a>ô^P^@^@^@^@0<9b>ô^P^@^@^@';
>
>
> headers: Connection: close
> Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
> Accept-Encoding: gzip, deflate
> Accept-Language: zh;q=0.9,en;q=0.8
> Host: wbc-inco.net
> User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
> Content-Length: 927
> Content-Type: application/x-www-form-urlencoded
> REFER: http://b------.net/“
>
> ————————————
>
> to understand the logging above: this is what i added /changed in the Catalyst::Plugin::Unicode::Encoding
>
> ————————————————————
> around line 168:
>
> my $val;
> eval {
> $val = Encode::is_utf8( $value ) ? $value : $enc->decode( $value, $CHECK );
> };
> if ($@){
> # UPS !
> # get request infos
> use Data::Dumper;
> my $params = $self->req->parameters;
> my $headers= $self->req->headers->as_string;
> die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" . Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " . $headers;
> ….
> ————————————————————
>
> I guess my Catalyst Apps are not the only ones with these errors ?
>
>
> about my App settings / config:
>
> app-config has
> encoding UTF-8
>
> App.pm does not load Unicode::Encoding anymore (since this is not need when using latest Catalyst: 5.90065)
>
> i am using postgres with
> pg_enable_utf8 1
> (but the error about is far away from any DB related problem i guess)
>
> using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with catalyxt)
>
> i just checked out the tracker for catalyst on cpan, there is an UTF8 issue ticket
> https://rt.cpan.org/Public/Bug/Display.html?id=94957
> but i does not look as it was this problem ...
>
> Any ideas what todo ?
> Add a issue/ticket ?
>
> thanks for feedback,
> bernhard bauch
>
>
>
> —
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum für soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
> !DSPAM:53cd09a3104511692032419! _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> !DSPAM:53cd09a3104511692032419!

—
Bernhard Bauch, Webdevelopment
ZSI - Zentrum für soziale Innovation
bauch@zsi.at
Skype: berni-zsi

Re: UTF8 problems with plugin::encoding [ In reply to ]

bauch at zsi

Jul 22, 2014, 2:33 AM

Post #6 of 9 (3382 views)

Permalink

hey all,

this pyton3 script triggers the error â€¦.

--------------------------------
import httplib2
import urllib.parse

somestr = 'æ·±å…¥ so what'
encodedstr = somestr.encode('gb2312')
url = 'http://myappdomain.com/search'
body = { encodedstr:encodedstr }
headers = {
'Content-type': 'application/x-www-form-urlencoded',
'Accept': 'text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'zh;q=0.9,en;q=0.8'
}
http = httplib2.Http()
response, content = http.request(url, 'POST', headers=headers, body=urllib.parse.urlencode(body))
â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”

now its possible to reproduce the error :)

any ideas how to solve this ?
ruby people did this with adding a utf8-sanitizer in the middleware..

bye, bernhard

On 21 Jul 2014, at 22:19, Bernhard Bauch <bauch@zsi.at> wrote:

> more news..
>
> the crawler/searcheinge that triggers these errors is
> http://easou.com
>
> this searchengine delivers their pages not in UTF8 â€” but in â€œgb2312â€ which is â€œsimple chineseâ€
> if i open the â€œwrong utf8â€ parameters from the faulty requests with â€œgb2312â€ some readable signs appear.
>
> any ideas what to do ?
>
> bye, bernhard
>
>
>
> On 21 Jul 2014, at 14:36, Roman Winfinit <winfinit@gmail.com> wrote:
>
>> Hello,
>>
>> How are you running your application? Ie: mod_perl, fcgi, fcgi + httpd/nginx, plack + ... also what version of perl are you using and what os?
>>
>> -roman
>>
>> On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <bauch@zsi.at> wrote:
>> Hey all,
>>
>> on most of my website running on (latest catalyst: 5.90065) i always get utf8 related errors.
>> the usually appear if a spider
>> Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
>> comes accross.
>>
>> the error is:
>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/â€¦./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 167.
>>
>> It took me while to get the actual parameters the spiders sends because the debug-message of catalyst do not tell that much :...
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/â€¦./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s) [10682] [Wed Jul 16 15:08:47 2014] ***
>> [2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim /usr/local/â€¦./lib/perl5/Catalyst.pm +2309: Response Code: 400; Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s (154.059/s)
>> .---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
>> | Action | Time |
>> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
>> '---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>
>> i changed to Plugin::Unicode::Encoding plugin a bit to find out what the client sends â€¦ the results are these:
>> UTF8 trash arrives - and the module seems unable to deal with itâ€¦
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/â€¦../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 170.
>> -
>>
>> URL: notice/list
>>
>> PARAMS:$VAR1 = {
>> 'X*Ã–^K^@^@^@^@Â¸Â®Ã¤
>> ^@^@^@^@8<83>^H^K^@^@^@^@hÂ¡Ã¤
>> ^@^@^@^@HÂµÃ¤
>> ^@^@^@^@X^Z^N^Q^@^@^@^@Ã¸<91>^F^Q^@^@^@^@Ã˜^F^N^Q^@^@^@^@Â¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@Â¸ÃºÃ^P^@^@^@^@^X%q^G^@^@^@^@Ã˜Â§Ã±^O^@^@^@^@Ã˜Ã¸B.^@^@^@^@Ã¨Ã¢Ã^P^@^@^@^@XÃ›_^L^@^@^@^@ÃˆÃÃ^P^@^@^@^@Â¸~P^S^@^@^@^@Ã¨Ã¥Ã^P^@^@^@^@Ã˜nÃ½^O^@^@^@^@<88>ÃºÃ^P^@^@^@^@^XÃ¡( ^@^@^@^@Ã˜Â¦Ã†
>> ^@^@^@^@Ã˜Ã¯*^Q^@^@^@^@^X' => '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[.^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@ <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@ <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@'
>> };
>>
>>
>> // value: $VAR1 = '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@ <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@ <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@';
>>
>>
>> headers: Connection: close
>> Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
>> Accept-Encoding: gzip, deflate
>> Accept-Language: zh;q=0.9,en;q=0.8
>> Host: wbc-inco.net
>> User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
>> Content-Length: 927
>> Content-Type: application/x-www-form-urlencoded
>> REFER: http://b------.net/â€œ
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>
>> to understand the logging above: this is what i added /changed in the Catalyst::Plugin::Unicode::Encoding
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>> around line 168:
>>
>> my $val;
>> eval {
>> $val = Encode::is_utf8( $value ) ? $value : $enc->decode( $value, $CHECK );
>> };
>> if ($@){
>> # UPS !
>> # get request infos
>> use Data::Dumper;
>> my $params = $self->req->parameters;
>> my $headers= $self->req->headers->as_string;
>> die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" . Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " . $headers;
>> â€¦.
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>
>> I guess my Catalyst Apps are not the only ones with these errors ?
>>
>>
>> about my App settings / config:
>>
>> app-config has
>> encoding UTF-8
>>
>> App.pm does not load Unicode::Encoding anymore (since this is not need when using latest Catalyst: 5.90065)
>>
>> i am using postgres with
>> pg_enable_utf8 1
>> (but the error about is far away from any DB related problem i guess)
>>
>> using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with catalyxt)
>>
>> i just checked out the tracker for catalyst on cpan, there is an UTF8 issue ticket
>> https://rt.cpan.org/Public/Bug/Display.html?id=94957
>> but i does not look as it was this problem ...
>>
>> Any ideas what todo ?
>> Add a issue/ticket ?
>>
>> thanks for feedback,
>> bernhard bauch
>>
>>
>>
>> â€”
>> Bernhard Bauch, Webdevelopment
>> ZSI - Zentrum fÃ¼r soziale Innovation
>> bauch@zsi.at
>> Skype: berni-zsi
>>
>>
>> _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>> !DSPAM:53cd09a3104511692032419! _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
>> !DSPAM:53cd09a3104511692032419!
>
> â€”
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum fÃ¼r soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> !DSPAM:53cd7626104517769513966!

â€”
Bernhard Bauch, Webdevelopment
ZSI - Zentrum fÃ¼r soziale Innovation
bauch@zsi.at
Skype: berni-zsi

Re: UTF8 problems with plugin::encoding [ In reply to ]

bauch at zsi

Jul 22, 2014, 3:21 AM

Post #7 of 9 (3384 views)

Permalink

hereâ€™s also a perl-script that does it

------------------------------------------
use Encode qw(decode encode);
use LWP::UserAgent;

my $str = 'æ·±å…¥ so what';
my $oct = encode("gb2312", $str);
my $url = 'http://wbc-inco.net/object/event/past';
my $ua = LWP::UserAgent->new();
my $response = $ua->post( $url, { $oct => $oct } );
my $content = $response->decoded_content();
------------------------------------------

On 22 Jul 2014, at 11:33, Bernhard Bauch <bauch@zsi.at> wrote:

> hey all,
>
> this pyton3 script triggers the error â€¦.
>
> --------------------------------
> import httplib2
> import urllib.parse
>
> somestr = 'æ·±å…¥ so what'
> encodedstr = somestr.encode('gb2312')
> url = 'http://myappdomain.com/search'
> body = { encodedstr:encodedstr }
> headers = {
> 'Content-type': 'application/x-www-form-urlencoded',
> 'Accept': 'text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1',
> 'Accept-Encoding': 'gzip, deflate',
> 'Accept-Language': 'zh;q=0.9,en;q=0.8'
> }
> http = httplib2.Http()
> response, content = http.request(url, 'POST', headers=headers, body=urllib.parse.urlencode(body))
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>
> now its possible to reproduce the error :)
>
> any ideas how to solve this ?
> ruby people did this with adding a utf8-sanitizer in the middleware..
>
> bye, bernhard
>
>
> On 21 Jul 2014, at 22:19, Bernhard Bauch <bauch@zsi.at> wrote:
>
>> more news..
>>
>> the crawler/searcheinge that triggers these errors is
>> http://easou.com
>>
>> this searchengine delivers their pages not in UTF8 â€” but in â€œgb2312â€ which is â€œsimple chineseâ€
>> if i open the â€œwrong utf8â€ parameters from the faulty requests with â€œgb2312â€ some readable signs appear.
>> >> this leads me to: catalyst does not handle requests with gb2312 encoded parameters (because they are not utf8) -and the request does not promote that it is encoded in other than utf8.
>>
>> any ideas what to do ?
>>
>> bye, bernhard
>>
>>
>>
>> On 21 Jul 2014, at 14:36, Roman Winfinit <winfinit@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> How are you running your application? Ie: mod_perl, fcgi, fcgi + httpd/nginx, plack + ... also what version of perl are you using and what os?
>>>
>>> -roman
>>>
>>> On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <bauch@zsi.at> wrote:
>>> Hey all,
>>>
>>> on most of my website running on (latest catalyst: 5.90065) i always get utf8 related errors.
>>> the usually appear if a spider
>>> Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
>>> comes accross.
>>>
>>> the error is:
>>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/â€¦./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 167.
>>>
>>> It took me while to get the actual parameters the spiders sends because the debug-message of catalyst do not tell that much :...
>>>
>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/â€¦./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s) [10682] [Wed Jul 16 15:08:47 2014] ***
>>> [2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim /usr/local/â€¦./lib/perl5/Catalyst.pm +2309: Response Code: 400; Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
>>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s (154.059/s)
>>> .---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
>>> | Action | Time |
>>> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
>>> '---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>
>>> i changed to Plugin::Unicode::Encoding plugin a bit to find out what the client sends â€¦ the results are these:
>>> UTF8 trash arrives - and the module seems unable to deal with itâ€¦
>>>
>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/â€¦../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 170.
>>> -
>>>
>>> URL: notice/list
>>>
>>> PARAMS:$VAR1 = {
>>> 'X*Ã–^K^@^@^@^@Â¸Â®Ã¤
>>> ^@^@^@^@8<83>^H^K^@^@^@^@hÂ¡Ã¤
>>> ^@^@^@^@HÂµÃ¤
>>> ^@^@^@^@X^Z^N^Q^@^@^@^@Ã¸<91>^F^Q^@^@^@^@Ã˜^F^N^Q^@^@^@^@Â¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@Â¸ÃºÃ^P^@^@^@^@^X%q^G^@^@^@^@Ã˜Â§Ã±^O^@^@^@^@Ã˜Ã¸B.^@^@^@^@Ã¨Ã¢Ã^P^@^@^@^@XÃ›_^L^@^@^@^@ÃˆÃÃ^P^@^@^@^@Â¸~P^S^@^@^@^@Ã¨Ã¥Ã^P^@^@^@^@Ã˜nÃ½^O^@^@^@^@<88>ÃºÃ^P^@^@^@^@^XÃ¡( ^@^@^@^@Ã˜Â¦Ã†
>>> ^@^@^@^@Ã˜Ã¯*^Q^@^@^@^@^X' => '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[.^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@ <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@ <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@'
>>> };
>>>
>>>
>>> // value: $VAR1 = '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@ <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@ <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@';
>>>
>>>
>>> headers: Connection: close
>>> Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
>>> Accept-Encoding: gzip, deflate
>>> Accept-Language: zh;q=0.9,en;q=0.8
>>> Host: wbc-inco.net
>>> User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
>>> Content-Length: 927
>>> Content-Type: application/x-www-form-urlencoded
>>> REFER: http://b------.net/â€œ
>>>
>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>
>>> to understand the logging above: this is what i added /changed in the Catalyst::Plugin::Unicode::Encoding
>>>
>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>> around line 168:
>>>
>>> my $val;
>>> eval {
>>> $val = Encode::is_utf8( $value ) ? $value : $enc->decode( $value, $CHECK );
>>> };
>>> if ($@){
>>> # UPS !
>>> # get request infos
>>> use Data::Dumper;
>>> my $params = $self->req->parameters;
>>> my $headers= $self->req->headers->as_string;
>>> die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" . Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " . $headers;
>>> â€¦.
>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>
>>> I guess my Catalyst Apps are not the only ones with these errors ?
>>>
>>>
>>> about my App settings / config:
>>>
>>> app-config has
>>> encoding UTF-8
>>>
>>> App.pm does not load Unicode::Encoding anymore (since this is not need when using latest Catalyst: 5.90065)
>>>
>>> i am using postgres with
>>> pg_enable_utf8 1
>>> (but the error about is far away from any DB related problem i guess)
>>>
>>> using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with catalyxt)
>>>
>>> i just checked out the tracker for catalyst on cpan, there is an UTF8 issue ticket
>>> https://rt.cpan.org/Public/Bug/Display.html?id=94957
>>> but i does not look as it was this problem ...
>>>
>>> Any ideas what todo ?
>>> Add a issue/ticket ?
>>>
>>> thanks for feedback,
>>> bernhard bauch
>>>
>>>
>>>
>>> â€”
>>> Bernhard Bauch, Webdevelopment
>>> ZSI - Zentrum fÃ¼r soziale Innovation
>>> bauch@zsi.at
>>> Skype: berni-zsi
>>>
>>>
>>> _______________________________________________
>>> List: Catalyst@lists.scsys.co.uk
>>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>>> Dev site: http://dev.catalyst.perl.org/
>>>
>>> !DSPAM:53cd09a3104511692032419! _______________________________________________
>>> List: Catalyst@lists.scsys.co.uk
>>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>>> Dev site: http://dev.catalyst.perl.org/
>>>
>>>
>>> !DSPAM:53cd09a3104511692032419!
>>
>> â€”
>> Bernhard Bauch, Webdevelopment
>> ZSI - Zentrum fÃ¼r soziale Innovation
>> bauch@zsi.at
>> Skype: berni-zsi
>>
>> _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
>> !DSPAM:53cd7626104517769513966!
>
> â€”
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum fÃ¼r soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> !DSPAM:53ce305e104511469956211!

â€”
Bernhard Bauch, Webdevelopment
ZSI - Zentrum fÃ¼r soziale Innovation
bauch@zsi.at
Skype: berni-zsi

Re: UTF8 problems with plugin::encoding [ In reply to ]

m at rkellis

Jul 22, 2014, 4:31 AM

Post #8 of 9 (3399 views)

Permalink

I don't think there's anything you can do, you're app wants utf8 and
they're sending something else which doesn't map. and since you can't know
what format it is in, then all you can do is die if it doesn't map, which
is what the plugin does.

as far as i can tell the ruby middleware i found handles this by returning
a 400 bad request, which cataylst does as well. so there's no affect, other
than the noise in the logs.

On 22 July 2014 11:21, Bernhard Bauch <bauch@zsi.at> wrote:

> hereâ€™s also a perl-script that does it
>
> ------------------------------------------
> use Encode qw(decode encode);
> use LWP::UserAgent;
>
> my $str = 'æ·±å…¥ so what';
> my $oct = encode("gb2312", $str);
> my $url = 'http://wbc-inco.net/object/event/past';
> my $ua = LWP::UserAgent->new();
> my $response = $ua->post( $url, { $oct => $oct } );
> my $content = $response->decoded_content();
> ------------------------------------------
>
> On 22 Jul 2014, at 11:33, Bernhard Bauch <bauch@zsi.at> wrote:
>
> hey all,
>
> this pyton3 script triggers the error â€¦.
>
> --------------------------------
> import httplib2
> import urllib.parse
>
> somestr = 'æ·±å…¥ so what'
> encodedstr = somestr.encode('gb2312')
> url = 'http://myappdomain.com/search'
> body = { encodedstr:encodedstr }
> headers = {
> 'Content-type': 'application/x-www-form-urlencoded',
> 'Accept': 'text/html, application/xml;q=0.9, application/xhtml+xml,
> image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1',
> 'Accept-Encoding': 'gzip, deflate',
> 'Accept-Language': 'zh;q=0.9,en;q=0.8'
> }
> http = httplib2.Http()
> response, content = http.request(url, 'POST', headers=headers,
> body=urllib.parse.urlencode(body))
> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>
> now its possible to reproduce the error :)
>
> any ideas how to solve this ?
> ruby people did this with adding a utf8-sanitizer in the middleware..
>
> bye, bernhard
>
>
> On 21 Jul 2014, at 22:19, Bernhard Bauch <bauch@zsi.at> wrote:
>
> more news..
>
> the crawler/searcheinge that triggers these errors is
> http://easou.com
>
> this searchengine delivers their pages not in UTF8 â€” but in â€œgb2312â€ which
> is â€œsimple chineseâ€
> if i open the â€œwrong utf8â€ parameters from the faulty requests with
> â€œgb2312â€ some readable signs appear.
> parameters (because they are not utf8) -and the request does not promote
> that it is encoded in other than utf8.
>
> any ideas what to do ?
>
> bye, bernhard
>
>
>
> On 21 Jul 2014, at 14:36, Roman Winfinit <winfinit@gmail.com> wrote:
>
> Hello,
>
> How are you running your application? Ie: mod_perl, fcgi, fcgi +
> httpd/nginx, plack + ... also what version of perl are you using and what
> os?
>
> -roman
> On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <bauch@zsi.at> wrote:
>
>> Hey all,
>>
>> on most of my website running on (latest catalyst: 5.90065) i always get
>> utf8 related errors.
>> the usually appear if a spider
>> Mozilla/5.0 (compatible; EasouSpider; +
>> http://www.easou.com/search/spider.html)
>> comes accross.
>>
>> the error is:
>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to
>> Unicode at /usr/local/â€¦./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line
>> 167.
>>
>> It took me while to get the actual parameters the spiders sends because
>> the debug-message of catalyst do not tell that much :...
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim
>> /usr/local/â€¦./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s)
>> [10682] [Wed Jul 16 15:08:47 2014] ***
>> [2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim
>> /usr/local/â€¦./lib/perl5/Catalyst.pm +2309: Response Code: 400;
>> Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim
>> /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s
>> (154.059/s)
>>
>> .---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
>> | Action
>>
>> | Time |
>>
>> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
>>
>> '---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>
>> i changed to Plugin::Unicode::Encoding plugin a bit to find out what the
>> client sends â€¦ the results are these:
>> UTF8 trash arrives - and the module seems unable to deal with itâ€¦
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to
>> Unicode at /usr/local/â€¦../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm
>> line 170.
>> -
>>
>> URL: notice/list
>>
>> PARAMS:$VAR1 = {
>> 'X*Ã–^K^@^@^@^@Â¸Â®Ã¤
>> ^@^@^@^@8<83>^H^K^@^@^@^@hÂ¡Ã¤
>> ^@^@^@^@HÂµÃ¤
>> ^@^@^@^@X^Z^N^Q^@^@^@^@Ã¸<91>^F^Q^@^@^@^@Ã˜^F^N^Q^@^@^@^@Â¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@Â¸ÃºÃ^P^@^@^@^@^X%q^G^@^@^@^@Ã˜Â§Ã±^O^@^@^@^@Ã˜Ã¸B.^@^@^@^@Ã¨Ã¢Ã^P^@^@^@^@XÃ›_^L^@^@^@^@ÃˆÃÃ^P^@^@^@^@Â¸~P^S^@^@^@^@Ã¨Ã¥Ã^P^@^@^@^@Ã˜nÃ½^O^@^@^@^@<88>ÃºÃ^P^@^@^@^@^XÃ¡(
>> ^@^@^@^@Ã˜Â¦Ã†
>> ^@^@^@^@Ã˜Ã¯*^Q^@^@^@^@^X'
>> => '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@
>> J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@
>> <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@
>> <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@'
>> };
>>
>>
>> // value: $VAR1
>> = '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@
>> J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@
>> <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@
>> <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@';
>>
>>
>> headers: Connection: close
>> Accept: text/html, application/xml;q=0.9, application/xhtml+xml,
>> image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
>> Accept-Encoding: gzip, deflate
>> Accept-Language: zh;q=0.9,en;q=0.8
>> Host: wbc-inco.net
>> User-Agent: Mozilla/5.0 (compatible; EasouSpider; +
>> http://www.easou.com/search/spider.html)
>> Content-Length: 927
>> Content-Type: application/x-www-form-urlencoded
>> REFER: http://b------.net/â€œ
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>
>> to understand the logging above: this is what i added /changed in
>> the Catalyst::Plugin::Unicode::Encoding
>>
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>> around line 168:
>>
>> my $val;
>> eval {
>> $val = Encode::is_utf8( $value ) ? $value : $enc->decode(
>> $value, $CHECK );
>> };
>> if ($@){
>> # UPS !
>> # get request infos
>> use Data::Dumper;
>> my $params = $self->req->parameters;
>> my $headers= $self->req->headers->as_string;
>> die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" .
>> Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " .
>> $headers;
>> â€¦.
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>
>> I guess my Catalyst Apps are not the only ones with these errors ?
>>
>>
>> about my App settings / config:
>>
>> app-config has
>> encoding UTF-8
>>
>> App.pm does not load Unicode::Encoding anymore (since this is not need
>> when using latest Catalyst: 5.90065)
>>
>> i am using postgres with
>> pg_enable_utf8 1
>> (but the error about is far away from any DB related problem i guess)
>>
>> using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with
>> catalyxt)
>>
>> i just checked out the tracker for catalyst on cpan, there is an UTF8
>> issue ticket
>> https://rt.cpan.org/Public/Bug/Display.html?id=94957
>> but i does not look as it was this problem ...
>>
>> Any ideas what todo ?
>> Add a issue/ticket ?
>>
>> thanks for feedback,
>> bernhard bauch
>>
>>
>>
>> â€”
>> Bernhard Bauch, Webdevelopment
>> ZSI - Zentrum fÃ¼r soziale Innovation
>> bauch@zsi.at
>> Skype: berni-zsi
>>
>>
>> _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>> !DSPAM:53cd09a3104511692032419!
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> !DSPAM:53cd09a3104511692032419!
>
>
> â€”
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum fÃ¼r soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> !DSPAM:53cd7626104517769513966!
>
>
> â€”
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum fÃ¼r soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> !DSPAM:53ce305e104511469956211!
>
>
> â€”
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum fÃ¼r soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>

Re: UTF8 problems with plugin::encoding [ In reply to ]

winfinit at gmail

Jul 22, 2014, 6:34 AM

Post #9 of 9 (3387 views)

Permalink

Bernhard,

To stop exception, you can modify $CHECK in Catalyst::Plugin::Unicode::Encoding by removing â€œFB_CROAKâ€, that way it wonâ€™t throw exception, and let the code go through, however it will not decode content correctly, but in this case, since it is a spider, i donâ€™t know if it matters to you.

you can just add this to your catalyst application:

use Catalyst::Plugin::Unicode::Encoding;
$Catalyst::Plugin::Unicode::Encoding::CHECK = Encode::LEAVE_SRC;

more isolated test that is illustrating this issue:

use strict;
use warnings;

use Encode qw(decode encode);

our $CHECK = Encode::FB_CROAK | Encode::LEAVE_SRC;

my $str = 'æ·±å…¥ so what';
my $oct = encode("euc-cn", $str);

my $obj = Encode::find_encoding('UTF-8');
my $res = $obj->decode($oct, $CHECK);
warn $res;

-roman

On Jul 22, 2014, at 7:31 AM, Mark Ellis <m@rkellis.com> wrote:

> I don't think there's anything you can do, you're app wants utf8 and they're sending something else which doesn't map. and since you can't know what format it is in, then all you can do is die if it doesn't map, which is what the plugin does.
>
> as far as i can tell the ruby middleware i found handles this by returning a 400 bad request, which cataylst does as well. so there's no affect, other than the noise in the logs.
>
>
> On 22 July 2014 11:21, Bernhard Bauch <bauch@zsi.at> wrote:
> hereâ€™s also a perl-script that does it
>
> ------------------------------------------
> use Encode qw(decode encode);
> use LWP::UserAgent;
>
> my $str = 'æ·±å…¥ so what';
> my $oct = encode("gb2312", $str);
> my $url = 'http://wbc-inco.net/object/event/past';
> my $ua = LWP::UserAgent->new();
> my $response = $ua->post( $url, { $oct => $oct } );
> my $content = $response->decoded_content();
> ------------------------------------------
>
> On 22 Jul 2014, at 11:33, Bernhard Bauch <bauch@zsi.at> wrote:
>
>> hey all,
>>
>> this pyton3 script triggers the error â€¦.
>>
>> --------------------------------
>> import httplib2
>> import urllib.parse
>>
>> somestr = 'æ·±å…¥ so what'
>> encodedstr = somestr.encode('gb2312')
>> url = 'http://myappdomain.com/search'
>> body = { encodedstr:encodedstr }
>> headers = {
>> 'Content-type': 'application/x-www-form-urlencoded',
>> 'Accept': 'text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1',
>> 'Accept-Encoding': 'gzip, deflate',
>> 'Accept-Language': 'zh;q=0.9,en;q=0.8'
>> }
>> http = httplib2.Http()
>> response, content = http.request(url, 'POST', headers=headers, body=urllib.parse.urlencode(body))
>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>
>> now its possible to reproduce the error :)
>>
>> any ideas how to solve this ?
>> ruby people did this with adding a utf8-sanitizer in the middleware..
>>
>> bye, bernhard
>>
>>
>> On 21 Jul 2014, at 22:19, Bernhard Bauch <bauch@zsi.at> wrote:
>>
>>> more news..
>>>
>>> the crawler/searcheinge that triggers these errors is
>>> http://easou.com
>>>
>>> this searchengine delivers their pages not in UTF8 â€” but in â€œgb2312â€ which is â€œsimple chineseâ€
>>> if i open the â€œwrong utf8â€ parameters from the faulty requests with â€œgb2312â€ some readable signs appear.
>>> >> this leads me to: catalyst does not handle requests with gb2312 encoded parameters (because they are not utf8) -and the request does not promote that it is encoded in other than utf8.
>>>
>>> any ideas what to do ?
>>>
>>> bye, bernhard
>>>
>>>
>>>
>>> On 21 Jul 2014, at 14:36, Roman Winfinit <winfinit@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> How are you running your application? Ie: mod_perl, fcgi, fcgi + httpd/nginx, plack + ... also what version of perl are you using and what os?
>>>>
>>>> -roman
>>>>
>>>> On Jul 21, 2014 6:58 AM, "Bernhard Bauch" <bauch@zsi.at> wrote:
>>>> Hey all,
>>>>
>>>> on most of my website running on (latest catalyst: 5.90065) i always get utf8 related errors.
>>>> the usually appear if a spider
>>>> Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
>>>> comes accross.
>>>>
>>>> the error is:
>>>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/â€¦./lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 167.
>>>>
>>>> It took me while to get the actual parameters the spiders sends because the debug-message of catalyst do not tell that much :...
>>>>
>>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/â€¦./lib/perl5/Catalyst.pm +2016: *** Request 164 (0.032/s) [10682] [Wed Jul 16 15:08:47 2014] ***
>>>> [2014/07/16 15:08:47] [5.255.253.218] [DEBUG] vim /usr/local/â€¦./lib/perl5/Catalyst.pm +2309: Response Code: 400; Content-Type: text/plain; charset=UTF-8; Content-Length: unknown
>>>> [2014/07/16 15:08:47] [5.255.253.218] [INFO] vim /usr/local/.../lib/perl5/Catalyst.pm +1880: Request took 0.006491s (154.059/s)
>>>> .---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------.
>>>> | Action | Time |
>>>> +---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+
>>>> '---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------'
>>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>>
>>>> i changed to Plugin::Unicode::Encoding plugin a bit to find out what the client sends â€¦ the results are these:
>>>> UTF8 trash arrives - and the module seems unable to deal with itâ€¦
>>>>
>>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>> Caught exception in engine "UTF8 Error: utf8 "\x98" does not map to Unicode at /usr/local/â€¦../lib/perl5/Catalyst/Plugin/Unicode/Encoding.pm line 170.
>>>> -
>>>>
>>>> URL: notice/list
>>>>
>>>> PARAMS:$VAR1 = {
>>>> 'X*Ã–^K^@^@^@^@Â¸Â®Ã¤
>>>> ^@^@^@^@8<83>^H^K^@^@^@^@hÂ¡Ã¤
>>>> ^@^@^@^@HÂµÃ¤
>>>> ^@^@^@^@X^Z^N^Q^@^@^@^@Ã¸<91>^F^Q^@^@^@^@Ã˜^F^N^Q^@^@^@^@Â¸<92>^F^Q^@^@^@^@(^K^N^Q^@^@^@^@<88>^B^N^Q^@^@^@^@Â¸ÃºÃ^P^@^@^@^@^X%q^G^@^@^@^@Ã˜Â§Ã±^O^@^@^@^@Ã˜Ã¸B.^@^@^@^@Ã¨Ã¢Ã^P^@^@^@^@XÃ›_^L^@^@^@^@ÃˆÃÃ^P^@^@^@^@Â¸~P^S^@^@^@^@Ã¨Ã¥Ã^P^@^@^@^@Ã˜nÃ½^O^@^@^@^@<88>ÃºÃ^P^@^@^@^@^XÃ¡( ^@^@^@^@Ã˜Â¦Ã†
>>>> ^@^@^@^@Ã˜Ã¯*^Q^@^@^@^@^X' => '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[.^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@ <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@ <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@'
>>>> };
>>>>
>>>>
>>>> // value: $VAR1 = '^F^L^@^@^@^@<98>Ã›Ã¸^O^@^@^@^@Ã˜~^A^N^@^@^@^@<98>=H>^@^@^@^@Ã¸<99>Ã³^K^@^@^@^@hÃ”u^R^@^@^@^@Â¸<8e>Ã³^K^@^@^@^@^XÃ¤_^L^@^@^@^@Ã˜<90>a^G^@^@^@^@hÃ°Ã‰^O^@^@^@^@8Ã£*^G^@^@^@^@Ã˜Ã˜Ã½^M^@^@^@^@XÃ¹Ã«^F^@^@^@^@^HÃœÃ½^M^@^@^@^@8W6^H^@^@^@^@Ã¸ÃÃ½^M^@^@^@^@xÃ¿Ãƒ^K^@^@^@^@X]i^O^@^@^@^@8^MÃ¿^H^@^@^@^@XÃ»<98>^Q^@^@^@^@xÂ¦h^H^@^@^@^@XÃ½<98>^Q^@^@^@^@^X=5^H^@^@^@^@^XÂ¦Ãº^K^@^@^@^@^XVQ^P^@^@^@^@^H^YÃ»^N^@^@^@^@xÂ¤h^H^@^@^@^@^XÃ¥<98>^Q^@^@^@^@Ã¸Â¤h^H^@^@^@^@XÃ©<98>^Q^@^@^@^@XÂ¼h^H^@^@^@^@Ã˜Â¡h^H^@^@^@^@Ã¸f<82>^Q^@^@^@^@^X>Ã©H^@^@^@^@xv<82>^Q^@^@^@^@X6Ã©H^@^@^@^@xl<82>^Q^@^@^@^@83ÃŒ^G^@^@^@^@Xl<82>^Q^@^@^@^@Â¸Ã‘Ã½^M^@^@^@^@xr<82>^Q^@^@^@^@H[^H^Q^@^@^@^@^X|<82>^Q^@^@^@^@Â¸Ã‹Â¢^K^@^@^@^@Â¸u<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Ã˜p<82>^Q^@^@^@^@8ÃÂ¢^K^@^@^@^@Ã˜l<82>^Q^@^@^@^@XÃ‹Â¢^K^@^@^@^@Xq<82>^Q^@^@^@^@^Xi^W^H^@^@^@^@Xc<82>^Q^@^@^@^@Â¸Ã…Â¢^K^@^@^@^@8h<82>^Q^@^@^@^@<98>ÃÂ¢^K^@^@^@^@Â¨fÃ^Q^@^@^@^@Ã˜Ã‰=^R^@^@^@^@Ã€C<95>^M^@^@^@^@Â°S<95>^M^@^@^@^@^PI<95>^M^@^@^@^@Ã€\\<95>^M^@^@^@^@Ã°E<95>^M^@^@^@^@<80>B<95>^M^@^@^@^@@P<95>^M^@^@^@^@<80>Q<95>^M^@^@^@^@ J<95>^M^@^@^@^@p\\<95>^M^@^@^@^@Ã U<95>^M^@^@^@^@Ã F<95>^M^@^@^@^@Ã A<95>^M^@^@^@^@^@<9e>Ã´^P^@^@^@^@Â°<9d>Ã´^P^@^@^@^@0<91>Ã´^P^@^@^@^@ <9e>Ã´^P^@^@^@^@^P<8e>Ã´^P^@^@^@^@ <88>Ã´^P^@^@^@^@Ã<82>Ã´^P^@^@^@^@ <8d>Ã´^P^@^@^@^@<90><95>Ã´^P^@^@^@^@Ã <90>Ã´^P^@^@^@^@@<95>Ã´^P^@^@^@^@P<8f>Ã´^P^@^@^@^@<90><81>Ã´^P^@^@^@^@ <97>Ã´^P^@^@^@^@Ã<8c>Ã´^P^@^@^@^@p<88>Ã´^P^@^@^@^@P<99>Ã´^P^@^@^@^@<90><90>Ã´^P^@^@^@^@@<9a>Ã´^P^@^@^@^@0<9b>Ã´^P^@^@^@';
>>>>
>>>>
>>>> headers: Connection: close
>>>> Accept: text/html, application/xml;q=0.9, application/xhtml+xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
>>>> Accept-Encoding: gzip, deflate
>>>> Accept-Language: zh;q=0.9,en;q=0.8
>>>> Host: wbc-inco.net
>>>> User-Agent: Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)
>>>> Content-Length: 927
>>>> Content-Type: application/x-www-form-urlencoded
>>>> REFER: http://b------.net/â€œ
>>>>
>>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>>
>>>> to understand the logging above: this is what i added /changed in the Catalyst::Plugin::Unicode::Encoding
>>>>
>>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>> around line 168:
>>>>
>>>> my $val;
>>>> eval {
>>>> $val = Encode::is_utf8( $value ) ? $value : $enc->decode( $value, $CHECK );
>>>> };
>>>> if ($@){
>>>> # UPS !
>>>> # get request infos
>>>> use Data::Dumper;
>>>> my $params = $self->req->parameters;
>>>> my $headers= $self->req->headers->as_string;
>>>> die "UTF8 Error: $@ - \n\nURL: " . $self->req->path . "\n\nPARAMS:" . Dumper( $params ) . "\n\n // value: " . Dumper($value) . "\n\nheaders: " . $headers;
>>>> â€¦.
>>>> â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”â€”
>>>>
>>>> I guess my Catalyst Apps are not the only ones with these errors ?
>>>>
>>>>
>>>> about my App settings / config:
>>>>
>>>> app-config has
>>>> encoding UTF-8
>>>>
>>>> App.pm does not load Unicode::Encoding anymore (since this is not need when using latest Catalyst: 5.90065)
>>>>
>>>> i am using postgres with
>>>> pg_enable_utf8 1
>>>> (but the error about is far away from any DB related problem i guess)
>>>>
>>>> using Catalyst::Plugin::Unicode::Encoding version 2.1 (coming with catalyxt)
>>>>
>>>> i just checked out the tracker for catalyst on cpan, there is an UTF8 issue ticket
>>>> https://rt.cpan.org/Public/Bug/Display.html?id=94957
>>>> but i does not look as it was this problem ...
>>>>
>>>> Any ideas what todo ?
>>>> Add a issue/ticket ?
>>>>
>>>> thanks for feedback,
>>>> bernhard bauch
>>>>
>>>>
>>>>
>>>> â€”
>>>> Bernhard Bauch, Webdevelopment
>>>> ZSI - Zentrum fÃ¼r soziale Innovation
>>>> bauch@zsi.at
>>>> Skype: berni-zsi
>>>>
>>>>
>>>> _______________________________________________
>>>> List: Catalyst@lists.scsys.co.uk
>>>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>>>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>>>> Dev site: http://dev.catalyst.perl.org/
>>>>
>>>> !DSPAM:53cd09a3104511692032419! _______________________________________________
>>>> List: Catalyst@lists.scsys.co.uk
>>>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>>>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>>>> Dev site: http://dev.catalyst.perl.org/
>>>>
>>>>
>>>> !DSPAM:53cd09a3104511692032419!
>>>
>>> â€”
>>> Bernhard Bauch, Webdevelopment
>>> ZSI - Zentrum fÃ¼r soziale Innovation
>>> bauch@zsi.at
>>> Skype: berni-zsi
>>>
>>> _______________________________________________
>>> List: Catalyst@lists.scsys.co.uk
>>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>>> Dev site: http://dev.catalyst.perl.org/
>>>
>>>
>>> !DSPAM:53cd7626104517769513966!
>>
>> â€”
>> Bernhard Bauch, Webdevelopment
>> ZSI - Zentrum fÃ¼r soziale Innovation
>> bauch@zsi.at
>> Skype: berni-zsi
>>
>> _______________________________________________
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
>> !DSPAM:53ce305e104511469956211!
>
>
> â€”
> Bernhard Bauch, Webdevelopment
> ZSI - Zentrum fÃ¼r soziale Innovation
> bauch@zsi.at
> Skype: berni-zsi
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
> _______________________________________________
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/

Mailing List Archive

Attached Files:

Attached Files:

Attached Files:

Attached Files:

Attached Files: