Advanced
Mailing List Archive

Mailing List Archive

  1. Home >
  2. Trac>
  3. Tickets
[The Trac Project] #3723: Problems with database encoding
noreply at edgewall
Sep 14, 2006, 2:35 AM
Post #1 of 13 (5014 views)
Permalink
#3723: Problems with database encoding
--------------------+-------------------------------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: high | Milestone:
Component: wiki | Version: 0.10b1
Severity: major | Keywords:
--------------------+-------------------------------------------------------
I am using Trac v.0.10-b1 with MySQL and I get the following error:


{{{
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 335, in
dispatch_request
dispatcher.dispatch(req)
File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 220, in
dispatch
resp = chosen_handler.process_request(req)
File "/usr/lib/python2.4/site-packages/trac/wiki/web_ui.py", line 97, in
process_request
page = WikiPage(self.env, pagename, version, db)
File "/usr/lib/python2.4/site-packages/trac/wiki/model.py", line 32, in
__init__
self._fetch(name, version, db)
File "/usr/lib/python2.4/site-packages/trac/wiki/model.py", line 53, in
_fetch
(name,))
File "/usr/lib/python2.4/site-packages/trac/db/util.py", line 47, in
execute
return self.cursor.execute(sql_escape_percent(sql), args)
File "/usr/lib/python2.4/site-packages/trac/db/util.py", line 47, in
execute
return self.cursor.execute(sql_escape_percent(sql), args)
File "/usr/lib/python2.4/site-packages/MySQLdb/cursors.py", line 163, in
execute
self.errorhandler(self, exc, value)
File "/usr/lib/python2.4/site-packages/MySQLdb/connections.py", line 35,
in defaulterrorhandler
raise errorclass, errorvalue
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 919-922:
invalid data
}}}

Any help?

--
Ticket URL: <http://trac.edgewall.org/ticket/3723>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Sep 20, 2006, 4:06 AM
Post #2 of 13 (4906 views)
Permalink
#3723: Problems with database encoding
----------------------------+-----------------------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone:
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: trac wiki utf8 |
----------------------------+-----------------------------------------------
Changes (by anonymous):

* keywords: => trac wiki utf8
* priority: high => highest
* severity: major => blocker

Comment:

I'm trying hard to fix this but to no success. Can anyone help as this bug
renders the whole wiki unusable?? Thanks in advance...

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:1>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Sep 24, 2006, 10:59 AM
Post #3 of 13 (4911 views)
Permalink
#3723: Problems with database encoding
----------------------------+-----------------------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone:
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: trac wiki utf8 |
----------------------------+-----------------------------------------------
Comment (by movex@bayern-mail.de):

Hi,
I'm using also Trac 0.10b1 with Python 2.3.5 and Mysql 4.1.11 and I get
the same error.
Mysql database wokrs in utf-8 constellation. When restarting apache the
error is gone for some hours after that same thing again.
This also affects timeline and tickets.

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:2>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Sep 25, 2006, 6:30 AM
Post #4 of 13 (4890 views)
Permalink
#3723: Problems with database encoding
----------------------------------+-----------------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 |
----------------------------------+-----------------------------------------
Changes (by cboos):

* keywords: trac wiki utf8 => mysql trac wiki utf8
* milestone: => 0.10.1

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:3>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 3, 2006, 9:47 AM
Post #5 of 13 (4910 views)
Permalink
#3723: Problems with database encoding
-----------------------------------------+----------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite |
-----------------------------------------+----------------------------------
Changes (by anonymous):

* keywords: mysql trac wiki utf8 => mysql trac wiki utf8 sqlite

Comment:

Seeing the same sort of problem since upgrade to 0.10 with sqlite 3.2.1 &
python 2.4. It only seems to affect searches on tickets, not on the wiki.

{{{
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 356, in
dispatch_request
dispatcher.dispatch(req)
File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 224, in
dispatch
resp = chosen_handler.process_request(req)
File "/usr/lib/python2.4/site-packages/trac/Search.py", line 181, in
process_request
results += list(source.get_search_results(req, terms, filters))
File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 265, in
get_search_results
for summary, desc, author, keywords, tid, date, status in cursor:
File "/usr/lib/python2.4/site-packages/trac/db/util.py", line 40, in
__iter__
row = self.cursor.fetchone()
File "/usr/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line
73, in fetchone
return row and self._convert_row(row) or None
File "/usr/lib/python2.4/site-packages/trac/db/sqlite_backend.py", line
69, in _convert_row
return tuple([(isinstance(v, str) and [v.decode('utf-8')] or [v])[0]
File "/usr/lib/python2.4/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1784-1785:
invalid data
}}}

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:4>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 3, 2006, 10:00 AM
Post #6 of 13 (4890 views)
Permalink
#3723: Problems with database encoding
-----------------------------------------+----------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite |
-----------------------------------------+----------------------------------
Comment (by cboos):

Replying to [comment:4 anonymous]:
> Seeing the same sort of problem since upgrade to 0.10 with sqlite 3.2.1
& python 2.4. It only seems to affect searches on tickets, not on the
wiki.
>
> {{{ return tuple([(isinstance(v, str) and [v.decode('utf-8')] or
[v])[0] }}}

Looks like you're using pysqlite 1.x. You could (should) upgrade to 2.3.2.
Check the PySqlite page.

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:5>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 4, 2006, 1:22 AM
Post #7 of 13 (4874 views)
Permalink
#3723: Problems with database encoding
-----------------------------------------+----------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite |
-----------------------------------------+----------------------------------
Comment (by anonymous):

Hi,

pysqlite 2.3.2 caused apache to segfault, so I tried version 2.0.7, which
worked ok, but I still get an error (although a slightly different one)
{{{
Traceback (most recent call last):
File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 356, in
dispatch_request
dispatcher.dispatch(req)
File "/usr/lib/python2.4/site-packages/trac/web/main.py", line 224, in
dispatch
resp = chosen_handler.process_request(req)
File "/usr/lib/python2.4/site-packages/trac/Search.py", line 181, in
process_request
results += list(source.get_search_results(req, terms, filters))
File "/usr/lib/python2.4/site-packages/trac/ticket/api.py", line 267, in
get_search_results
if status == 'closed':
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 1784-1785:
invalid data
}}}

Again this only happens on ticket searches.

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:6>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 4, 2006, 2:12 AM
Post #8 of 13 (4893 views)
Permalink
#3723: Problems with database encoding
-----------------------------------------+----------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite |
-----------------------------------------+----------------------------------
Comment (by cboos):

Replying to [comment:6 anonymous]:
> Hi,
>
> pysqlite 2.3.2 caused apache to segfault,

Probably the same issue as pysqlite:ticket:174 (fixed in pysqlite's
trunk).

>so I tried version 2.0.7, which worked ok, but I still get an error
(although a slightly different one)
> ... [[br]]
> ` if status == 'closed':` [[br]]
> `UnicodeDecodeError: 'utf8' codec can't decode bytes in position
1784-1785: invalid data` [[br]]
> ...

I can't see why the `status` would have such a length ... can you insert a
`print tid, repr(status)` just before line 267?

--
Ticket URL: <http://trac.edgewall.com/ticket/3723#comment:7>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 4, 2006, 5:44 AM
Post #9 of 13 (4879 views)
Permalink
#3723: Problems with database encoding
-----------------------------------------+----------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite |
-----------------------------------------+----------------------------------
Comment (by frederic.duarte@oneacces-net.com):

Hi gentlemens,

Same problem as Szymon (except the bytes position : UnicodeDecodeError:
'utf8' codec can't decode bytes in position 2-4: invalid data) with trac
0.10 (tag), apache 2.0.52, mysql 4.1.20 in using UTF8, Python 2.3.4.

As Movex said, the trange thing is that it disappear for some time when
restarting apache. And when I come back to work the day after, it is
broken again.

I didnt find any clue in the apache logs :(

Tell me if/how I could help.

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:8>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 5, 2006, 3:41 AM
Post #10 of 13 (4892 views)
Permalink
#3723: Problems with database encoding
-----------------------------------------+----------------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite |
-----------------------------------------+----------------------------------
Comment (by anonymous):

Hopefully of some help...

I tried writing the values out to a file. If I do f.write(tid) I get the
same Unicode error for the write. However if I do f.write(repr(tid)) the
write works fine and just prints a four digit ticket number. I'm learning
python as I go on this one so I'm not sure if that's what you'd expect or
not :o)

jon at the hyphen mill dot com

Replying to [comment:7 cboos]:
> Replying to [comment:6 anonymous]:
> > Hi,
> >
> > pysqlite 2.3.2 caused apache to segfault,
>
> Probably the same issue as pysqlite:ticket:174 (fixed in pysqlite's
trunk).
>
> >so I tried version 2.0.7, which worked ok, but I still get an error
(although a slightly different one)
> > ... [[br]]
> > ` if status == 'closed':` [[br]]
> > `UnicodeDecodeError: 'utf8' codec can't decode bytes in position
1784-1785: invalid data` [[br]]
> > ...
>
> I can't see why the `status` would have such a length ... can you insert
a `print tid, repr(status)` just before line 267?

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:9>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 5, 2006, 11:13 PM
Post #11 of 13 (4887 views)
Permalink
#3723: Problems with database encoding
-------------------------------------------------+--------------------------
Reporter: szymon | Owner: jonas
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: wiki | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite unicode |
-------------------------------------------------+--------------------------
Changes (by cboos):

* keywords: mysql trac wiki utf8 sqlite => mysql trac wiki utf8 sqlite
unicode

Comment:

Replying to [comment:9 anonymous]:
> I tried writing the values out to a file. If I do `f.write(tid)` I get
the same Unicode error for the write. However if I do `f.write(repr(tid))`
the write works fine and just prints a four digit ticket number.

That's very surprising...
Please try: `f.write(tid.encode('unicode_internal'))`, and also:
{{{
from locale import getlocale
f.write(repr(getlocale()))
}}}

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:10>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 6, 2006, 1:08 AM
Post #12 of 13 (4875 views)
Permalink
#3723: Problems with database encoding
-------------------------------------------------+--------------------------
Reporter: szymon | Owner: cmlenz
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: mod_python frontend | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite unicode |
-------------------------------------------------+--------------------------
Changes (by frederic.duarte):

* owner: jonas => cmlenz
* component: wiki => mod_python frontend

Comment:

I suggest to change the component as the error occurs on every page of any
trac project. Please correct if I'm wrong.

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:11>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---
Re: [The Trac Project] #3723: Problems with database encoding [ In reply to ]
noreply at edgewall
Oct 6, 2006, 2:53 AM
Post #13 of 13 (4889 views)
Permalink
#3723: Problems with database encoding
-------------------------------------------------+--------------------------
Reporter: szymon | Owner: cmlenz
Type: defect | Status: new
Priority: highest | Milestone: 0.10.1
Component: mod_python frontend | Version: 0.10b1
Severity: blocker | Resolution:
Keywords: mysql trac wiki utf8 sqlite unicode |
-------------------------------------------------+--------------------------
Comment (by anonymous):

Ok, hopefully some progress! :o)

f.write(tid.encode('unicode_internal')) gives the error: 'int' object has
no attribute 'encode'.

However, if I just put the
{{{
from locale import getlocale
}}}
line in above line 267 and nothing else, the problem goes away and the
search displays correctly. If I put the import line at the top of api.py I
still get the error (not sure if that's expected or not)

If I write out the locale I get: ('en_GB', 'ISO8859-1')

Replying to [comment:10 cboos]:

> Please try: `f.write(tid.encode('unicode_internal'))`, and also:
> {{{
> from locale import getlocale
> f.write(repr(getlocale()))
> }}}

--
Ticket URL: <http://trac.edgewall.org/ticket/3723#comment:12>
The Trac Project <http://trac.edgewall.org/>

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "Trac Tickets" group.
To post to this group, send email to trac-tickets@googlegroups.com
To unsubscribe from this group, send email to trac-tickets-unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.se/group/trac-tickets
-~----------~----~----~----~------~----~------~--~---

©2024 GT.net. A Gossamer Threads company.
5th Floor, 455 Granville St., Vancouver, BC V6C 1T1, Canada | Legal