Mailing List Archive

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Changes (by cboos):

* severity: normal => major
* priority: normal => high
* milestone: => 0.9.4

Comment:

See also #2472, where this error was also reported and discussed.
As that other ticket was about another issue with the 1.3.0 bindings,
it has been closed and this ticket should be used to focus on the
`'apr_pool_t *' is expected` issue.

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Changes (by anonymous):

* cc: manuzhai@gmail.com => manuzhai@gmail.com,odela01@ca.com

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Changes (by pjenvey@groovie.org):

* cc: manuzhai@gmail.com,odela01@ca.com =>
manuzhai@gmail.com,odela01@ca.com,pjenvey@groovie.org

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Changes (by anonymous):

* cc: manuzhai@gmail.com,odela01@ca.com,pjenvey@groovie.org =>
manuzhai@gmail.com,odela01@ca.com,pjenvey@groovie.org,
shishz@hotpop.com

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by Manuzhai):

This error seems to appear quite often on editing a wiki page. Refreshing
the edit (resending POST data) gives the same error again, going back to
the wiki page, then sending it again too, hard-refreshing the page then
redoing the changes and submitting them still seems to work. Not sure
whether this is always true...

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by cboos):

I've not been able to investigate and reproduce this myself,
but could you try the following?
{{{
Index: svn_fs.py
===================================================================
--- svn_fs.py (revision 2819)
+++ svn_fs.py (working copy)
@@ -28,7 +28,7 @@
from trac.versioncontrol.svn_authz import SubversionAuthorizer

try:
- from svn import fs, repos, core, delta
+ from svn import core, fs, repos, delta
has_subversion = True
except ImportError:
has_subversion = False
}}}

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by Manuzhai):

I've made the change, and will report back to see if it helps.

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Changes (by anonymous):

* cc: manuzhai@gmail.com,odela01@ca.com,pjenvey@groovie.org,
shishz@hotpop.com =>
manuzhai@gmail.com,odela01@ca.com,pjenvey@groovie.org,
shishz@hotpop.com, james82@gmail.com

Comment:

I installed mod_python 3.1.4 / Python 2.3 / Subversion 1.3.0 on a single
processor RHEL4 machine and I can't reproduce this issue. Ideas?

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by pjenvey@groovie.org):

I am seeing this problem with python 2.4. The original bug reported to the
subversion-users list also occured under python 2.4

Myabe the fact that you're using python 2.3 is a factor, or possibly some
other dependency version mismatch?

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by Manuzhai):

Yes, I'm also using Python 2.4. The change of import order doesn't solve
all of the problem, though it does seem that it doesn't happen as often
now. That might have to do with my usage pattern, though...

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by cboos):

Can you try to split the `import` in several lines, i.e.
{{{
try:
from svn import core
from svn import fs
# etc.
}}}

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cmlenz
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by Manuzhai):

Done...

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Changes (by cboos):

* owner: cmlenz => cboos

Comment:

I think I understand the issue: having all the imports
in one line allows two threads (A and B) to start
simultaneously the import process:
* A: trac/versioncontrol/svn_fs.py: `from svn import fs, repos, core,
delta`
* B: trac/versioncontrol/svn_fs.py: `from svn import fs, repos, core,
delta`
* A: in svn/fs.py, line 19: `from libsvn.core import *`,
* A: now in libsvn/core.py, is going through the > 3000 lines...
* B: in svn/fs.py, line 19: `from libsvn.core import *`,
* B: now in libsvn/core.py, is going through the > 3000 lines...
* A: in libsvn/core.py, line 3049: does a `_core.svn_pool_create` [[BR]]
and here, `_core` is ''really'' `_core` (i.e. the native extension)
* A: in svn/fs.py, line 29: `import svn.core as _core`
* B: in libsvn/core.py, line 3049: does a `_core.svn_pool_create` [[BR]]
and now, `_core` is '''not''' `_core` but `svn.core` and therefore,
`_core.svn_pool_create` is not the native function but the (deprecated)
compatibility function, which expects a pool as first argument...
hence the error.

Hopefully splitting the imports does solve the issue,
but the bindings should really be more robust about that
and ''not'' rename different things to a same name...

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Changes (by mark@openmailadmin.org):

* cc: manuzhai@gmail.com,odela01@ca.com,pjenvey@groovie.org,
shishz@hotpop.com, james82@gmail.com =>
manuzhai@gmail.com,odela01@ca.com,pjenvey@groovie.org,
shishz@hotpop.com, james82@gmail.com,
mark@openmailadmin.org

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by james82@gmail.com):

I have not been able to reproduce the problem (even with Python 2.4 and
mod_python 3.1.3), but, I think I have figured out how to fix it. Please
try the below patch (for Subversion 1.3.0).

{{{
#!diff
Index: subversion/bindings/swig/include/svn_types.swg
===================================================================
--- subversion/bindings/swig/include/svn_types.swg (revision 18239)
+++ subversion/bindings/swig/include/svn_types.swg (working copy)
@@ -701,11 +701,17 @@

%exception {
#ifdef SWIGPYTHON
- svn_swig_py_release_py_lock();
+ int released_lock = 0;
+ if (_global_svn_swig_py_pool != NULL) {
+ released_lock = 1;
+ svn_swig_py_release_py_lock();
+ }
#endif
$action
#ifdef SWIGPYTHON
- svn_swig_py_acquire_py_lock();
+ if (released_lock) {
+ svn_swig_py_acquire_py_lock();
+ }
#endif
}

}}}

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by james82@gmail.com):

'''Important note:'''
In order for the above patch to take effect, you'll need to install SWIG
1.3.25, and run the following commands:
{{{
make extraclean
./autogen.sh
./configure
make swig-py
make install-swig-py
}}}
Note that you need to `make extraclean` to remove any old headers shipped
with the tarball.

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by pjenvey@groovie.org):

Manuzhai -

Did splitting the import into multiple lines solve the problem?

If not, I'll see if I can give james' patch a shot later today

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by Manuzhai):

Splitting the import didn't solve the problem.

James' fix is a little involved for me, I just install SVN & bindings
through emerge, I'm afraid. I could try it, but I'm not sure when I get to
it.

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by pjenvey@groovie.org):

james - I upgraded to swig 1.3.27, applied your patch and did the whole
make extraclean/autogen. Now I am noticing a different, similar error:

{{{
Traceback (most recent call last):
File "/usr/local/lib/python2.4/site-
packages/trac/web/modpython_frontend.py", line 206, in handler
dispatch_request(mpr.path_info, mpr, env)
File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line
139, in dispatch_request
dispatcher.dispatch(req)
File "/usr/local/lib/python2.4/site-packages/trac/web/main.py", line
107, in dispatch
resp = chosen_handler.process_request(req)
File "/usr/local/lib/python2.4/site-
packages/trac/versioncontrol/web_ui/browser.py", line 91, in
process_request
repos = self.env.get_repository(req.authname)
File "/usr/local/lib/python2.4/site-packages/trac/env.py", line 157, in
get_repository
authname)
File "/usr/local/lib/python2.4/site-
packages/trac/versioncontrol/api.py", line 61, in get_repository
return self._connector.get_repository(repos_type, repos_dir, authname)
File "/usr/local/lib/python2.4/site-
packages/trac/versioncontrol/svn_fs.py", line 205, in get_repository
repos = SubversionRepository(dir, authz, self.log)
File "/usr/local/lib/python2.4/site-
packages/trac/versioncontrol/svn_fs.py", line 224, in __init__
self.path = repos.svn_repos_find_root_path(path, self.pool())
File "/usr/local/lib/python2.4/site-packages/libsvn/repos.py", line 43,
in svn_repos_find_root_path
return apply(_repos.svn_repos_find_root_path, args)
TypeError: argument number 2: a 'apr_pool_t *' is expected,
'instance(<libsvn.core.GenericSWIGWrapper instance at 0x8b88f6c>)' is
received
}}}

Instead of seeing the stack trace in my error log (where I saw the
original apr_pool_t exception), it shows up on an actual trac "Oops...
Trac detected an internal error:" error page. I can't tell if this is due
to your patch, the swig upgrade, or both

This new error is intermittent like the old one. Occasionally I receive
either 404 responses or blank pages instead of the error message. Nothing
in the error_log

Also: I got the impression the make extraclean/reautogen would end up
creating a brand new subversion/includes/svn_types.h file corresponding to
your change to svn_types.swg. That was not the case -- the patch was
successfully applied, but I ended up with the same svn_types.h file. Let
me know if that is actually a problem

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by David James <james82@gmail.com>):

Hi pjenvey,

It looks like you applied the patch correctly, but my patch did not help.
I am not surprised by your new error message, because it is the same as
the one that was reported earlier on the users list.

Could you send the full details for your system, and the full details of
your apache configuration including your httpd.conf and a description of
which "mpm" you are using (e.g. the worker mpm, or the prefork mpm). I
would like to try to reproduce the problem by copying your configuration.
Are you using Linux, Mac OS, or something else?

Here's a question for the group: In mod_python, does the "global
interpreter lock" lock just the current interpreter or does it lock all
interpreters? If the "global interpreter lock" only locks the current
interpreter, then it explains our problem. We will need to use a different
locking system in order to prevent race conditions during pool
initialization. Perhaps an APR lock would do the trick?

Cheers,

David

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by Manuzhai):

From http://docs.python.org/api/threads.html (and my understanding), the
GIL just locks all but one threads '''within''' one interpreter. So if
multiple interpreters are at work, we need a different locking mechanism
(in which case the APR lock would seem like the best solution).

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by David James <james82@gmail.com>):

Hi pjenvey, Manzuhai:

After thinking about this problem some more, I have written a patch which
uses Python locking to solve the race condition in application pool
initialization. Could you give this patch a try?


{{{
#!diff
Index: subversion/bindings/swig/include/proxy_apr.swg
===================================================================
--- subversion/bindings/swig/include/proxy_apr.swg (revision 18283)
+++ subversion/bindings/swig/include/proxy_apr.swg (working copy)
@@ -60,7 +60,10 @@
*/

%pythoncode %{
+import threading
+
application_pool = None
+application_pool_lock = threading.Lock()
class GenericSWIGWrapper:
def __init__(self, this, pool):
"""Create new Generic SWIG wrapper object"""
@@ -98,21 +101,26 @@
def set_parent_pool(self, parent_pool=None):
"""Create a new memory pool"""
global application_pool
+
+ try:
+ application_pool_lock.acquire()
+
+ self._parent_pool = parent_pool or application_pool
+ self._mark_valid()
+
+ # Protect important functions from GC
+ self._apr_pool_destroy = _core.apr_pool_destroy
+ self._svn_swig_py_clear_application_pool = \
+ _core.svn_swig_py_clear_application_pool
+
+ # If we are an application-level pool,
+ # then set this pool to be the application-level pool
+ if not self._parent_pool:
+ svn_swig_py_set_application_pool(self, self)
+ application_pool = self
+ finally:
+ application_pool_lock.release()

- self._parent_pool = parent_pool or application_pool
- self._mark_valid()
-
- # Protect important functions from GC
- self._apr_pool_destroy = _core.apr_pool_destroy
- self._svn_swig_py_clear_application_pool = \
- _core.svn_swig_py_clear_application_pool
-
- # If we are an application-level pool,
- # then set this pool to be the application-level pool
- if not self._parent_pool:
- svn_swig_py_set_application_pool(self, self)
- application_pool = self
-
def valid(self):
"""Check whether this memory pool and its parents
are still valid"""
}}}

'''Important note:''' Once again, in order for this patch to take effect,
you'll need to type:
{{{
make extraclean
./autogen.sh
./configure
make swig-py
make install-swig-py
}}}

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by Manuzhai):

Could you attach this patch to the ticket insteading of showing it inline?
I'm having a hard time getting it applied correctly. Thanks!

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>

#2611: Problem with SVN bindings (SVN 1.3.0, Trac r2771)
-----------------------------+----------------------------------------------
Reporter: Manuzhai | Owner: cboos
Type: defect | Status: new
Priority: high | Milestone: 0.9.4
Component: version control | Version: devel
Severity: major | Resolution:
Keywords: |
-----------------------------+----------------------------------------------
Comment (by David James <james82@gmail.com>):

Hi Manzuhai,

The patch is attached. I've also posted it to the Subversion developers
list, at http://svn.haxx.se/dev/archive-2006-01/0946.shtml

Cheers,

David

--
Ticket URL: <http://projects.edgewall.com/trac/ticket/2611>
The Trac Project <http://trac.edgewall.com/>