Mailing List Archive

Zope database connectivity
Greetings. I'm having a problem with database connectors, and maybe you can
give me a hand.

Is there a way to manage the opening and closing of database connectors,
apart from doing it from the zope management interface?
By this I mean, I want to implement a Python script that checks wether a
database connection is open or closed. If it's closed, I'll open it, all
coded, without having to click the "Open connection" button.

I haven't found info on this. Can you help me out with this?
Thank you very much in advance.
Re: Zope database connectivity [ In reply to ]
Basically, no.

You could recode DA's to close the connection at the end of every
transaction and reopen them on demand, but to the best of my
knowledge none of the DAs work that way.

Once a connection object opens its connection it stays open but in
the object cache of that thread. This gives it both an indeterminate
lifespan and a fair amount of isolation from other threads -- sadly,
the thread that you're running your ZMI transaction on. The ZMI cant
"see" what objects the other threads are using (well you actually CAN
program that but -- dont! -- you'd most likely break Zope in a severe
way without intending to)

Most of the open buttons on database connections are pretty useless
-- they just make the connection open early. Most DA's open a
connection on demand. For "Most" read "all" because I don't know of
one that DOESNT work that way.

On Jul 13, 2006, at 4:41 AM, Jose Carlos Balderas Alberico wrote:

> Greetings. I'm having a problem with database connectors, and maybe
> you can give me a hand.
>
> Is there a way to manage the opening and closing of database
> connectors, apart from doing it from the zope management interface?
> By this I mean, I want to implement a Python script that checks
> wether a database connection is open or closed. If it's closed,
> I'll open it, all coded, without having to click the "Open
> connection" button.
>
> I haven't found info on this. Can you help me out with this?
> Thank you very much in advance.
> _______________________________________________
> Zope-DB mailing list
> Zope-DB@zope.org
> http://mail.zope.org/mailman/listinfo/zope-db

_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Matthew T. Kromer wrote:
> Basically, no.
>
> You could recode DA's to close the connection at the end of every
> transaction and reopen them on demand, but to the best of my knowledge
> none of the DAs work that way.
>
> Once a connection object opens its connection it stays open but in the
> object cache of that thread. This gives it both an indeterminate
> lifespan and a fair amount of isolation from other threads -- sadly, the
> thread that you're running your ZMI transaction on. The ZMI cant "see"
> what objects the other threads are using (well you actually CAN program
> that but -- dont! -- you'd most likely break Zope in a severe way
> without intending to)
>
> Most of the open buttons on database connections are pretty useless --
> they just make the connection open early. Most DA's open a connection
> on demand. For "Most" read "all" because I don't know of one that
> DOESNT work that way.

I'll correct that slightly: Zope DAs usually open the connection
as soon as the connection object is loaded into memory, e.g.
if you look at the folder contents containing the connection
object.

This is not the same as opening the connection on demand,
since demand for a connection will only occur at the time
you actually ask the connection object to execute a query
by calling it.

If a connection object only opens the connection if you actually
run a query, that's "connect on demand" and the way we've
implemented this feature in the mxODBC Zope DA.

The "connect on load" which most other DAs implement can
cause problems in some situations e.g. if the
connection to the database is down or not available
for some reason, but you still want to edit the connection
object via the ZMI.

"Connect on demand" also tends to save resources and plays
well with connections that time out after a while.

> On Jul 13, 2006, at 4:41 AM, Jose Carlos Balderas Alberico wrote:
>
>> Greetings. I'm having a problem with database connectors, and maybe
>> you can give me a hand.
>>
>> Is there a way to manage the opening and closing of database
>> connectors, apart from doing it from the zope management interface?
>> By this I mean, I want to implement a Python script that checks wether
>> a database connection is open or closed. If it's closed, I'll open it,
>> all coded, without having to click the "Open connection" button.
>>
>> I haven't found info on this. Can you help me out with this?

The DAs will usually have a method which implements this. The
name of the method is DA-dependent, e.g. for the mxODBC Zope DA
it's called .connect() or .lazy_connect() if you want to open
the connection object in "connect on demand" mode.

--
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source (#1, Sep 20 2006)
>>> Python/Zope Consulting and Support ... http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/
________________________________________________________________________

::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! ::::
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
M.-A. Lemburg wrote at 2006-9-20 23:00 +0200:
> ...
>I'll correct that slightly: Zope DAs usually open the connection
>as soon as the connection object is loaded into memory, e.g.
>if you look at the folder contents containing the connection
>object.

In my view, this is a bug in Zope's DA code.

However, it really is now as you describe it...



--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Dieter Maurer wrote:
> M.-A. Lemburg wrote at 2006-9-20 23:00 +0200:
>> ...
>> I'll correct that slightly: Zope DAs usually open the connection
>> as soon as the connection object is loaded into memory, e.g.
>> if you look at the folder contents containing the connection
>> object.
>
> In my view, this is a bug in Zope's DA code.
>
> However, it really is now as you describe it...

Well, you should be a foundation member by now, right?

Put in a proposal and fix it :-D

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Chris Withers wrote at 2006-9-22 10:48 +0100:
>Dieter Maurer wrote:
>> M.-A. Lemburg wrote at 2006-9-20 23:00 +0200:
>>> ...
>>> I'll correct that slightly: Zope DAs usually open the connection
>>> as soon as the connection object is loaded into memory, e.g.
>>> if you look at the folder contents containing the connection
>>> object.
>>
>> In my view, this is a bug in Zope's DA code.
>>
>> However, it really is now as you describe it...
>
>Well, you should be a foundation member by now, right?

Yes, but my SVN access is not yet working (probably due to a '.' in
my Zope user account).



--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
> You could recode DA's to close the connection at the end of every
> transaction and reopen them on demand,
Seems for me that difference betwen "connect on load"
and "connect on demand" is in _begin, _finish and
__init__ methods of TransactionManager (usually defined in db.py
in specific database adapter).

> but to the best of my knowledge
> none of the DAs work that way.
ZSQLRelayDA opens connections in _begin method and
closes them in _finish method of it's TransactionManager.
It seems to be "on demand" scenario for me.

> The "connect on load" which most other DAs implement can
> cause problems in some situations e.g. if the
> connection to the database is down or not available
> for some reason, but you still want to edit the connection
> object via the ZMI.
It may cause problems too when you kill (on the server side)
your opened connection. This connection will stay broken on the
client (Zope), because some adapters (at last DCOracle2)
doesn't have ability to reconnect in this situation.

> Most of the open buttons on database connections are pretty useless --
> they just make the connection open early. Most DA's open a connection
> on demand. For "Most" read "all" because I don't know of one that
> DOESNT work that way.
The thing I am not sure are volatile attributes
_v_database_connection. AFAIR volatile atributes are
defined for connections to ZODB, so they're independent
between ZODB connestions. So... in "connect on load"
scenario you may have several connections to database
opened in the same time - the number of connections
to relational database may be even same as the number of
connections to ZODB. Am I right???.
If yes, then default 'Close connection' button
implementation is useless too because it closes only one
self._v_database_connection that was used for Close operation.


> "Connect on demand" also tends to save resources and plays
> well with connections that time out after a while.
Isn't it slower to open and close connection all the time
instead of using already opened connection?


As far as I understand whole process it is something like
(if you want, then please, correct me):

----------------------------------------------------------------
1. Demand appears (somebody request a query execution).
For example ZSQLMethod is run. ZSQLMethod inherits from
Shared.DC.ZRDB.DA.DA so Shared.DC.ZRDB.DA.DA.__call__
is executed. It may also be a ZPublisher which executes
Shared.DC.ZRDB.DA.DA.__call__ (???)

2. Shared.DC.ZRDB.DA.DA.__call__ finds
Shared.DC.ZRDB.Connection.Connection object which is
simply a DatabaseConnection visible in ZMI.

3. When Shared.DC.ZRDB.Connection.Connection is found
it is executed (__call__) and supposed to return
instance of Shared.DC.ZRDB.TM.TM
(or rather it's descendant defined in specific database
adapter like SQLRelayDA or DCoracle2)

4. Shared.DC.ZRDB.Connection.Connection.__call__
checks whether it has instance of Shared.DC.ZRDB.TM.TM and
if so it is returned. Otherwise instance of
Shared.DC.ZRDB.TM.TM is created, stored in
volatile attribute this way:

self._v_database_connection=DB(s)

and returned.

According to M.-A. Lemburg we have two scenarios here:

a) "connect on load" scenario
constructor of class DA(Shared.DC.ZRDB.TM.TM) creates
a connection to database like DCOracle2:

def __init__(self,connection_string):
self.connection_string=connection_string
db=self.db=DCOracle2.connect(connection_string)
self.cursor=db.cursor()

which means we have opened connection to database which
is held in volatile _v_database_connection.

b) "connect on demand" scenario
Constructor of class DA(Shared.DC.ZRDB.TM.TM) doesn't
create a connection to database. Like SQLRelayDA which
only stores necessary parameters in __init__

5. Shared.DC.ZRDB.DA.DA.__call__
executes 'query' function on connection object retured
by instance of Shared.DC.ZRDB.Connection.Connection.

6. Transaction manager is being registered - function _begin
of transaction manager is called. It may be defined like:
def _begin(self):
self.con = PySQLRDB.connect(self.host, self.port,
self.socket, self.user,
self.password,0,1)
self.cur = self.con.cursor()

in SQLRelayDA (connection is created during query execution).
Or function _begin may do nothing because connection is
already stored in TransactionManager (it was created in
constructor) (DCOracle2).

7. Query is executed with database connection

8. ZPublisher commits TransactionManagers

9. Function _finish of TransactionManager is called - it may close
the database connection like SQLRelayDA:
def _finish(self, *ignored):
self.con.commit()
self.con.close()

or just commit a connection (DCOracle2).
----------------------------------------------------------------


--
Maciej Wisniowski

_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Maciej Wisniowski wrote at 2006-9-23 00:27 +0200:
> > You could recode DA's to close the connection at the end of every
> > transaction and reopen them on demand,
>Seems for me that difference betwen "connect on load"
>and "connect on demand" is in _begin, _finish and
>__init__ methods of TransactionManager (usually defined in db.py
>in specific database adapter).

The bad behaviour is in standard Zope code:

Shared.DC.ZRDB.Connection.Connection.__setstate__

It connects to the database as soon as the instance is loaded from
ZODB. My modified code looks like:

def __setstate__(self, state):
Globals.Persistent.__setstate__(self, state)
if self.connection_string:
# DM 2005-06-24: what a stupidity to connect on load!
# It should be delayed until the connections is really used.
# I do not change this here to not break DAs.
# DM 2005-06-24: "connect_" in order to protect '_v_' attributes
#try: self.connect(self.connection_string)
try: self.connect_(self.connection_string)
except:
LOG('Shared.DC.ZRDB.Connection',
ERROR,
'Error connecting to relational database.',
error=exc_info())



--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
>ZODB. My modified code looks like:
>
> def __setstate__(self, state):
> Globals.Persistent.__setstate__(self, state)
> if self.connection_string:
> # DM 2005-06-24: what a stupidity to connect on load!
> # It should be delayed until the connections is really used.
> # I do not change this here to not break DAs.
> # DM 2005-06-24: "connect_" in order to protect '_v_' attributes
> #try: self.connect(self.connection_string)
> try: self.connect_(self.connection_string)
> except:
> LOG('Shared.DC.ZRDB.Connection',
> ERROR,
> 'Error connecting to relational database.',
> error=exc_info())
>
Do you have 'connect_' defined elsewhere or it is expected to raise
an exception here and just log this?

--
Maciej Wisniowski
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Maciej Wisniowski wrote at 2006-9-24 22:00 +0200:
> ...
>Do you have 'connect_' defined elsewhere or it is expected to raise
>an exception here and just log this?

Sure, I have defined "connect_".

It uses a private ZODB extension to prevent "_v_" attributes to be lost
mid transaction (which can have nasty, partly non deterministic and
very difficult to understand effects) and then calls "connect".

In the context of our discussion, it is equivalent to "connect".



--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Dieter Maurer wrote:
>>> However, it really is now as you describe it...
>> Well, you should be a foundation member by now, right?
>
> Yes,

Yay!

> but my SVN access is not yet working (probably due to a '.' in
> my Zope user account).

Could you not just register another account name at zope.org and use that?!

Is anyone currently working on the issue?

cheers,

Chris

--
Simplistix - Content Management, Zope & Python Consulting
- http://www.simplistix.co.uk

_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
>It uses a private ZODB extension to prevent "_v_" attributes to be lost
>mid transaction (which can have nasty, partly non deterministic and
>very difficult to understand effects) and then calls "connect".
>
>
Does this mean that there is a bug in ZODB?

Thanks for your explanations. I've overriden __setstate__
in my DCOracle2 and removed 'connect' from it.
I'm convinced it is not necessary in this place.

--
Maciej Wisniowski
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Maciej Wisniowski wrote at 2006-9-25 20:09 +0200:
>
>>It uses a private ZODB extension to prevent "_v_" attributes to be lost
>>mid transaction (which can have nasty, partly non deterministic and
>>very difficult to understand effects) and then calls "connect".
>>
>>
>Does this mean that there is a bug in ZODB?

Not strictly speaking in ZODB:

The "_v_" use for database connections is dangerous (and if you
want buggy).

It can lead to nasty, apparently non-deterministic effects.
You may lose your database connection mid transaction
and part of what you think is a transaction (i.e. atomical) can be lost.

There are too ways to fix this problem: do not use "_v_" attributes
to maintain the database connection or extend the ZODB to
allow more control over the lifetime of "_v_" attributes.

In our private copy of Zope, I went the second approach
as "_v_" attributes are essential for several other things
(beside database connections) as well that suffer from
the same deficiency.



--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
>Not strictly speaking in ZODB:
>
> The "_v_" use for database connections is dangerous (and if you
> want buggy).
>
> It can lead to nasty, apparently non-deterministic effects.
> You may lose your database connection mid transaction
> and part of what you think is a transaction (i.e. atomical) can be lost.
>
> There are too ways to fix this problem: do not use "_v_" attributes
> to maintain the database connection or extend the ZODB to
> allow more control over the lifetime of "_v_" attributes.
>
Thanks again for explanations. Seems that I'll change my DA's
(DCOracle2 and ZCxOracleDA) to use connection pool like psycopg
does. After quick look at their (psycopg) solution I think it is really good
and solves (or may solve) few other issues too - like closing
all connections with 'Close connection' button (useless with
_v_ implementation). Implementing reconnect of broken connection
i.e. when database was restarted shouldn't be too difficult too.

--
Maciej Wisniowski
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Maciej Wisniowski wrote at 2006-9-27 17:49 +0200:
> ...
>Thanks again for explanations. Seems that I'll change my DA's
>(DCOracle2 and ZCxOracleDA) to use connection pool like psycopg
>does. After quick look at their (psycopg) solution I think it is really good
>and solves (or may solve) few other issues too - like closing
>all connections with 'Close connection' button (useless with
>_v_ implementation). Implementing reconnect of broken connection
>i.e. when database was restarted shouldn't be too difficult too.

Pool based connections can be tricky as well:

When a DA instance is accessed several times during the
same request, it *MUST* use the same connection.

Otherwise, you will get the same hazardous behaviour
as with "_v_" variables being lost, just more often
and more reliable...

I know that the "ZPsycopgDA" which comes with "psycopg 1.x" uses
"_v_" variables. I do not know whether that of "psycopg 2.x"
uses a connection pool and uses it in the correct way.

I only know that the correct connection pool use is tricky...



--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
>Pool based connections can be tricky as well:
>
> When a DA instance is accessed several times during the
> same request, it *MUST* use the same connection.
>
>
To be able to finish whole transaction
for the relational DB...

>I know that the "ZPsycopgDA" which comes with "psycopg 1.x" uses
>"_v_" variables. I do not know whether that of "psycopg 2.x"
>uses a connection pool and uses it in the correct way.
>
>
Unfortuantelly it still uses _v_database_connection... I've just found
this.

>I only know that the correct connection pool use is tricky...
>
The problem is to be able to determine which connection from
pool is being used during request (transaction). I think it is
possible to use request object or maybe the
Transaction object for this.

In _begin function (possibly) of TM I may define something like:

tr = transaction.get()
if hasattr(tr, 'id_of_con_from_pool_used'):
# returns specific connection that is already marked
# as 'taken' from the pool
conn = pool.getConn(tr.id_of_con_from_pool_used')
else:
# returns free connection from the pool and marks
# it as 'taken'
conn = pool.getConn()
setattr(tr, 'id_of_con_from_pool_used', conn.identifier)

and in _finish (abort etc.) method of TM remove additional attribute
and release connection to pool. It should give me the ability to
use same connection during transaction and because it is
'taken', it won't be used by any other threads.

The question is whether it is possible to use tranasction.get() for this?
Isn't Transaction something persistent? Or maybe its better to
use request object instead (sounds a bit ugly but why not?)?

Seems to be too simple so I've possibly missed something :)
Any ideas?

--
Maciej Wisniowski
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
>
> It can lead to nasty, apparently non-deterministic effects.
> You may lose your database connection mid transaction
> and part of what you think is a transaction (i.e. atomical) can be lost.
>
>
What are the circumstances under which this can occur? I've found something
similiar (?) when I was dealing with stored procedures in DCOracle2:

When you have a request that causes the new _v_database_connection to be
created and this new connection is used to execute stored procedure which
causes exception. Then... transaction is aborted and resource
managers' abort() functions are called.
I've found that newly created _v_database_connection was destroyed
before(!) ResourceManager's (used to handle relational db connection)
abort() function was called, so there was no rollback to database... huh...
Same thing may apply to ZRDB.DA.DA but I'm not sure about that.

--
Maciej Wisniowski
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Maciej Wisniowski wrote at 2006-9-27 21:38 +0200:
> ...
>To be able to finish whole transaction
>for the relational DB...

No.

There are two reasons:

1. that later operations see the effects of earlier
operations in the same request

and more importantly

2. to avoid deadlocks.

The relational database system allocates resources to connections
and often keeps it until the transaction finishes.

Whenn all operations towards the same database use
the same connection, the database can recognized
resources already allocated for the request.

If, however, the request uses different connections,
it may require a resource it already has locked via
a different connection (therefore, the database cannot
know, that it is the owner already) and this will
inevitably result in a deadlock.

> ...
>The problem is to be able to determine which connection from
>pool is being used during request (transaction).

Yes.

If one is careful, one can find a reliable solution...



--
Viele Grüße
Dieter

Tel: 06894-870 177
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Maciej Wisniowski wrote at 2006-9-27 22:40 +0200:
> ...
>> It can lead to nasty, apparently non-deterministic effects.
>> You may lose your database connection mid transaction
>> and part of what you think is a transaction (i.e. atomical) can be lost.
>>
>>
>What are the circumstances under which this can occur? I've found something
>similiar (?) when I was dealing with stored procedures in DCOracle2:

Several events must occur together such that this happens (that's
why the problem is not yet fixed).

While the ZODB does not promise any definite lifetime for
"_v_" variables (they may disappear at any time),
the current implementation calls cache garbage collection
only at savepoint and transaction boundaries.
During cache garbage collection objects may be deactivated
which causes their content (including "_v_" variables) to be deleted.
While the ZODB provides other means to deactivate
objects than cache garbage collection, there are rarely used
(especially on a DA object).
Thus, one condition is that savepoints (formerly "subtransaction"s)
are used.
However, some Zope operations, e.g. large cataloguing operations
use savepoints (in order to reduce memory footprint by
early cache garbage collection), without you beeing aware of.

There is not problem when the DA is deactivated when it was
not used in the same transaction. If however, it is use in this
transaction, then it is used rather recently. Cache garbage collection
first deactivates the least recently used objects.
A second condition is therefore, that your transaction accesses
huge numbers of objects (more than the cache size) since it accessed
the DA and then it performs a savepoint.

If these conditions are met, you will lose your connection. Still
nothing bad happens as the transaction still holds a reference to it
and may eventually commit/abort it at transaction end time.

If however, the same request accesses the DA again, a new connection
will be opened. This new connection will not see the effects
performed via the former connection and may deadlock on
access to resources locked by the former connection.
In these rare cases, you will have problems.
And as they are very rare and apparently non-deterministic,
they are almost impossible to analyse (and quite difficult to
understand).


>When you have a request that causes the new _v_database_connection to be
>created and this new connection is used to execute stored procedure which
>causes exception. Then... transaction is aborted and resource
>managers' abort() functions are called.
>I've found that newly created _v_database_connection was destroyed
>before(!) ResourceManager's (used to handle relational db connection)
>abort() function was called, so there was no rollback to database... huh...

This might be possible but need not be a problem:

The order in which operations occur at commit time can be controlled
via the resource manager's "sortKey" (or similarly spelled).

As the sort order between ZODB resources and relational
resources is not really relevant, it may well be
that the ZODB connection is aborted before the relational connection.

In this case, you may see the loss of the "_v_" variable
(caused either by the cache garbage collection or the invalidation
of the DA object) before the relational "abort".

However, the transaction holds an additional (beside the "_v_" attribute)
reference to the connection and can commit/abort it despite
the loss of the "_v_" attribute.



--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
>If one is careful, one can find a reliable solution...
>
Hopefully I think I did this today...

I've found that psycopg pool implementation has nice
class:

class PersistentConnectionPool(AbstractConnectionPool):
"""A pool that assigns persistent connections to different threads.

Note that this connection pool generates by itself the required keys
using the current thread id. This means that untill a thread put away
a connection it will always get the same connection object by successive
.getconn() calls. This also means that a thread can't use more than one
single connection from the pool.
"""

I used this, and finally I have DCOracle2 with connection pool, without
volatile attributes (I didn't changed implementation of stored
procedures yet),
with ability to reconnect after things like DB restart and with REAL
ability to
open/close connections from ZMI.

I did one important change. Instead of having pool of connections I have
pool
of Resource Managers (or Transaction Managers - I'm not sure which name
is correct). These are descendants of Shared.DC.ZRDB.TM.TM and are usually
defined in db.py ad DA(Shared.DC.ZRDB.TM.TM). I'll call them RM.

Every RM in my pool is already connected to database - it has self.db.


With this implementation the typical request is:

0. pool of RM is created - usually it is during first call to 'getconn'
function
that pops connection from the pool.

1. ZSQLMethod (or other DA) calls database connection object
which is descendant of Shared.DC.ZRDB.Connection.Connection

2. Descendant of Shared.DC.ZRDB.Connection.Connection
in it's __call__ function gets free RM from the pool and
returns it to ZSQLMethod.
RM when taken from pool is marked as 'used' in the pool
and is assigned a key that is thread id. Every subsequent call
to 'getconn' checks if there already is 'used' RM under specific
key (thread id), and if so, returns this. During request we always
have same thread id so... it just works :)

3. ZSQLMethod executes 'query' on RM. RM registers itself
in Transaction

Steps 1-3 are repeated for each ZSQLMethod used during request.
Thanks to PersistentConnectionPool ZSQLs always get
same RM. As you probably noticed, RM taken from the pool
at the begining was not returned to the pool yet - it is still marked
as 'used' so no other thread is able to get it.

4. Request finishes so all registered RMs are commited (or aborted)
and their _finish method is called. A bit ugly thing happens here:

5. In _finish method RM puts itself :-/ back to the pool.
It is not possible (at last I have no idea how) to release RM in
different place because I don't know when request is finished.
But this implementation seems to work correcly.


One more thing I'm wondering about is getting RMs from
the pool (free or even used) because it is always done with code
like:

_connections_lock.acquire()
try:
....
finally:
_connections_lock.release()

may it be slow with high load? I mean that every time you need
the connection this code is used.

--
Maciej Wisniowski
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
> If these conditions are met, you will lose your connection. Still
> nothing bad happens as the transaction still holds a reference to it
> and may eventually commit/abort it at transaction end time.
>(...)
> And as they are very rare and apparently non-deterministic,
> they are almost impossible to analyse (and quite difficult to
> understand).
>
>
Thanks! I think, I have no such conditions met in my apps, ufff... :)

>This might be possible but need not be a problem:
>
> The order in which operations occur at commit time can be controlled
> via the resource manager's "sortKey" (or similarly spelled).
>
>
I've seen that but was not sure what value is returned
by ZODB resources in sortKey. Possibly it's easy to check this.

> In this case, you may see the loss of the "_v_" variable
> (caused either by the cache garbage collection or the invalidation
> of the DA object) before the relational "abort".
>
>
I don't know how abort of ZODB resource managers works but
doesn't rollback of ZODB resource manager remove _v_ variables?
or.. maybe it causes cache garbage collection or invalidation?

> However, the transaction holds an additional (beside the "_v_" attribute)
> reference to the connection and can commit/abort it despite
> the loss of the "_v_" attribute.
>
>
How can I get access to this? _v_db in Procedure class in
SP.py of DCOracle2 disappeared in my case...

--
Maciej Wisniowski
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Maciej Wisniowski wrote at 2006-9-28 23:05 +0200:
> ...
>I don't know how abort of ZODB resource managers works but
>doesn't rollback of ZODB resource manager remove _v_ variables?

Usually not.

An abort only deactivates (and deactivation kills the "_v_" variables
together with other content) those objects that have been
modified by the transaction.

>or.. maybe it causes cache garbage collection or invalidation?
>
>> However, the transaction holds an additional (beside the "_v_" attribute)
>> reference to the connection and can commit/abort it despite
>> the loss of the "_v_" attribute.
>>
>>
>How can I get access to this?

The value of the "_v_" attribute is *the* resource manager
registered with the transaction.


--
Dieter
_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
>>How can I get access to this?
>>
>>
>
>The value of the "_v_" attribute is *the* resource manager
>registered with the transaction.
>
Ups... of course! I've made mistake - sorry.
My error was about self.db attribute in
'_v_database_connection' not about '_v_database_connection'
attribute itself. During abort() of this resource manager
self.db was None so self.db.rollback() failed. I'll check
this again...

--
Maciej Wisniowski


_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
> My error was about self.db attribute in
> '_v_database_connection' not about '_v_database_connection'
> attribute itself. During abort() of this resource manager
> self.db was None so self.db.rollback() failed. I'll check
> this again...
I've messed up things. Error that I've found is in
StoredProcedures in DCOracle2 -> SP.py and
this issue seems to be specific to DCOracle2 SP class.

SP class is ResourceManager that uses self._v_db
attribute that is in fact another ResourceManger.

When stored procedure in database raises an exception
self._v_db attribute of SP class instance disappears
(if it was created during same request). As a result
"AttributeError: _v_db" is raised when ResourceManager (SP)
is being aborted because it tries to call self._v_db.db.rollback().

I'm not sure why DCOracle2 treats SP as another
ResourceManager while it still uses same db connection as
the other ZSQLs etc. As a result if we have a lot of SP calls
during one request there is a lot of commits (rollbacks) to the same
db connection. Maybe it's something about caching procedure
objects in volatile attributes.

--
Maciej Wisniowski

_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db
Re: Zope database connectivity [ In reply to ]
Hi,

Maciej Wisniowski napisa³(a):
>
>> If one is careful, one can find a reliable solution...
>>
> Hopefully I think I did this today...
>
> I've found that psycopg pool implementation has nice
> class:
>
> class PersistentConnectionPool(AbstractConnectionPool):
> """A pool that assigns persistent connections to different threads.
>
> Note that this connection pool generates by itself the required keys
> using the current thread id. This means that untill a thread put away
> a connection it will always get the same connection object by successive
> .getconn() calls. This also means that a thread can't use more than one
> single connection from the pool.
> """
>
> I used this, and finally I have DCOracle2 with connection pool, without
> volatile attributes (I didn't changed implementation of stored
> procedures yet),
> with ability to reconnect after things like DB restart and with REAL
> ability to
> open/close connections from ZMI.

Did anyone try SQL Relay as a bridge? They say "When using SQL Relay,
the database can be bounced without having to restart Zope"


Pawel Lewicki

_______________________________________________
Zope-DB mailing list
Zope-DB@zope.org
http://mail.zope.org/mailman/listinfo/zope-db

1 2  View All