Mailing List Archive: ZODB Key error in root Zope database

We've got a bit of a situation...

We run Zope 2.11 with a main ZODB and a few dozen separate file storages
for sub-sites.

Occasionally, we've encountered the problem where a cut-and-paste occurs
between storages and upon a pack results in a key error. Normally, we
avoid this because the FileStorages represent individual sites.

Last night something happened where a script was cut-and-pasted from one
of the sub-storages into the main, root-level, database.

Upon a restart we encountered a key error:

File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py",
line 361, in get_connection
new_con = self._db.databases[database_name].open(
KeyError: 'site1'

Normally, we've encountered these errors when a database is packed and
the solution is to revert to a pre-pack database and remove or fix the
offending object.

In this case, we can't even start Zope or mount the storage in a script
to remove the offending items.

Before I go on, I found that ZODB 3.9 provides options to disallow
cross-database references and we'll work to get to this version during
our next upgrade cycle.

In the mean time we've reverted to a previous backup prior to the
offending object being placed in the root database. This resulted in
about 200 MB of data loss. I would like a way to mount the corrupted
database and attempt to determine exactly what was lost.

What we've tried so far has gone like this:

from ZODB import FileStorage, DB
from OFS.Folder import Folder

storage = FileStorage.FileStorage('BadData.fs')
db = DB(storage)
conn = db.open()
root = conn.root()
app = root['Application']

app.manage_deleteObjects(['OffendingObject'])

# This throws the same exception so we tried working around the
# mechanics of manage_deleteObjects with this.

app._objects = tuple([i for i in app._objects if i['id'] !=
'OffendingObject'])

Same exception. Can't access the database. We've tried using _setOb to
replace the object.

Any suggestions? It seems as though it needs to be able to access
_objects before it mounts the mounted databases.

Thanks,

--
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
_______________________________________________
Zope maillist - Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )

FYI,

I was able to successfully remove the cross storage reference in the
root of my main Zope database by explicitly setting the missing database
connection to a hacked DummyConnection instance. I probably could have
dug deeper to create a real connection, but this worked.

from ZODB import FileStorage, DB
import transaction

storage = FileStorage.FileStorage('Data.fs')
app = DB(storage).open().root()['Application']
base_conn = app._p_jar

class DummyConnection:
""" implements enough of a connection to allow for
removing of the cross database reference """
def __init__(self):
self.connections = []
self._cache = {}

def get(self, key):
return None

def open(self, **kw):
return DummyConnection()

base_conn._db.databases['site1'] = DummyConnection()

app._objects = tuple([i for i in app._objects if i['id'] != 'badID'])

transaction.get().commit()

It's quite the hack, but allowed us to get the database back.

Brian Brinegar
Web Services Coordinator
Engineering Computer Network
--
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
_______________________________________________
Zope maillist - Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
https://mail.zope.org/mailman/listinfo/zope-announce
https://mail.zope.org/mailman/listinfo/zope-dev )